mxnv_acc.md

MXFP4 variants

Qwen3-8B

Method	avg	MMLU	arc_challenge	arc_easy	boolq	gsm8k	hellaswag	lambada_openai	openbookqa	piqa	truthfulqa_mc1	winogrande
ocp rtn	0.607072727	0.6613	0.4932	0.7715	0.8593	0.8127	0.5223	0.5851	0.2800	0.7301	0.3293	0.6330
rceil rtn	0.614190909	0.6718	0.4974	0.7656	0.8450	0.8271	0.5299	0.5905	0.2880	0.7427	0.3525	0.6456
rceil 7.25 RTN	0.617018182	0.6780	0.5000	0.7778	0.8498	0.8317	0.5338	0.5991	0.2940	0.7329	0.3476	0.6425
ocp iters 200	0.624609091	0.6771	0.5171	0.8119	0.8456	0.7961	0.5284	0.5979	0.3080	0.7546	0.3513	0.6827
rceil iters 200	0.618427273	0.6794	0.5111	0.7984	0.8550	0.7680	0.5242	0.5993	0.3040	0.7530	0.3378	0.6725
rceil 7.25 iters 200	0.627009091	0.6837	0.5307	0.8136	0.8569	0.8052	0.5317	0.6113	0.2960	0.7514	0.3378	0.6788

Llama3.1-8B-I

Method	avg	MMLU	arc_challenge	arc_easy	boolq	gsm8k	hellaswag	lambada_openai	openbookqa	piqa	truthfulqa_mc1	winogrande
ocp rtn	0.575163636	0.5734	0.4582	0.7630	0.8080	0.5011	0.5554	0.6216	0.3080	0.7606	0.2766	0.7009
rceil rtn	0.6031	0.6002	0.4710	0.7816	0.8235	0.5792	0.5661	0.6647	0.3260	0.7780	0.3390	0.7048
rceil 7.25 rtn	0.601372727	0.6057	0.4753	0.7803	0.8242	0.5663	0.5657	0.6637	0.3340	0.7769	0.3158	0.7072
ocp iters 200	0.605645455	0.6160	0.4659	0.8009	0.8257	0.6035	0.5577	0.6831	0.3000	0.7709	0.3415	0.6969
rceil iters 200	0.607863636	0.6133	0.4846	0.7959	0.8300	0.6156	0.5628	0.6645	0.3120	0.7835	0.3329	0.6914
rceil 7.25 iters 200	0.614072727	0.6149	0.4804	0.7984	0.8303	0.6262	0.5670	0.6829	0.3300	0.7807	0.3305	0.7135

Average accuracy of hellaswag,lambada_openai,mmlu,piqa,winogrande.

We evaluated using a fake model since we currently have no access to devices for running the real models. However, we have verified that in most cases the fake model closely matches the real model.

mxfp4 g32	llama3.1-8B-Instruct	Qwen2-7.5-Instruct	Phi4	Qwen3-32B
RTN	0.6212	0.6550	0.7167	0.6901
AutoRound	0.6686	0.6758	0.7247	0.7211
AutoRound+alg_ext	0.6732	0.6809	0.7225	0.7201

nvfp4 g16	llama3.1-8B-Instruct	Qwen2-7.5-Instruct	Phi4	Qwen3-32B
RTN	0.6876	0.6906	0.7296	0.7164
AutoRound	0.6918	0.6973	0.7306	0.7306
AutoRound+alg_ext	0.6965	0.6989	0.7318	0.7295