opt_rtn.md

🧮 Evaluation Results (LM-Eval)

For 2/3bit, we strongly recommend not using iter=0 except for GGUF:Q2_K_S which has a different quantization algorithm.

4BIT=W4A16 3BIT=W3A16 2BIT=W2A16G64

RTN mode

auto-round --model xxx --disable_opt_rtn --iters 0 

OPT RTN mode

auto-round --model xxx  --iters 0 

Model	RNT/OPT	AVG	HellaSwag	LAMBADA	MMLU	PIQA	WinoGrande
Meta-Llama-3.1-8B-Instruct	RTN-4BIT	0.69328	0.5896	0.7013	0.6538	0.7987	0.7230
	OPT-4BIT	0.69560	0.5882	0.7074	0.6631	0.7916	0.7277
	RTN-3BIT	0.64562	0.5410	0.6695	0.5449	0.7742	0.6985
	OPT-3BIT	0.65970	0.5490	0.6893	0.5711	0.7677	0.7214
	RTN-2BIT	0.33008	0.2918	0.0474	0.2321	0.5740	0.5051
	OPT-2BIT	0.38908	0.3241	0.1560	0.2822	0.6235	0.5596
Qwen2.5-7B-Instruct	RTN-4BIT	0.69560	0.6114	0.6713	0.7011	0.7878	0.7064
	OPT-4BIT	0.70034	0.6143	0.6945	0.7115	0.7845	0.6969
	RTN-3BIT	0.64144	0.5585	0.6092	0.6455	0.7476	0.6464
	OPT-3BIT	0.66764	0.5756	0.7013	0.6597	0.7481	0.6535
	RTN-2BIT	0.31856	0.2804	0.0351	0.2379	0.5256	0.5138
	OPT-2BIT	0.45146	0.3645	0.2992	0.4043	0.6415	0.5478
Qwen3-8B	RTN-4BIT	0.66240	0.5619	0.6150	0.7077	0.7573	0.6701
	OPT-4BIT	0.66992	0.5619	0.6346	0.7102	0.7633	0.6796
	RTN-3BIT	0.57322	0.4992	0.4260	0.6002	0.7361	0.6046
	OPT-3BIT	0.63698	0.5226	0.5814	0.6718	0.7437	0.6654
	RTN-2BIT	0.31150	0.2679	0.0041	0.2536	0.5283	0.5036
	OPT-2BIT	0.44254	0.3749	0.2005	0.4202	0.6670	0.5501
Qwen3-14B	RTN-4BIT	0.70448	0.5999	0.6511	0.7565	0.7998	0.7151
	OPT-4BIT	0.70798	0.6031	0.6627	0.7534	0.8009	0.7198
	RTN-3BIT	0.65876	0.5746	0.5467	0.7065	0.7628	0.7032
	OPT-3BIT	0.68610	0.5683	0.6633	0.7258	0.7699	0.7032
	RTN-2BIT	0.39398	0.3764	0.0607	0.3836	0.6480	0.5012
	OPT-2BIT	0.50080	0.4554	0.2451	0.4899	0.7138	0.5998