mmlu-redux.summary.md

Model	Mode	Acc	No answer	Total	Reason Lens
o1-preview-2024-09-12	greedy	92.84	0.61	2778	557.91
claude-3-5-sonnet-20241022	greedy	88.91	0	2778	787.31
chatgpt-4o-latest-24-09-07	greedy	88.88	0.07	2778	622.27
gpt-4o-2024-08-06	greedy	88.26	0.04	2778	691.73
gpt-4o-2024-05-13	greedy	88.01	0.14	2778	629.79
grok-2-1212	greedy	87.4	0	2778	812.14
o1-mini-2024-09-12	greedy	86.72	0.32	2778	464.2
Llama-3.1-405B-Inst@sambanova	greedy	86.21	0.25	2778	629.07
gemini-1.5-pro-exp-0827	greedy	86.14	1.26	2778	858.08
claude-3-5-sonnet-20240620	greedy	86	0.18	2778	907.1
Llama-3.1-405B-Inst-fp8@together	greedy	85.64	0.76	2778	449.71
Qwen2.5-72B-Instruct	greedy	85.57	0	2778	667.07
gemini-1.5-pro-exp-0801	greedy	85.53	1.22	2778	656.64
gpt-4-turbo-2024-04-09	greedy	85.31	0.04	2778	631.38
Llama-3.1-405B-Inst@hyperbolic	greedy	84.41	1.15	2778	563.13
Qwen2.5-32B-Instruct	greedy	83.66	0.04	2778	626.06
Meta-Llama-3.1-70B-Instruct	greedy	82.97	1.26	2778	668.14
Mistral-Large-2	greedy	82.97	0.14	2778	490.94
gemini-1.5-pro	greedy	82.76	1.94	2778	666.7
claude-3-opus-20240229	greedy	82.54	0.58	2778	500.35
yi-large-preview	greedy	82.15	0.14	2778	982.6
gemini-1.5-flash-exp-0827	greedy	82.11	1.48	2778	794.53
gpt-4-0314	greedy	81.64	0.04	2778	397.22
Qwen2-72B-Instruct	greedy	81.61	0.29	2778	486.41
gpt-4o-mini-2024-07-18	greedy	81.5	0.07	2778	526
yi-large	greedy	81.17	0	2778	774.85
deepseek-v2-chat-0628	greedy	80.81	0.11	2778	691.91
deepseek-v2.5-0908	greedy	80.35	0.22	2778	641.15
deepseek-v2-coder-0724	greedy	80.24	0.32	2778	650.65
deepseek-v2-coder-0614	greedy	79.63	0.14	2778	704.72
claude-3-5-haiku-20241022	greedy	79.63	0	2778	788.72
Meta-Llama-3-70B-Instruct	greedy	78.01	0.11	2778	520.77
gemini-1.5-flash	greedy	77.36	1.26	2778	583.45
Athene-70B	greedy	76.64	0.04	2778	552.61
reka-core-20240501	greedy	76.42	0.76	2778	701.67
gemma-2-27b-it	greedy	75.67	0.61	2778	446.51
Qwen2.5-7B-Instruct	greedy	75.13	0.11	2778	759.05
claude-3-sonnet-20240229	greedy	74.87	0.07	2778	671.75
gemma-2-9b-it	greedy	72.82	0.76	2778	499
Yi-1.5-34B-Chat	greedy	72.79	1.01	2778	620.1
claude-3-haiku-20240307	greedy	72.32	0.04	2778	644.59
Phi-3-mini-4k-instruct	greedy	70.34	0.43	2778	677.09
command-r-plus	greedy	68.61	0	2778	401.51
gpt-3.5-turbo-0125	greedy	68.36	0.04	2778	357.92
Phi-3.5-mini-instruct	greedy	67.67	4.5	2778	899.49
Meta-Llama-3.1-8B-Instruct	greedy	67.24	4.25	2778	732.7
Qwen2-7B-Instruct	greedy	66.92	0.72	2778	533.15
Mistral-Nemo-Instruct-2407	greedy	66.88	0.47	2778	464.19
Yi-1.5-9B-Chat	greedy	65.05	4.61	2778	542.87
reka-flash-20240226	greedy	64.72	0.32	2778	659.25
Qwen2.5-3B-Instruct	greedy	64.25	0.72	2778	708.95
Mixtral-8x7B-Instruct-v0.1	greedy	63.17	5.51	2778	324.31
Meta-Llama-3-8B-Instruct	greedy	61.66	0.97	2778	600.81
command-r	greedy	61.12	0.04	2778	382.23
Llama-3-Instruct-8B-SimPO-v0.2	greedy	55.22	1.19	2778	450.6
gemma-2-2b-it	greedy	51.94	2.59	2778	554.74
Qwen2-1.5B-Instruct	greedy	41.11	7.74	2778	280.56