benchmark.md

Benchmark

OpenCompass Benchmark

Check OpenCompass Leaderboard for more details.

Model		Vicuna-33B	WeMix-LLaMA2-7B	LLaMA-2-7B-Chat	Vicuna-7B	LLaMA-2-7B	Alpaca-7B	LLaMA-7B
	OVERALL	50	49.6	44.8	43.4	41.6	39.9	38.5
	EXAM	49.2	45.5	40.1	40.5	35.5	35.3	31.2
	LANGUAGE	44.9	45.1	44	39.6	44.1	39.5	40.5
	KNOWLEDGE	61.3	59.4	54.3	51.7	53.3	44.6	49.6
	UNDERSTANDING	58.5	55.5	50.9	50.5	42.4	45.1	38
	REASONING	44.7	47.4	41.4	39.9	40.1	38.1	38.5
EXAM	C-Eval	37.8	37.5	31.9	33.3	32.5	29.9	27.3
	AGIEval	32.8	31.3	28.6	26.6	21.8	24.3	20.6
	MMLU	59.2	54.1	46.2	48.2	46.8	41.7	35.6
	CMMLU	38.8	37.2	31.5	33.5	31.8	28.7	26.8
	GAOKAO-Bench	23.9	17	16.1	22.4	18.9	19.6	21.3
	ARC-c	69.2	65.1	54.9	51.2	40.3	41.4	34.6
	ARC-e	82.9	76.5	71.6	68.4	56.1	61.4	52.2
LANGUAGE	WiC	67.5	62.9	55	51.2	50	50.3	50.2
	CHID	35.1	42.1	44.1	29.2	46.5	26.7	31.2
	AFQMC	69	68.5	69	69	69	69	68.9
	WSC	68.3	66.3	69.2	61.5	66.3	65.4	65.4
	TyDiQA	22.6	24.8	21.6	21.7	26.8	20.7	22.5
	Flores	6.7	5.8	5.2	5	6	4.6	4.8
KNOWLEDGE	BoolQ	85.7	85.6	81.2	78.3	74.9	79.5	75.4
	CommonSenseQA	71	69.9	69.9	66.4	66.5	69.4	64.9
	TriviaQA	61.2	56.1	46.4	45.6	52.8	25.1	46.3
	NaturalQuestions	27.3	26.2	19.6	16.4	19.1	4.4	11.7
REASONING	CMNLI	30.6	42	36.1	40	34.9	33	34.6
	OCNLI	30.8	36.2	36.4	35.4	32.1	30.3	33.6
	AX-b	59.7	52.1	58.5	58.2	53.5	52.3	57.4
	AX-g	53.9	50	51.7	50	55.1	47.2	50
	RTE	52.7	56.7	55.2	48	52	55.6	50.9
	COPA	71	67	72	72	67	72	72
	ReCoRD	3.1	45	22.5	20.3	32.7	22.9	14.9
	HellaSwag	79	76.2	74.2	72.8	74	73.6	74.3
	PIQA	80.6	78.6	76.2	78.7	78.3	78.3	78.6
	SIQA	65.1	71.5	55.4	54.7	48.5	52.7	46.2
	MATH	6.8	4.6	3.9	3.1	3.3	3	2.9
	GSM8K	44	33	26.3	14.9	16.7	6	10
	DROP	44	44.4	28.7	28.2	27.2	23.1	27.9
	HumanEval	15.8	29.3	12.2	10.4	12.8	9.2	12.8
	MBPP	26.6	29.8	17.6	14.4	14.8	17.8	16.8
	BBH	52	42.4	35.6	37.5	38.2	32.8	33.5
UNDERSTANDING	C3	48.3	47.8	49.8	45.6	42.9	39.3	39
	RACE(Middle)	79	82.3	62	63.9	40.2	54.4	36.8
	RACE(High)	76.4	77.8	58.6	59.3	37.5	47.2	30.6
	OpenbookQA	80.4	84.6	74.4	73.2	57	65.4	34.6
	CSL	58.1	53.8	58.8	55	55.6	54.4	54.4
	LCSTS	11.1	4.2	9.6	11	9.1	4.3	6.7
	XSum	27.4	32.8	20.8	24.1	19.7	27.3	20.5
	EPRSTMT	73.8	46.2	57.5	53.1	46.2	46.2	46.2
	LAMBADA	72.1	69.6	66.9	69.2	73.3	67.1	73.3