Evaluation

We evaluate AutoPrune with different LLaVA models on a diverse set of 10 benchmarks. To ensure the reproducibility, we evaluate the models with greedy decoding following the originial LLaVA.

Scripts

Before preparing task-specific data, you MUST first download eval.zip. It contains custom annotations, scripts, and the prediction files with vanilla LLaVA-1.5. Extract it to ./playground/data/eval. This also provides a general structure for all datasets.

VQAv2

1. Download test2015 and put it under ./playground/data/eval/vqav2.
2. Multi-GPU inference.

CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/v1_5/eval/vqav2.sh 64

3. Submit the results to the evaluation server: ./playground/data/eval/vqav2/answers_upload.

GQA

1. Download the data and evaluation scripts following the official instructions and put under ./playground/data/eval/gqa/data. You may need to modify eval.py as this due to the missing assets in the GQA v1.2 release.
2. Multi-GPU inference.

CUDA_VISIBLE_DEVICES=0,1,2,3,4 bash scripts/v1_5/eval/gqa.sh 64

ScienceQA

1. Under ./playground/data/eval/scienceqa, download images, pid_splits.json, problems.json from the data/scienceqa folder of the ScienceQA repo.
2. Single-GPU inference and evaluate.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh 64

TextVQA

1. Download TextVQA_0.5.1_val.json and images and extract to ./playground/data/eval/textvqa.
2. Single-GPU inference and evaluate.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh 64

POPE

1. Download coco from POPE and put under ./playground/data/eval/pope.
2. Single-GPU inference and evaluate.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh 64

MME

1. Download the data following the official instructions here.
2. Downloaded images to MME_Benchmark_release_version.
3. put the official eval_tool and MME_Benchmark_release_version under ./playground/data/eval/MME.
4. Single-GPU inference and evaluate.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh 64

MMBench

1. Download mmbench_dev_20230712.tsv and put under ./playground/data/eval/mmbench.
2. Single-GPU inference.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench.sh 64

3. Submit the results to the evaluation server: ./playground/data/eval/mmbench/answers_upload/mmbench_dev_20230712.

MMBench-CN

1. Download mmbench_dev_cn_20231003.tsv and put under ./playground/data/eval/mmbench.
2. Single-GPU inference.

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench_cn.sh 64

3. Submit the results to the evaluation server: ./playground/data/eval/mmbench/answers_upload/mmbench_dev_cn_20231003.

Scripts with LLaVA-NeXT (LLaVA-1.6)

To evaluate AutoPrune with LLaVA-NeXT, you just need to replace the v1_5 with v1_6 in the shell scripts. For example, to evaluate VQAv2 with LLaVA-NeXT, you can run:

CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/v1_6/eval/vqav2.sh 160

Results

LLaVA-1.5-7B

Method	Present at	Avg. tokens	MME	MMB	SQA	GQA	TextVQA	Ratio	FLOPs
LLaVA-1.5-7B	NeurIPS’24	576	1862	64.7	69.5	61.9	58.2	100%	100%
Avg. tokens 192
ToMe [15]	arXiv’22	192	1563	60.5	65.2	54.3	52.1	89.9%	44.3%
FastV [18]	ECCV’24	192	1612	61.2	67.3	52.7	52.5	90.6%	45.7%
SparseVLM [20]	arXiv’24	192	1721	62.5	69.1	57.6	66.3	95.5%	46.3%
PDrop [13]	CVPR’25	192	1797	63.3	69.2	57.3	56.5	96.8%	43.9%
Ours	-	192	1832	64.9	69.6	60.4	57.7	99.0%	42.9%
Avg. tokens 128
ToMe [15]	arXiv’22	128	1343	53.3	59.6	52.4	49.1	81.1%	35.1%
FastV [18]	ECCV’24	128	1490	56.1	60.2	49.6	50.6	83.9%	36.8%
SparseVLM [20]	arXiv’24	128	1696	60.0	67.1	56.0	54.9	93.0%	37.3%
PDrop [13]	CVPR’25	128	1761	61.6	68.4	57.1	56.6	95.6%	35.1%
Ours	-	128	1785	64.3	69.7	59.9	57.4	98.1%	33.7%
Avg. tokens 64
ToMe [15]	arXiv’22	64	1138	43.7	50.0	48.6	45.3	70.5%	25.7%
FastV [18]	ECCV’24	64	1256	48.0	51.1	46.1	47.8	73.7%	27.9%
SparseVLM [20]	arXiv’24	64	1505	56.2	62.2	52.7	51.8	85.9%	28.2%
PDrop [13]	CVPR’25	64	1561	58.8	69.0	47.5	50.6	87.6%	25.5%
Ours	-	64	1745	63.6	69.6	57.7	57.1	96.7%	23.2%

Method	Present at	Avg. tokens	GQA	SQA	TextVQA	POPE	MME	MMB	MMBCN	Ratio (%)
LLaVA-1.5-7B	NeurIPS’24	576	61.9	69.5	58.2	85.9	1511	64.7	58.3	100.0
Avg. tokens 128
ToMe	ICLR’23	128	52.4	59.6	49.1	62.8	1088	53.3	48.8	80.9%
FastV	ECCV’24	128	49.6	60.2	50.6	59.6	1209	56.1	51.4	82.6%
SparseVLM	ICML’25	128	56.0	67.1	54.9	80.5	1376	60.0	51.1	92.4%
PruMerge+	arXiv’24	128	57.8	67.6	54.3	81.5	1421	61.3	54.7	94.5%
VisionZip	CVPR’25	128	57.6	68.9	56.8	83.2	1432	62.0	56.7	96.4%
FasterVLM	arXiv’24	128	58.2	69.1	57.0	84.6	1461	62.7	57.3	97.4%
Ours	-	128	59.9	69.6	57.7	85.2	1458	64.3	58.3	98.7%
Avg. tokens 64
ToMe	ICLR’23	64	48.6	50.0	45.3	52.5	922	43.7	38.9	69.2%
FastV	ECCV’24	64	46.1	51.1	47.8	48.0	1020	48.0	42.7	71.6%
SparseVLM	ICML’25	64	52.7	62.2	51.8	75.1	1221	56.2	46.1	85.4%
PruMerge+	arXiv’24	64	54.9	68.6	53.0	77.4	1198	59.3	51.0	89.6%
VisionZip	CVPR’25	64	55.1	69.0	55.5	77.0	1366	60.1	55.4	93.1%
FasterVLM	arXiv’24	64	55.4	69.1	55.8	80.4	1370	61.3	55.1	94.0%
Ours	-	64	57.7	69.6	57.1	82.5	1445	63.6	57.1	97.1%

LLaVA-1.5-13B

Method	Present at	# Token	VQAV2	GQA	TextVQA	POPE	MME	Ratio (%)
LLaVA-1.5-13B	-	576	80.0	63.3	61.2	86.0	1531	100%
Tokens 288
FastV	ECCV’24	288	79.5	62.6	60.9	85.2	1545	99.6%
SparseVLM	ICML’25	288	78.5	59.9	59.5	71.3	1497	94.1%
FasterVLM	arXiv’24	288	79.0	61.0	60.0	86.0	1530	98.6%
Ours	-	288	79.8	63.0	60.9	86.1	1530	99.8%
Tokens 144
FastV	ECCV’24	144	77.2	59.9	60.0	79.4	1494	95.8%
SparseVLM	ICML’25	144	76.1	58.0	57.9	68.6	1499	91.8%
FasterVLM	arXiv’24	144	77.4	58.7	59.0	83.1	1467	95.7%
Ours	-	144	79.0	61.5	60.2	86.7	1506	98.7%
Tokens 58
FastV	ECCV’24	58	70.3	54.9	55.6	67.3	1360	86.5%
SparseVLM	ICML’25	58	68.3	54.4	52.6	62.6	1285	82.8%
FasterVLM	arXiv’24	58	73.1	56.0	57.4	74.7	1371	90.0%
Ours	-	58	77.2	58.5	59.0	83.6	1478	95.8%

LLaVA-NeXT-7B

Method	Present at	Tokens	VQAV2	GQA	TextVQA	POPE	MME	Ratio
LLAVA-NeXT-7B	NeurIPS’24	2880	81.2	62.9	59.6	86.3	1513.8	100.0%
Tokens 640
FastV	ECCV’24	640	78.9	60.4	58.4	83.1	1477.3	97.0%
SparseVLM	arXiv’24	640	78.2	59.1	56.2	80.9	1456.3	94.9%
VisionZip	CVPR’25	640	79.2	60.1	58.5	82.2	1468.4	96.7%
FasterVLM	arXiv’24	640	79.8	61.6	59.3	85.9	1480.7	98.6%
Ours	–	640	80.5	62.6	59.6	86.7	1515.7	99.7%
Tokens 320
FastV	ECCV’24	320	71.9	55.9	55.7	71.7	1282.9	87.7%
SparseVLM	arXiv’24	320	71.4	56.5	52.4	73.5	1342.7	87.9%
VisionZip	CVPR’25	320	74.2	58.1	55.3	75.0	1348.8	90.5%
FasterVLM	arXiv’24	320	75.7	58.4	57.6	80.4	1370.1	93.3%
Ours	–	320	78.9	61.3	59.5	85.6	1471.6	98.2%
Tokens 160
FastV	ECCV’24	160	61.8	49.8	51.9	51.7	1079.5	74.7%
SparseVLM	arXiv’24	160	62.2	50.2	45.1	54.6	1167.1	74.9%
VisionZip	CVPR’25	160	67.3	54.3	54.7	59.4	1239.7	82.3%
FasterVLM	arXiv’24	160	70.6	54.7	56.0	72.9	1226.0	86.7%
Ours	–	160	76.4	59.4	57.2	81.4	1457.0	94.9%