Yahong Wang*¹, Juncheng Wu*^2,3, Zhangkai Ni^1✉, Longzhen Yang¹, Yihang Liu¹,
Chengmei Yang¹, Ying Wen⁴, Lianghua He^1,5✉, Xianfeng Tang, Hui Liu³, Yuyin Zhou³,

¹Tongji University, ²University of California, Santa Cruz, ³Amazon,
⁴East China Normal University, ⁵Shanghai Eye Disease Prevention and Treatment Center

📢 News

• 2026.05.27 Code is available!
• 2026.02.21 Our paper is accepted at CVPR 2026 main!
• 2025.12.08 Our paper is released!

Star 🌟 us if you think it is helpful!!

⚡Introduction

TLDR: By quantifying a visual token's information content(measured by the change in model output probabilities upon its removal), We identify an "information horizon" in Vision Large Language Models, beyond which deep-layer tokens lose salience and become redundant. Crucially, this horizon is dynamic: it extends deeper for visually intensive tasks (e.g., OCR vs. simple VQA) and strongly correlates with higher model capacity. Based on these findings, applying simple random pruning beyond this horizon outperforms complex training-free methods.

🛠 Preparation

LLaVA

1. Clone this repository.

git clone https://github.com/YahongWang1/Information-Horizon
cd Information-Horizon

2. Environment Setup.

 conda create -n Info python=3.10 -y
 conda activate Info
 pip install -e .
 pip install flash_attn==2.5.9.post1 --no-build-isolation

3. Download Benchmark.

Please follow instructions in LLaVA-Evaluation.

🎯 Usage

LLaVA

📖 Calculate Token Information (Figure 4)

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/cal_info/mme.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/cal_info/textvqa.sh

✂️ Prune with Token Information (Figure 6)

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/info_prune/mme.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/info_prune/textvqa_gt.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/info_prune/textvqa.sh

textvqa_gt.sh analyzes TextVQA ground-truth answers to calculate per-token information across all samples.

🎲 Random Pruning with DART/DivPrune (Table 1)

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/allbench_dart_random.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/allbench_dart_vtw.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/allbench_divprune_random.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/allbench_divprune_vtw.sh

📌 Citation

@misc{wang2026tokenpruningworserandom,
      title={When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs}, 
      author={Yahong Wang and Juncheng Wu and Zhangkai Ni and Longzhen Yang and Yihang Liu and Chengmei Yang and Ying Wen and Lianghua He and Xianfeng Tang and Hui Liu and Yuyin Zhou},
      year={2026},
      eprint={2512.07580},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.07580}, 
}

❤️ Acknowledgment

Thanks to the open-source contributions of LLaVA, DivPrune, and DART.

📭 Contact

For any questions about our paper or code, please email yahongwang@tongji.edu.cn.