[ICLR 2026] AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

August 9, 2026 · View on GitHub

Changwoo Baek^*, Jouwon Song^*, Sohyeon Kim^*, Kyeongbo Kong^†

^*Equal contribution, ^†Corresponding author

🎉 News

[2026/01] 🔥 Our paper has been accepted to ICLR 2026! 🎊
[2026/02] 🚀 Project page is now live!

Large Vision-Language Models (LVLMs) have adopted visual token pruning strategies to mitigate substantial computational overhead incurred by extensive visual token sequences. While prior works primarily focus on either attention-based or diversity-based pruning methods, in-depth analysis of these approaches' characteristics and limitations remains largely unexplored.

In this work, we conduct thorough empirical analysis using effective rank (erank) as a measure of feature diversity and attention score entropy to investigate visual token processing mechanisms and analyze the strengths and weaknesses of each approach.

🔍 Key Findings

Our analysis reveals two key insights:

Diversity aware hybrid pruning methods preserve less feature diversity than intended, and the diversity they do retain is closely tied to increased hallucination frequency compared to attention-based pruning.

Key Findings

Attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features.

Key Findings

Building on these empirical insights, we show that incorporating image-aware adjustments into existing hybrid pruning strategies consistently improves their performance. We also provide a minimal instantiation of our empirical findings through a simple adaptive pruning mechanism.

💻 Code

This repository contains the full training-free implementation on top of LLaVA. The pruning method itself — effective rank (effective_rank), the adaptive threshold rule (calculate_adaptive_tau, Eq. 6 in the paper), and the token-selection procedure (select_diverse_tokens_by_attention_and_distance) — lives in llava/model/llava_arch.py.

🏝️ Environment

git clone https://github.com/cvsp-lab/AgilePruner.git
cd AgilePruner
conda create -n agilepruner python=3.10 -y
conda activate agilepruner
pip install -e .

(Optional) Install FlashAttention for further inference acceleration:

pip install flash-attn --no-build-isolation

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/${DATASET}.sh ${VISUAL_TOKEN_NUMBER}

For example, to keep 64 of the 576 visual tokens (89% reduction) on POPE:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh 64

Full per-benchmark setup and submission instructions are in EVAL.md.

🔖 Citation

If you find AgilePruner useful for your research, please cite:

@inproceedings{baek2026agilepruner,
      title={AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models},
      author={Baek, Changwoo and Song, Jouwon and Kim, Sohyeon and Kong, Kyeongbo},
      booktitle={International Conference on Learning Representations (ICLR)},
      year={2026},
      eprint={2603.01236},
      archivePrefix={arXiv},
}