README.md

May 27, 2026 · View on GitHub

🧷 Variation-aware Vision Token Dropping for Faster Large Vision-Language Models 🚀

Junjie Chen1*, Xuyang Liu1*,†, Zichen Wen2, Yiyu Wang2, Siteng Huang3, Honggang Chen1✉

1Sichuan University, 2EPIC Lab, Shanghai Jiao Tong University, 3Zhejiang University

🔥 News

  • 2026.03.15 💻💻 Our Code of Qwen2-VL is available! This work also references DRAT, thanks for their contributions.
  • 2026.02.21 🎊🎊 Our V2Drop has been accepted by CVPR 2026!
  • 2025.08.27 🤗🤗 We release our latest work V2Drop, a variation-aware vision token dropping method for plug-and-play inference LVLM acceleration. Code is available!

TLDR: Token-wise variation intuitively reflects vision token importance (green boxes) while maintaining compatibility with efficient operators. Thus, we present V2Drop, a plug-and-play framework that measures token-wise variation across adjacent LLM layers and progressively drops vision tokens with minimal variation, thereby achieving plug-and-play inference acceleration.

💥 Core Codes

The core implementation of our code is in llava/model/language_model/V2Drop.py.

🛠 Preparation

LLaVA

  1. Clone this repository.
git clone https://github.com/xuyang-liu16/V2Drop
cd V2Drop
  1. Environment Setup and Preparation
 conda create -n V2Drop python=3.10 -y
 conda activate V2Drop
 pip install -e .
 pip install flash-attn --no-build-isolation
  1. Go to your transformers/src/transformers/models/llama/modeling_llama.py in the conda envs, change all three
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)

with:

cos, sin = self.rotary_emb(value_states, seq_len=position_ids.max().item() + 1)
  1. Download Multimodal Benchmark

Please follow the detailed instruction in LLaVA-Evaluation.

  1. Download LLaVA-1.5-7B and put them under ./liuhaotian/llava-v1.5-7b.

For users with limited access to Hugging Face (e.g., from mainland China), you can refer to this you can refer this alternative guide and use the following command, with LLaVA-1.5-7B as an example:

pip install -U huggingface_hub hf_transfer -i https://mirrors.aliyun.com/pypi/simple/
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download liuhaotian/llava-v1.5-7b --local-dir ./liuhaotian/llava-v1.5-7b

🚀 Evaluation

Example for evaluating TextVQA results:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh

Example for evaluating MME results:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

📌 Citation

Please consider citing our paper in your publications, if our findings help your research.

@misc{chen2025variationawarevisiontokendropping,
      title={Variation-aware Vision Token Dropping for Faster Large Vision-Language Models}, 
      author={Junjie Chen and Xuyang Liu and Zichen Wen and Yiyu Wang and Siteng Huang and Honggang Chen},
      year={2025},
      eprint={2509.01552},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.01552}, 
}