HybridToken-VLM: Hybrid Token Compression for Vision-Language Models

CVPR 2026

Jusheng Zhang, Xiaoyang Guo, Kaitong Cai, Qinhan Lv, Yijia Fan, Wenhao Chai, Jian Wang, Keze Wang

TL;DR

HTC-VLM is a hybrid token compression method that disentangles semantic anchors (discrete MGVQ tokens) from appearance (continuous ViT patch features) and compresses their hybrid representation into a single <voco> token. By preserving both high-level semantics and fine-grained details, HTC-VLM reduces token and compute costs while maintaining grounded visual understanding. It provides a practical path to scale vision–language models.

Preparation

Install

1. Clone this repository and navigate to HTC-VLM folder

git clone https://xxxx/HTC-VLM.git
cd HTC-VLM

2. Install Package

conda create -n htc_vlm python=3.10 -y
conda activate htc_vlm
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation
cp HTC-VLM/llava/model/language_model/cache_py/modeling_attn_mask_utils.py /data/miniconda3/envs/htc_vlm/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py

Data and Pre-trained weights

HTC-VLM only requires visual instruction fine-tuning. Download the compatible LLaVA checkpoints (base LLM and projection layers) from the LLaVA model zoo: MODEL_ZOO.md. In addition, obtain the instruction-tuning annotations file llava_v1_5_mix665k.json and the images from the datasets listed below.

Also download the MGVQ checkpoint mgvq_f16c32_g8.pt and place it at llava/model/MGVQ/checkpoints/ .

• COCO: train2017
• GQA: images
• OCR-VQA: download script, we save all files as .jpg
• TextVQA: train_val_images
• VisualGenome: part1, part2

After downloading all of them, organize the data as follows in ./playground/data,

├── coco
│   └── train2017
├── gqa
│   └── images
├── ocr_vqa
│   └── images
├── textvqa
│   └── train_images
└── vg
    ├── VG_100K
    └── VG_100K_2

Train

HTC-VLM is trained on 8 A100 GPUs (80GB each). If you have fewer GPUs available, decrease the per_device_train_batch_size and compensate by raising gradient_accumulation_steps so that the effective global batch size stays constant: per_device_train_batch_size × gradient_accumulation_steps × num_gpus.

To run vision instruction tuning on your setup, execute:

bash scripts/finetune.sh

Evaluation

We evaluate visual understanding using the same protocols as LLaVA. For complete instructions on dataset preparation and testing procedures, see the LLaVA repository: Evaluation.md.

Citation

If you find this work useful, please consider citing our paper:

@misc{zhang2025hybridtokenvlmhybridtokencompression,
      title={HybridToken-VLM: Hybrid Token Compression for Vision-Language Models}, 
      author={Jusheng Zhang and Xiaoyang Guo and Kaitong Cai and Qinhan Lv and Yijia Fan and Wenhao Chai and Jian Wang and Keze Wang},
      year={2025},
      eprint={2512.08240},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.08240}, 
}

Acknowledgement

• LLaVA, VoCo-LLaMA: the projects we built upon and adapted from.
• Vicuna: our base model (Vicuna‑7B), which provides strong language capabilities.