CROP

Official LLaVA implementation of CROP: Contextual Region-Oriented Visual Token Pruning (EMNLP 2025).

This repository contains the two CROP compression strategies used in the paper:

• PLC (Pre-LLM Compression): compresses visual tokens before the LLM with learnable regional and contextual queries.
• ILP (Inner-LLM Pruning): uses localized regional indices to prune non-regional visual tokens from an early LLM layer.

The localization model is intentionally not included here. Evaluation and training files must provide a bounding box for each image-question pair.

Repository layout

crop/
  regions.py                 # bbox-to-token mapping
  model/
    plc.py                   # PLC compression module
    ilp.py                   # ILP mask/drop utilities
    language_model/
      crop_llama.py          # unified CROP LLaVA model
    builder.py               # checkpoint loader
  eval/model_vqa_loader.py   # bbox-guided evaluation
  train.py                   # PLC training entry
llava/                       # minimal LLaVA-1.5 backbone
scripts/crop/                # reproducible commands
tests/                       # CROP unit tests

Installation

pip install -e ".[train]"

The implementation targets LLaVA-1.5 with CLIP ViT-L/14 visual tokens and transformers==4.43.1.

Region format

Bounding boxes use pixel coordinates:

{
  "question_id": "example-1",
  "image": "000000.jpg",
  "text": "What is the person holding?",
  "gtbbox_small": [120, 64, 310, 288],
  "image_size": [640, 480]
}

Coordinates follow [x1, y1, x2, y2]; image sizes follow [width, height].

PLC training

PLC uses 64 regional output tokens and 4 contextual output tokens by default.

MODEL_PATH=/path/to/llava-v1.5-7b \
DATA_PATH=/path/to/train.json \
IMAGE_FOLDER=/path/to/images \
OUTPUT_DIR=/path/to/crop-plc \
bash scripts/crop/train_plc.sh

Each training sample must contain bbox (or gtbbox_small) in addition to standard LLaVA conversation fields.

Evaluation

MODEL_PATH=/path/to/crop-plc \
QUESTION_FILE=/path/to/questions.jsonl \
IMAGE_FOLDER=/path/to/images \
ANSWERS_FILE=/path/to/plc_answers.jsonl \
bash scripts/crop/eval_plc.sh

ILP is training-free and can be applied directly to a LLaVA-1.5 checkpoint:

MODEL_PATH=/path/to/llava-v1.5-7b \
QUESTION_FILE=/path/to/questions.jsonl \
IMAGE_FOLDER=/path/to/images \
ANSWERS_FILE=/path/to/ilp_answers.jsonl \
bash scripts/crop/eval_ilp.sh

ilp_mode=mask reproduces pruning behavior while retaining KV-cache compatibility. ilp_mode=drop physically shortens the sequence for cache-free prefill/latency measurement and cannot be combined with KV cache.

Citation

@inproceedings{guo-etal-2025-crop,
  title = "{CROP}: Contextual Region-Oriented Visual Token Pruning",
  author = "Guo, Jiawei and Zhai, Feifei and Jian, Pu and Wei, Qianrun and Zhou, Yu",
  booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
  year = "2025",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2025.emnlp-main.492/",
  doi = "10.18653/v1/2025.emnlp-main.492",
  pages = "9756--9772"
}

Acknowledgements

The LLaVA backbone is derived from the official LLaVA project. The original Apache-2.0 license is retained.