OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models

May 19, 2026 · View on GitHub

Minseok Kang¹, Minhyeok Lee¹, Jungho Lee¹, Minjung Kim², Donghyeong Kim¹, Dayeon Lee¹, Heeseung Choi³, Ig-Jae Kim³, Sangyoun Lee^1†
¹Yonsei University ²LG Electronics ³KIST
_{^†Corresponding author}

Training-free, plug-and-play video token compression for Video-LLMs (Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video). Operates on vision-encoder features — no fine-tuning, no architectural changes.

✨ Highlights

Two-stage spatiotemporal compression — per-frame spatial selection followed by OT (Sinkhorn) cross-frame merging with content-adaptive budget allocation.
Strong-prune via cost threshold — matches with high OT cost are completely dropped (the absorbed token and its downstream subtree), giving robust compression at extreme retention ratios (r ≤ 0.10).
Training-free, plug-and-play — single function call apply_ottvid(model, ...) monkey-patches three backbones: Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video.

📋 Method

Two stages, both fixed-implementation:

Spatial Compression — Per-frame, select a token subset that best "covers" the frame in feature space, saliency-weighted.
Temporal (OT merge) — Sinkhorn-OT between consecutive frames' kept tokens.
- Cost: C = α · feat_dist + (1 − α) · cent_dist, α ∈ [0.5, 1.0] from same-position cosine similarity.
- Source mass: terminal Leave-One-Out (low mass = salient = preserved).
- Budget: softmax(−W / τ_b) redistributes merges to static pairs.
- Strong prune: discard absorbed tokens whose match cost exceeds θ.

📦 Installation

# 1. Install OTT-Vid package
git clone https://github.com/minseokii/OTT-Vid.git
cd OTT-Vid && pip install -e . && cd ..

# 2. (For benchmarks) install lmms-eval + drop in OTT-Vid wrappers
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
cd lmms-eval && pip install -e . && cd ..
cp OTT-Vid/lmms_eval_wrappers/qwen2_5_vl.py      lmms-eval/lmms_eval/models/simple/qwen2_5_vl.py
cp OTT-Vid/lmms_eval_wrappers/llava_onevision.py lmms-eval/lmms_eval/models/simple/llava_onevision.py
cp OTT-Vid/lmms_eval_wrappers/llava_vid.py       lmms-eval/lmms_eval/models/simple/llava_vid.py

# 3. Qwen-specific helper (only for Qwen2.5-VL backbone)
pip install qwen-vl-utils

# 4. Flash-attention (optional but recommended for Qwen2.5-VL)
pip install flash-attn --no-build-isolation

Tested versions: PyTorch 2.7.0+cu128, transformers 4.57.3, flash-attn 2.8.3, decord 0.6.0, qwen-vl-utils 0.0.14. See docs/installation.md for details.

🚀 Quickstart

OTT-Vid is plug-and-play — wrap the model once with apply_ottvid().

Qwen2.5-VL

import torch
from transformers import AutoModelForImageTextToText
from ottvid import apply_ottvid

model = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
).eval()

model = apply_ottvid(
    model,
    total_retention_ratio=0.10,    # final 10 % token survival
    temporal_share=0.3,            # γ — splits r into r_s = r**(1-γ), r_t = r**γ
    enable_temporal=True,
    ot_mass_tau=0.3,
    ot_budget_temperature=0.3,
    ot_cost_threshold=0.3,
    ot_sinkhorn_eps=0.01,
    ot_sinkhorn_iters=200,
)
# Use `model.generate(...)` as usual.

LLaVA-OneVision / LLaVA-Video

from ottvid import apply_ottvid_llava_vid

model = apply_ottvid_llava_vid(
    model,
    total_retention_ratio=0.10,
    temporal_share=0.3,
    enable_temporal=True,
    ot_mass_tau=0.3,
    ot_budget_temperature=0.3,
    ot_cost_threshold=0.3,
)

A complete single-video example is in examples/inference_qwen.py:

python examples/inference_qwen.py \
    --model /path/to/Qwen2.5-VL-7B-Instruct \
    --video /path/to/video.mp4 \
    --question "Describe this video." \
    --total_retention 0.10

🤖 Supported Models

Backbone	HF Model card
Qwen2.5-VL-7B-Instruct	`Qwen/Qwen2.5-VL-7B-Instruct`
LLaVA-OneVision-7B	`lmms-lab/llava-onevision-qwen2-7b-ov`
LLaVA-Video-7B	`lmms-lab/LLaVA-Video-7B-Qwen2`

📚 Supported Datasets

All datasets are pulled by lmms-eval at first run.

Benchmark	Task	HF Dataset
MVBench	VQA (4-way MC, 20 subtasks)	`OpenGVLab/MVBench`
Video-MME	VQA (4-way MC)	`lmms-lab/Video-MME`
MLVU (dev)	VQA (4-way MC)	`MLVU/MVLU`
LongVideoBench	VQA (5-way MC)	`longvideobench/LongVideoBench`
Charades-TimeLens	VTG (mIoU)	`TencentARC/TimeLens-Bench`
ActivityNet-TimeLens	VTG (mIoU)	`TencentARC/TimeLens-Bench`

📊 Evaluation

We use LMMs-Eval. Sample launchers are in scripts/.

Qwen2.5-VL on MVBench

export MODEL=/path/to/Qwen2.5-VL-7B-Instruct
bash scripts/eval_qwen_mvbench.sh

Default config (paper, r=0.10)

total_retention_ratio = 0.10
temporal_share        = 0.3   → spatial_retention ≈ 0.20
ot_mass_tau           = 0.3
ot_budget_temperature = 0.3
ot_cost_threshold     = 0.3
ot_sinkhorn_eps       = 0.01
ot_sinkhorn_iters     = 200

👏 Acknowledgement

This project builds upon excellent open-source efforts: FastV, VisionZip, PruneVID, FastVID, FlashVID, HoliTom, UniComp, LLaVA-NeXT, Qwen2.5-VL, LMMs-Eval. Thanks for their excellent work!

📜 Citation

@article{kang2026ott,
  title   = {OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models},
  author  = {Kang, Minseok and Lee, Minhyeok and Lee, Jungho and Kim, Minjung and Kim, Donghyeong and Lee, Dayeon and Choi, Heeseung and Kim, Ig-jae and Lee, Sangyoun},
  journal = {arXiv preprint arXiv:2605.11803},
  year    = {2026}
}

License

MIT (see LICENSE).