🚀 Accelerating Streaming Video LLMs via Hierarchical Token Compression

Yiyu Wang^1*, Xuyang Liu^1,2*†, Xiyan Gui^1,3, Xinying Lin⁴, Boxue Yang¹,
Chenfei Liao^1,5, Tailai Chen¹, Linfeng Zhang<sup>1✉

1</sup> EPIC Lab, SJTU   ² Sichuan University   ³ HUST   ⁴ SYSU   ⁵ HKUST (Guangzhou)

⚡ The first plug-and-play token-compression framework for streaming video understanding.

↓24.5% ViT encoding latency  ·  ↓45.3% LLM pre-filling latency  ·  up to 99% accuracy retained
_{on the ReKV framework — see Results}

Highlights · Method · Supported Frameworks · Results · Installation & Reproduction · Evaluation · Latency Benchmark · Citation

🔥 News

• 2026.06.04  🚀 Added a runtime latency benchmark (speed_benchmark/), add support for StreamForest & Dispider.
• 2026.06.03  🧱 Refactored the codebase into a standalone stc Python package — STC-Cacher, STC-Pruner, HF ViT integrations in a clean layout.
• 2026.02.21  🎊 STC is accepted by CVPR 2026!
• 2025.12.02  🤗 We release STC, the first plug-and-play inference-acceleration framework for streaming video understanding.

✨ Highlights

🧠 Method

STC compresses visual tokens hierarchically: STC-Cacher acts within the ViT (reusing static tokens, recomputing only the dynamic ones across frames), and STC-Pruner acts before the LLM (keeping the most salient tokens to shorten pre-fill).

STC-Cacher exploits temporal change: tokens that barely change between frames are reused, and only the dynamic regions are re-encoded.

🧩 Supported Frameworks

STC's core package is framework-agnostic. STC-Cacher attaches to any HuggingFace pre-LN CLIP / SigLIP vision tower via a one-line monkey-patch; STC-Pruner is an explicit call before LLM pre-fill.

📊 Results

TL;DR — On ReKV, STC retains up to 99% of accuracy while cutting ViT encoding latency by 24.5% and LLM pre-filling latency by 45.3%, surpassing VidCom² by 1.6 on both OVO-Bench and StreamingBench (and ToMe by 5.6 / 5.8). Evaluated on 4 streaming VideoLLM baselines × 5 benchmarks at a 0.5 fps streaming protocol. Latencies are in seconds (ViT: encode 16 frames; LLM: pre-fill time).

Streaming benchmarks — ReKV (LLaVA-OV-7B)

STC-Cacher generality — OVO-Bench, baseline → + STC-Cacher

Offline long-video understanding — ReKV

_{See the paper for full results, per-subset VideoMME breakdowns, and ablations.}

🛠 Installation

pip install -e .            # core package (requires torch)
pip install -e .[hf]        # + transformers, for HF CLIP / SigLIP integrations

Reproducing the baseline frameworks

To make the vendored frameworks easy to reproduce, we made small, documented modifications to each upstream repo (path handling, model discovery, launch scripts, benchmark entrypoints). For every framework we ship a doc pair — a REPRODUCE guide (fresh GPU machine → environment → smoke run) and a CHANGES doc (exactly what we adapted, with the upstream git diff). Just follow the guide for the framework you want to run.

🧪 Evaluation

Copy a block, replace the /path/to/... placeholders with your own, and run. Outputs land under results/. Each block runs + STC; for the baseline, change the mode arg (rekv_stc→rekv, sf_stc→sf, dispider_stc→dispider).

ReKV (Baseline)

# Offline benchmark (dataset: mlvu / egoschema / videomme / ...)
export STC_PATCH_VISION=1 STC_TOKEN_PER_FRAME=64 STC_UPDATE_TOKEN_RATIO=0.25
bash scripts/eval_rekv/eval_offline_benchs.sh \
  --dataset mlvu --model llava_ov_7b --save_dir results/mlvu_stc

# OVO-Bench
export ANNO_PATH=/path/to/ovo_bench_new.json
export VIDEO_DIR=/path/to/src_videos
export CHUNKED_DIR=/path/to/chunked_videos
export STC_PATCH_VISION=1 STC_TOKEN_PER_FRAME=64 STC_UPDATE_TOKEN_RATIO=0.25
bash scripts/eval_rekv/ovobench_scripts/eval_rekv.sh
bash scripts/eval_rekv/ovobench_scripts/score_rekv.sh

StreamForest (Baseline)

export STREAMFOREST_CKPT_PATH=/path/to/StreamForest-Qwen2-7B
TASKS=ovobench bash scripts/eval_streamforest/eval_streamforest.sh sf_stc

Dispider (Baseline)

export MODEL_PATH=/path/to/Dispider
export CLIP_CKPT_PATH=/path/to/clip-vit-large-patch14
export ANNO_PATH=/path/to/ovo_bench_new.json
export CHUNKED_DIR=/path/to/chunked_videos
bash scripts/eval_dispider/eval_dispider_ovobench.sh dispider_stc

export STC_PATCH_VISION=1          # enable STC-Cacher (0 = baseline)
export STC_TOKEN_PER_FRAME=64      # STC-Pruner token budget per frame (196 = full; ReKV only)
export STC_UPDATE_TOKEN_RATIO=0.25 # STC-Cacher selective-recompute ratio
export STC_CACHE_INTERVAL=4        # full reference frame every N frames

⏱️ Latency Benchmark

Measure the latency reduction (baseline vs +STC) on your GPU — runtime-instrumented, no code changes, 16-frame default. One script per framework under speed_benchmark/:

GPU=0 bash speed_benchmark/run_rekv.sh          # ReKV: ViT encoding + LLM pre-fill
GPU=0 bash speed_benchmark/run_streamforest.sh  # StreamForest (SigLIP): ViT encoding
GPU=0 bash speed_benchmark/run_dispider.sh      # Dispider (CLIP): ViT encoding

Each runs the baseline and +STC configurations and prints the reduction. Pin a dedicated GPU and read min / median over repeats — absolute latency is GPU-dependent, the reduction ratio is what reproduces. See speed_benchmark/README.md for options and methodology.

🏗️ Architecture

The codebase is organized around the standalone stc package; each framework consumes it through its own model wrappers / eval drivers.

STC/
├── stc/                      Standalone Python package
│   ├── config.py                 Dataclasses + env-driven config
│   ├── core/                     Shared algorithms (token similarity, layer ratios)
│   ├── cacher/                   STC-Cacher
│   │   ├── state.py                  Per-stream cache state
│   │   ├── reference_forward.py      Selective ViT-layer forwards
│   │   └── graph.py                  CUDA-graph runner for selective frames
│   ├── pruner/                   STC-Pruner (scoring, anchors, index mapping, specs)
│   └── integrations/
│       ├── hf_vit.py                 register_stc_cacher() for HF CLIP / SigLIP
│       └── streaming.py              enable_streaming_cacher() — per-frame wrapper
├── models/                   Framework code + entrypoints (rekv, StreamForest, Dispider, livecc)
├── speed_benchmark/          Latency harness (benchmark.py + run.sh)
├── scripts/eval_rekv/        ReKV evaluation drivers
└── results/                  Run outputs (git-ignored)

How the two components attach:

• STC-Cacher is a monkey-patch: register_stc_cacher() swaps each vision_model.encoder.layers[*].forward for a selective-recompute forward. For batched towers, enable_streaming_cacher() additionally splits a clip into per-frame calls so the cache advances frame-by-frame (reset once per video with reset_streaming_cacher()).
• STC-Pruner is an explicit call: STCPruner().compress(...) runs after vision encoding / projection / pooling and before LLM pre-fill.

🙏 Acknowledgements

Built on the excellent work of ReKV, StreamForest, Dispider, and LiveCC — thanks to all authors for releasing their code.

✏️ Citation

If STC helps your research, please consider citing:

@inproceedings{wang2026stc,
  title     = {Accelerating Streaming Video Large Language Models via Hierarchical Token Compression},
  author    = {Wang, Yiyu and Liu, Xuyang and Gui, Xiyan and Lin, Xinying and
               Yang, Boxue and Liao, Chenfei and Chen, Tailai and Zhang, Linfeng},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

📮 Contact

Questions about the paper or code? Email liuxuyang@stu.scu.edu.cn or ustywan8@ljmu.ac.uk.