---

📢 News

• [2026/07/12] 🚀 Released SEATS code for Qwen3-Omni-30B, with baselines and evaluation scripts.
• [2026/06/07] 🚀 Released SEATS code for Qwen2.5-Omni-7B, with LMMs-Eval integration and baselines including VisionZip, FastV, OmniZip, and their omni-modal variants.
• [2026/05/19] 📄 Paper released on arXiv and project page is online.

👀 Overview

SEATS is a training-free, stage-adaptive token selection method for efficient omni-modal LLM inference. By analyzing layer-wise token dependency, it reveals that visual and audio dependencies follow a block-wise pattern and weaken with depth. SEATS removes spatiotemporal redundancy before the LLM, progressively prunes tokens inside the LLM, and fully removes non-textual tokens in late layers.

✨ Key Highlights

• 💡 New Insight: Reveals a block-wise dependence pattern in omni-modal LLMs, where reliance on visual and audio tokens weakens with layer depth.
• ⚡ Strong Efficiency: 9.3x FLOPs reduction and 4.8x prefill speedup at 10% token retention while preserving 96.3% performance.
• 🎯 Stage-adaptive Design: Diversity-based pre-LLM selection + query-guided inner-LLM progressive pruning + late-layer full removal.
• 🔌 Broad Compatibility: Plug-and-play and training-free for direct application to Qwen2.5-Omni-7B and Qwen3-Omni-30B.

📅 TODO

• Support Qwen2.5-Omni-7B
• Release benchmark adaptation code for LMMs-Eval (WorldSense, Daily-Omni, OmniVideoBench, Video-MME, LVOmniBench)
• Evaluation scripts and reproduction guide (adapted for LMMs-Eval)
• Release more baseline implementations (FastV, VisionZip, Random)
• Support Qwen3-Omni-30B
• Release more baseline implementations (OmniZip)
• future work: Support more models (OmniVinci-7B)

🏗️ Method

SEATS is a three-stage method:

1. Pre-LLM Token Selection: Removes spatiotemporal redundancy within each temporal window via attention-weighted diversity selection.
2. Inner-LLM Token Selection: Progressively prunes tokens with a block-wise token retention ratio decay schedule and top-down budget allocation (inter-window then intra-window) guided by query relevance.
3. Late-block Removal: Removes all remaining non-textual tokens in late layers where cross-modal fusion is complete.

🔧 Dependencies and Installation

We used Anaconda to setup a deep learning workspace that supports PyTorch. Run the following script to install all the required packages.

# git clone this repository
git clone https://github.com/xxayt/SEATS.git
cd SEATS

# create a new anaconda env
conda create -n SEATS_env python=3.10 -y
conda activate SEATS_env

# install dependencies
bash scripts/base/setup.sh

# install the bundled lmms-eval in editable mode
cd lmms-eval
pip install -e .
cd ..

# (Recommended) install torch and flash-attn
# pip install torch==2.8.0 torchvision==0.23.0
pip install flash-attn --no-build-isolation

🚀 Evaluation

We adapt 5 omni-modal benchmarks into LMMs-Eval, so you can run them directly through this repository. Please first download the corresponding annotation data and videos from the links below.

Benchmark	Data	Videos	Task name
Daily-Omni	xxayt/Daily-Omni	liarliar/Daily-Omni	dailyomni
WorldSense	lmms-lab/WorldSense	lmms-lab/WorldSense	worldsense
OmniVideoBench	NJU-LINK/OmniVideoBench	NJU-LINK/OmniVideoBench	omnivideobench
Video-MME	lmms-lab/Video-MME	lmms-lab/Video-MME	videomme
LVOmniBench	xxayt/LVOmniBench	KD-TAO/LVOmniBench	lvomnibench

Once the data is ready, launch evaluation with the scripts under scripts/. Results are written to output/. We implement qwen2_5_omni_zip and qwen3_omni_zip as unified LMMs-Eval model wrappers that dispatch to SEATS and all baselines for omni-modal LLM token compression.

Full tokens

bash scripts/eval_qwen2_5_omni_full_tokens.sh  # Qwen2.5-Omni-7B
bash scripts/eval_qwen3_omni_full_tokens.sh    # Qwen3-Omni-30B

SEATS (our method)

To evaluate our SEATS method on the five benchmarks, use the following command:

bash scripts/eval_qwen2_5_omni_seats.sh  # Qwen2.5-Omni-7B
bash scripts/eval_qwen3_omni_seats.sh    # Qwen3-Omni-30B

You can customize the compression settings by editing:

• scripts/eval_qwen2_5_omni_seats.sh — tasks_list (which benchmarks to run) and ratio_pairs (per-modality token retention budgets, swept over multiple settings).
• seats/config.yaml — SEATS method hyperparameters (e.g., progressive drop layers, late-block layer, window size).

Qwen3-Omni-30B scripts follow the same pattern (scripts/eval_qwen3_omni_seats.sh + seats/config_qwen3_30b.yaml).

Baselines

We also provide the following scripts to evaluate the baseline methods adapted for omni-modal LLMs:

Method	Qwen2.5-Omni-7B	Qwen3-Omni-30B
Random	scripts/eval_qwen2_5_omni_random.sh	scripts/eval_qwen3_omni_random.sh
FastV	scripts/eval_qwen2_5_omni_fastv.sh	scripts/eval_qwen3_omni_fastv.sh
FastV-om	scripts/eval_qwen2_5_omni_fastv_omni.sh	scripts/eval_qwen3_omni_fastv_omni.sh
VisionZip	scripts/eval_qwen2_5_omni_visionzip.sh	scripts/eval_qwen3_omni_visionzip.sh
VisionZip-om	scripts/eval_qwen2_5_omni_visionzip_omni.sh	scripts/eval_qwen3_omni_visionzip_omni.sh
OmniZip	scripts/eval_qwen2_5_omni_omnizip.sh	scripts/eval_qwen3_omni_omnizip.sh

FLOPs Profiling

To compute the average LLM prefill TFLOPs per sample, add the following environment variable to your evaluation script:

export COST_ANALYSE=1

📁 Repo Structure

SEATS/
├── scripts/                          # Shell entry points (one per method) + shared base
│   ├── base/
│   │   ├── setup.sh                  # Python dependency installation
│   │   ├── eval_qwen2_5_omni_zip.sh  # Shared accelerate + lmms-eval launcher (Qwen2.5-Omni)
│   │   └── eval_qwen3_omni_zip.sh    # Shared accelerate + lmms-eval launcher (Qwen3-Omni)
│   ├── eval_qwen2_5_omni_seats.sh    # SEATS (our method, Qwen2.5-Omni)
│   ├── eval_qwen3_omni_seats.sh      # SEATS (our method, Qwen3-Omni)
│   └── ...
├── seats/                            # SEATS three-stage implementation
│   ├── pre_llm_units.py              # Stage I: winDivPrune
│   ├── inner_llm_units.py            # Stage II: inner-LLM stage-adaptive selection
│   ├── ratio_decay_scheduler.py      # block-wise TRR decay schedule
│   ├── modeling_qwen2_5_omni_seats.py # patched Thinker / TextModel forwards (Qwen2.5-Omni)
│   ├── modeling_qwen3_omni_seats.py  # patched Thinker / TextModel forwards (Qwen3-Omni)
│   └── config.yaml                   # SEATS hyperparameters
├── baselines/                        # Per-method patches; one subfolder per baseline
│   ├── utils.py                      # apply_zip_method_patch() dispatcher
│   ├── cost_metrics.py               # LLM prefill FLOPs estimation (theoretical)
│   ├── full_tokens/                  # No compression (config only)
│   ├── visionzip_omni/               # VisionZip adapted for omni-modal
│   └── ...
├── models/
│   ├── qwen2_5_omni/                 # Vendored Qwen2.5-Omni model code
│   └── qwen3_omni_moe/              # Vendored Qwen3-Omni model code
└── lmms-eval/                        # Vendored LMMs-Eval (registers `qwen2_5_omni_zip`)

🤝 Acknowledgement

This implementation relies on resources from Qwen2.5-Omni, Qwen3-Omni, LMMs-Eval, OmniZip, VisionZip, and DivPrune. We thank the original authors for their excellent contributions and for making their work publicly available.

✏️ Citation

If you find this work useful, please consider citing:

@article{xin2026seats,
  title={Stage-adaptive Token Selection for Efficient Omni-modal LLMs},
  author={Xin, Zijie and Yang, Jie and Zhao, Ruixiang and Wang, Tianyi and Rao, Fengyun and Lyu, Jing and Li, Xirong},
  journal={arXiv preprint arXiv:2605.20035},
  year={2026}
}

📜 License

This project is licensed under the MIT License. For commercial licensing or any use beyond research, please contact the authors.

📬 Contact for Issues

For any questions about this project (e.g., corrupted files or loading errors), please reach out at: xinzijie@ruc.edu.cn