🎧 VoxZip

Official code for the ACM MM 2026 paper "VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference".

---

✨ Overview

Long-context audio inference on multimodal LLMs (e.g., Qwen3-Omni) is bottlenecked by the KV cache: audio frames dominate the sequence length, so memory and latency grow quickly with conversation duration. VoxZip compresses this audio-side KV cache along the temporal axis, anchored to ASR segment boundaries.

The idea is simple. Run Whisper ASR on each history audio to obtain segment-level timestamps — one [start, end] per ASR segment, not per-word timestamps. Each segment is aligned to its text-token span and audio-frame span, the audio and text embeddings are fused at segment granularity, and a temporal heavy-hitter policy then merges / delays redundant KV entries. The result is a shorter fused sequence that preserves semantic coverage while cutting cache size.

Why segment-level, not word-level? Per-word timestamp fusion aligns audio frames to individual tokens, which is noisy and underperforms. Segment-level fusion uses one timestamp per ASR segment — coarser, cleaner, and empirically stronger. The word-level variant was removed from this repo for that reason.

---

🔥 Highlights

• 🎯 Segment-anchored fusion — ASR segments (not words) define the alignment
• 🧠 Temporal KV compression — Heavy-hitter + time merge / delay (ours)
• 🧩 Drop-in on Qwen3-Omni — uses only the public transformers==4.57.2 API, no fork needed
• 📊 6 benchmarks — Vox-Infinity, IEMOCAP, AudioMarathon, MMAR, MMAU, MMSU, SPIRAL
• 🔇 Noise robustness — built-in SNR sweep via the *_noise drivers
• 🧹 Clean & English-commented — dead code removed, comments translated, env-var paths

---

🗂️ Project Structure

VoxZip-ACMMM2026/
├── kv_compression/                  # KV-cache methods
│   ├── ablation/                    #   ours: temporal_kv_merge / temporal_kv_delay / temporal_kv_delay_merge
│   ├── embedding_compression/       #   ours: ASR segment-fusion (EmbedAudio2Text, ...)
│   ├── snapkv/  pyramid/  streamingLLM/  segKV/  cam/  l2norm_cluster/   # baselines
├── utils/                           # custom_processor, custom_processor_embed, filter_word
├── eval/                            # scoring scripts (GPT-judge, exact-match, ABCD, ...)
├── scripts/                         # run_*.sh per benchmark
├── results/                         # output JSONL (created on run)
├── setup_env.sh                     # exports VOXZIP_* path env vars
├── requirements.txt
├── README.md
└── *.py                             # benchmark drivers (root, so `from kv_compression...` works)

---

🚀 Installation

Step 1: Create environment

conda create -n voxzip python=3.10 -y
conda activate voxzip

Step 2: Install dependencies

VoxZip uses only the public API of transformers 4.57.2 (it inherits from Qwen3OmniMoe* classes and rewrites the attention layer), so the upstream wheel works — no patched fork required.

cd VoxZip-ACMMM2026
pip install -r requirements.txt

Step 3: Configure paths

Edit setup_env.sh so the VOXZIP_* fallbacks point to your local copies of the model weights, Whisper weights, and datasets. Then source it:

source setup_env.sh

Every driver reads these variables via os.environ.get("VOXZIP_...", "<fallback>"), so on the original development host the defaults already work.

---

📦 Data & Weights (not bundled)

This repo ships code only. Download separately and point the env vars at them:

Asset	Source	Env var
Qwen3-Omni-30B-A3B-Instruct	Qwen/Qwen3-Omni-30B-A3B-Instruct	VOXZIP_MODEL_PATH
Whisper large-v3-turbo	openai/whisper-large-v3-turbo	VOXZIP_WHISPER_WEIGHT
Vox-Infinity (+QA)	vox-infinity/Vox-Infinity	VOXZIP_DATA_DIR
MMSU	ddwang2000/MMSU	VOXZIP_DATA_DIR
MMAR	BoJack/MMAR	VOXZIP_MMAR_DATA
MMAU-test-mini	gamma-lab-umd/MMAU-test-mini	VOXZIP_MMAU_DATA
SPIRAL	(see original spiral_datasets/)	VOXZIP_SPIRAL_DATA
AudioMarathon	(see original AudioMarathon/)	VOXZIP_AUDIOMARATHON_DATA

---

🎯 Running Experiments

All commands assume you have run source setup_env.sh. Pick GPUs with --visible_devices.

Part 1: Vox-Infinity (main benchmark)

VoxZip temporal component — KV-compression ablation across all baselines + ours:

bash scripts/run_vox_infinity.sh                       # all 4 splits, temporal_kv_merge (ours)
bash scripts/run_vox_infinity.sh Conversational snapkv # single split + a baseline

VoxZip full method — ASR segment fusion + temporal merge (the complete pipeline):

bash scripts/run_voxzip_full.sh Ultra-Multi-Turn-Dialogues

Part 2: IEMOCAP

bash scripts/run_iemocap.sh h2o_time_merge

Part 3: AudioMarathon

bash scripts/run_audiomarathon.sh seg RACE       # VoxZip full method
bash scripts/run_audiomarathon.sh fullkv GTZAN   # full-KV baseline
bash scripts/run_audiomarathon.sh baselines RACE # SnapKV / PyramidKV / ...

Part 4: MMAR / MMAU / MMSU / SPIRAL

Each benchmark has a kv_cache_* driver (baselines) and an audio2text_seg_* driver (VoxZip full method):

bash scripts/run_bench.sh mmar seg               # VoxZip full method on MMAR
bash scripts/run_bench.sh spiral kv_cache h2o    # H2O baseline on SPIRAL
bash scripts/run_bench.sh mmsu seg_noise         # + white-noise robustness (20 dB)

Part 5: Noise robustness

Any *_noise driver accepts --noise_snr_db:

python Ablation_test_multi_noise.py --temporal_kv_merge=True --noise_snr_db=20 ...

---

📊 Evaluation

# Generic GPT-4o-mini judge (Vox-Infinity, MMAR, MMAU, SPIRAL) — needs an OpenAI-compatible API key
bash scripts/run_eval.sh results/Vox/*_temporal_kv_merge.jsonl

# Benchmark-specific scorers
python eval/eval_iemocap.py            --input_file <iemocap.jsonl>     # per-emotion accuracy
python eval/evaluate_mmsu.py           --input_file <mmsu.jsonl>        # ABCD by category
python eval/evaluate_Audio_Marathon.py --input_file <audiomarathon.jsonl>
python eval/summary_results.py         --input_file <vox.jsonl>         # duration-bucketed accuracy
python eval/plot_niah.py                                                 # NIAH heatmap (paper figure)
python eval/statics.py                                                  # KV-length statistics

The GPT judge in eval/GetGPTScore_meituan_api.py calls an OpenAI-compatible endpoint; set the API key / base URL inside that file.

---

🔑 API Key (GPT judge)

# Linux / macOS
export OPENAI_API_KEY="your_api_key_here"
bash scripts/run_eval.sh results/Vox/*.jsonl

# Windows PowerShell
$env:OPENAI_API_KEY="your_api_key_here"
python eval/evalute_output.py --input_file results/Vox/test.jsonl

---

⚙️ Key configuration

Flag	Meaning	Typical value
--temporal_kv_merge	VoxZip temporal merge (ours)	True
--temporal_kv_delay	VoxZip temporal delay (ours)	True
--heavy_budget / --recent_budget	Temporal KV cache size	900 / 300 (Vox-Infinity)
--asr_thresold	Whisper word confidence filter	0.3
--embed_compression	enable ASR segment fusion	True
--noise_snr_db	white-noise SNR (noise drivers)	20 / 10 / 5
--visible_devices	GPU ids	"0,1"

Per-benchmark budgets differ — see each driver's --help.

---

🧪 Minimal usage

Run a single Vox-Infinity split with the VoxZip temporal method:

source setup_env.sh
python Ablation_test_multi.py \
  --datasets_content $VOXZIP_DATA_DIR/Vox-Infinity/Conversational \
  --datasets_qa      $VOXZIP_DATA_DIR/Vox-Infinity-QA/Conversational \
  --output_jsonl     $VOXZIP_OUTPUT_DIR/Vox/Conversational_temporal_kv_merge.jsonl \
  --visible_devices 0,1 \
  --temporal_kv_merge=True \
  --heavy_budget 900 --recent_budget 300

---

🧠 Methods at a glance

Component	What it does	Code
Semantic-Anchored	Whisper ASR → segment alignment → interleave fusion	audio2text_embedding_compression_seg.py, kv_compression/embedding_compression/
Temporal	H2O heavy-hitter + KV merge / delay	kv_compression/ablation/h2o_time_merge.py, temporal_kv_delay.py, temporal_kv_delay_merge.py
Baselines	H2O / SnapKV / PyramidKV / StreamingLLM / CAM / L2Norm / ChunkKV	kv_compression/{ablation/temporal_kv_base, snapkv, pyramid, streamingLLM, cam, l2norm_cluster, segKV}

D2O and TOVA baselines were removed per request.

---

📚 Citation

If this repository is helpful to your research, please cite:

@inproceedings{voxzip2026,
      title     = {VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference},
      author    = {Jia, Wenxu and Fu, Dongjie and Cheng, Xize and Feng, Fangming and Li, Linjun and Chen, Wenshi and Li, Yingming and Zhao, Zhou and Jin, Tao},
      booktitle = {Proceedings of the 34th ACM International Conference on Multimedia (ACM MM)},
      year      = {2026},
      doi       = {10.1145/3767308.3835719},
      eprint    = {2608.08569},
      archivePrefix = {arXiv},
      primaryClass  = {cs.AI},
}