Updates

• [2026.07.07]:🚀 We release the efficiency benchmark suite (decode latency, end-to-end throughput, and SVD overhead).
• [2026.06.15]:🎉 xKV accepted at ICML 2026! We release the code of xK-SR and xKV-SR.
• [2025.03.24]:🚀 We release the 1st version of arXiv and code of xKV

TL;DR

We introduce xKV, a simple yet effective post-training KV-Cache compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, leveraging the well-aligned dominant singular vectors across layers. xKV achieves up to 8× KV-Cache compression while maintaining accuracy on long-context tasks. Combined with Selective Reconstruction (SR) at decode time, xKV-SR achieves up to 4.23× end-to-end speedup over standard attention and 30% higher throughput over strong baselines at similar accuracy.

Environment Setup

1. Clone the repository (Make sure you have Git installed on your system)

git clone https://github.com/abdelfattah-lab/xKV.git
cd xKV

2. Prepare environment To run the code in this project, first, create a Python virtual environment using e.g. uv. To install uv, follow the UV Installation Guide.

uv venv --python 3.11 && source .venv/bin/activate && uv pip install --upgrade pip

Next, install dependencies

git submodule update --init --recursive
uv pip install -r requirements.txt
uv pip install flash-attn==2.7.4.post1 --no-build-isolation
uv pip install -e 3rdparty/MInference --no-build-isolation
cd 3rdparty/MInference
git am ../0001-Change-KIVI-kernel-to-Triton-version.patch

Install the optional hadamard transform dependency for better quantization integration,

uv pip install -e 3rdparty/fast-hadamard-transform --no-build-isolation # Optional

3. Create Datasets (for RULER evaluation)

bash scripts/build_datasets.sh

Accuracy Evaluations

We provide an evaluation script evaluate/eval_acc.py to measure the accuracy of KV-Cache compression methods. The following methods are supported:

xKV variants (cross-layer SVD):

• Single SVD — per-layer SVD baseline (--method xkv --layer_group_size 1)
• xKV — cross-layer SVD with dense reconstruction (--method xkv --layer_group_size W)
• MiniCache — SLERP-based cross-layer merging (--method xkv --layer_merge_impl slerp)

xKV-SR variants (sparse + selective reconstruction, built on ShadowKV):

• xK-SR — cross-layer SVD on keys only, values offloaded to CPU (--method xk_sr)
• xKV-SR — cross-layer SVD on both keys and values (--method xkv_sr)
• ShadowKV — original ShadowKV baseline; equivalent to xK-SR with --layer_group_size 1 (single-layer SVD on keys)

Baselines (via MInference):

• StreamingLLM (--method streamingllm), SnapKV (--method snapkv), PyramidKV (--method pyramidkv), KIVI (--method kivi), Quest (--method quest)

Key Arguments

• --model_name_or_path: Path or HuggingFace name of the model (e.g., meta-llama/Meta-Llama-3.1-8B-Instruct).
• --dataset_name: Comma-separated list of datasets (e.g., ruler/vt,ruler/qa_1,long_bench/qasper).
• --datalen: Input sequence length in tokens (e.g., 65536).
• --method: Compression method. xKV pipeline: xkv, streamingllm, snapkv, pyramidkv, kivi, quest. xKV-SR engine: xk_sr, xkv_sr, shadowkv. Omit or use dense for full-KV baseline.
• --merge_k, --merge_v: Enable SVD compression for keys / values respectively. --layer_merge_impl: compression backend [svd (default), slerp for MiniCache].
• --start_layer_idx, --end_layer_idx: Layer range to apply compression (default: all layers; --end_layer_idx -1 = last layer).
• --layer_group_size: Number of consecutive layers grouped for cross-layer SVD (default: 1; use 2/4 for xKV).
• --rank_k, --rank_v: SVD rank per group for keys and values (default: 256 / 768). Scale proportionally with --layer_group_size for iso-compression comparisons.
• --sparse_budget: Sparse token budget for xKV-SR attention (default: 2048). --chunk_size: token chunk granularity for sparse selection (default: 8).

Evaluation on RULER Benchmark

Below we provide example commands for running the RULER benchmark.

xKV

Enables xKV compression for all layers (start_layer_idx=0 to end_layer_idx=-1), grouping every 4 layers (layer_group_size=4), using ranks 384 and 576 for each grouped keys and values.

# xKV-4
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 --flash2 \
    --method xkv --merge_k --merge_v --rank_k 384 --rank_v 576 --layer_group_size 4 --start_layer_idx 0 --end_layer_idx -1

Single SVD

For evaluation of Single SVD under a similar compression level, use --layer_group_size 1 with --rank_k 96 --rank_v 144.

# Single SVD (gs=1, rank_k=96, rank_v=144)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 --flash2 \
    --method xkv --merge_k --merge_v --rank_k 96 --rank_v 144 --layer_group_size 1 --start_layer_idx 0 --end_layer_idx -1

MiniCache

This command enables the MiniCache approach by specifying --layer_merge_impl slerp. The layers 16 through 31 are compressed.

# MiniCache (slerp, gs=2, layers 16-31)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 --flash2 \
    --method xkv --merge_k --merge_v --layer_merge_impl slerp --layer_group_size 2 --start_layer_idx 16 --end_layer_idx 31

ShadowKV / xK-SR / xKV-SR

ShadowKV (Sun et al., 2024) uses single-layer SVD on keys with sparse token selection; xK-SR replaces the per-layer SVD with cross-layer SVD. xKV-SR further compresses values on-GPU.

# ShadowKV baseline (gs=1 ≡ original ShadowKV, rank_k=96, sparse_budget=2048)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 \
    --method xk_sr --sparse_budget 2048 --chunk_size 8 --layer_group_size 1 --rank_k 96

# xK-SR (gs=4, rank_k=384, sparse_budget=2048)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 \
    --method xk_sr --sparse_budget 2048 --chunk_size 8 --layer_group_size 4 --rank_k 384

# xKV-SR (gs=4, rank_k=384, rank_v=576, sparse_budget=2048)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/vt" --datalen 65536 \
    --method xkv_sr --sparse_budget 2048 --chunk_size 8 --layer_group_size 4 --rank_k 384 --rank_v 576

Customized Merge Config

We also support customized merge config by providing a yaml file to the --customized_merge_config argument. By writing a yaml file you can experiment with different merging groups and different ranks for each group. Please refer to the configs/example.yaml for the format.

# Customized merge config (configs/example.yaml)
python evaluate/eval_acc.py \
    --model_name_or_path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dataset_name "ruler/niah_single_1" --datalen 65536 --batch_size 1 \
    --method xkv --customized_merge_config configs/example.yaml

Evaluation on DeepSeek Models

DeepSeek’s MLA (multi-latent attention) architecture has two types of hidden states that can be cached during inference:

• Non-RoPE Latents (the learned, position-agnostic latent vectors).
• RoPE-based Key States (rotary-positioned keys). We reuse the Key and Value compression interfaces for these two elements:
• --merge_k and --rank_k control compression of the non-RoPE latents (treated like "Keys").
• --merge_v and --rank_v control compression of the RoPE-based Key states (treated like "Values"). In our paper, we focus on compressing only the non-RoPE latents only.

xKV for DeepSeek (compress only non-RoPE latents)

Enables xKV compression for all layers (start_layer_idx=0 to end_layer_idx=-1), grouping every 4 layers (layer_group_size=4), using ranks 512 for grouped latents.

# xKV for DeepSeek (gs=4, rank_k=512, non-RoPE latents only)
python evaluate/eval_acc.py \
    --model_name_or_path deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct \
    --dataset_name "long_bench/repobench-p" --datalen 65536 --batch_size 1 --flash2 \
    --method xkv --merge_k --rank_k 512 --layer_group_size 4 --start_layer_idx 0 --end_layer_idx -1

Efficiency

Efficiency Benchmarks

Kernel-level benchmarks measuring decode-step latency and generation throughput across attention methods and sequence lengths. The suite covers three benchmarks:

• bench_decode_attn.py — decode-step attention-kernel latency
• bench_e2e_throughput.py — end-to-end generation throughput (tokens/s)
• bench_svd_overhead.py — prefill-time cross-layer SVD overhead

Compared methods (--mode):

• fa — FlashAttention-2 (full-KV baseline)
• xkv — xKV dense reconstruction (all tokens reconstructed)
• xkey_sr — xK-SR (cross-layer key SVD, CPU-offloaded values, selective reconstruction)
• xkv_sr — xKV-SR (cross-layer K+V SVD on GPU, selective reconstruction)
• shadowkv — ShadowKV baseline (per-layer key SVD + sparse retrieval)

Key Arguments

• --mode: Attention method(s) to benchmark, or all (default: all).
• --warmup: Number of warmup iterations (default: 3).
• --iters: Number of timed iterations (default: 10).
• --output_dir: Directory to save results (default: results/efficiency/).

Build the CUDA kernel (one-time)

Requires CUDA 12.x. The kernel uses CUTLASS (fetched via git submodule update --init 3rdparty/cutlass).

bash examples/efficiency/build_kernel.sh

Run benchmarks

bash examples/efficiency/run_benchmarks.sh

run_benchmarks.sh runs decode-attention latency, SVD overhead, and E2E throughput in sequence. Each benchmark can also be run directly, e.g.:

python efficiency/bench_e2e_throughput.py --mode fa xkv_sr xkey_sr --seqlen 60000

Results are saved to results/efficiency/. To override defaults:

CUDA_VISIBLE_DEVICES=1 WARMUP=5 ITERS=20 bash examples/efficiency/run_benchmarks.sh

Citation

If you find xKV useful or relevant to your project and research, please kindly cite our paper:

@article{chang2025xkv,
  title   = {xKV: Cross-Layer {KV}-Cache Compression via Aligned Singular Vector Extraction},
  author  = {Chang, Chi-Chih and Lin, Wei-Cheng and Lin, Chien-Yu and Chiang, Hung-Yueh and Akhauri, Yash and Dai, Xilai and Jiang, Huiqiang and Li, Yucheng and Ceze, Luis and Wu, Kai-Chiang and Abdelfattah, Mohamed S.},
  journal = {arXiv preprint arXiv:2503.18893},
  year    = {2025}
}

Acknowledgement

The xKV-SR inference engine is built upon ShadowKV. The evaluation framework is adapted from ShadowKV and Palu.