README.md

NexusQuant

Near-lossless KV cache quantization. Training-free. One line of code.

---

E8 lattice quantization applied to the KV cache after prefill. No training, no calibration data, no model modifications.

Live demo: huggingface.co/spaces/jmarquex/nexusquant-demo

Headline: on Llama-3.1-Inst at 4K (n=30, one harness and one needle set), at iso-bpe (2.125 bpe for both once the fp16 scale overhead is counted), NexusQuant K2V2 pb=0 holds 30/30 NIAH while a 2-bit TurboQuant cache drops to 0/30 (NQ-vs-TQ McNemar b=30, c=0; FP16-vs-TQ floor b=29; nq_tq_h2h_llamainst.json). The TurboQuant side is our own reimplementation on our harness, so a faithfulness gap to the reference code cannot be excluded. Near-lossless without eviction: K3V2 pb=0 costs +0.43% PPL on Mistral-Inst-v0.3 (n=161; nq_mistral_subbpe_frontier.json). Validated on 10 architecture families; NIAH recall holds through 32K context (24-25/25 per cell; nq_32k_only.json).

The earlier PyPI/HuggingFace figure of +0.274% PPL on Mistral-7B-v0.1 (n=60, KIVI-iso) is committed in rank116_kivi_iso_bpe.json. The 5.83x compression figure is measured on Mistral-Inst-v0.3 (nq_e8_bpwmatrix.json). PPL deltas above were measured with the 0.5.x quantizer; 0.6.0 corrects the E8 tie-break, so published deltas are conservative for current installs.

Install

pip install "nexusquant-kv[hf]"  # with HuggingFace transformers

compress_kv_cache (quant-only) needs transformers >= 4.46. nexusquant_evict needs transformers >= 4.56 and torch >= 2.2. It hooks DynamicLayer and DynamicSlidingWindowLayer, and DynamicSlidingWindowLayer first ships in transformers 4.56.0, so the pair only imports together from that release.

Quickstart

Prefill all but the last token, compress the cache, then generate.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache
from nexusquant import compress_kv_cache

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
inputs = tokenizer("The KV cache stores key and value tensors for each attention layer.", return_tensors="pt").to(model.device)

cache = DynamicCache()
with torch.no_grad():
    out = model(inputs["input_ids"][:, :-1], use_cache=True, past_key_values=cache)

compressed = compress_kv_cache(out.past_key_values, mode="quant_only", rope_base=getattr(model.config, "rope_theta", 10000.0))
with torch.no_grad():
    print(tokenizer.decode(model.generate(inputs["input_ids"], past_key_values=compressed, max_new_tokens=200, do_sample=False)[0], skip_special_tokens=True))

With eviction (higher compression, needs transformers >= 4.56):

from nexusquant import nexusquant_evict
with nexusquant_evict(model, quality="balanced") as nq:
    output = model.generate(inputs["input_ids"], max_new_tokens=512)

A DynamicCache subclass (E8QuantizedCache) and a kvpress adapter (E8Press, which needs Python 3.10+ since upstream kvpress floors there) are also available. Full argument reference is in the docstrings.

Results

PPL (quant-only, Mistral-7B-v0.1, wikitext-2, n=60, KIVI-iso)

Config	bpe (honest)	PPL delta vs FP16
FP16	16.000	0.000%
K3V2 pb=0	2.625	+0.274%
K2V2 pb=0	2.125	+0.96%

bpe = (key_bits + value_bits)/2 + 0.125 (the per-head fp16 scale). Source: rank116_kivi_iso_bpe.json.

NIAH recall (quant-only, chat-template, Mistral-Inst-v0.3, A100-80GB)

Context	FP16	K3V2 pb=0	K2V2 pb=0
12K	25/25	25/25	25/25
32K	25/25	25/25	24/25

Sources: nq_32k_only.json, nq_32k_modal_partial.json.

2-bit head-to-head (Llama-3.1-Inst, 4K, n=30, iso-bpe)

TurboQuant and Block-GTQ are our own reimplementations on our harness; a faithfulness gap to each reference code cannot be excluded. TurboQuant's own KV-cache experiments are reported at 2.5-bit and 3.5-bit, and its 2.5-bit configuration is mixed precision (32 outlier channels at 3 bits, 96 channels at 2); our bw2 leg is uniform 2-bit with no outlier split, so it sits below the lowest KV operating point its authors published.

Config	bpe	NIAH hits	Notes
FP16	16.0	29/30	1 grader-precedence artifact (correct answer, scored as a miss)
TurboQuant bw2	2.125	0/30	Complete retrieval collapse
Block-GTQ K2V2	2.0	29/30	Calibrated, pre-RoPE K-alloc (non-iso)
NexusQuant K2V2 pb=0	2.125	30/30	Calibration-free, post-RoPE E8

NQ-vs-TQ McNemar b=30, c=0 (30 discordant pairs, all favor NexusQuant). Sources: nq_tq_h2h_llamainst.json, nq_blockgtq_h2h_4k.json.

Supported architectures

KV head count is one axis of difficulty, not a predictor of quality. On the AQUA-iso sweep in the architectures paper, Yi-6B-Chat with 4 KV heads measures +0.237% at K3V2 and Llama-3.1-Inst with 8 KV heads measures +0.385%, and RoPE configuration is a second axis. K3V2 at protect_boundary=0 (the library default) holds wikitext-2 perplexity within 1% of the FP16 baseline on twelve of the fourteen checkpoints in that paper's perplexity sweep, so it is the setting worth trying first and not a guarantee on a model we have not measured.

The fourteen are Mistral-7B-v0.1, Mistral-Inst-v0.3, Llama-3.1-8B-base, Llama-3.1-Inst, Yi-6B, Yi-6B-Chat, Phi-3-mini-4k-Inst, Llama-2-7B, Llama-2-13B-Chat, Qwen3-8B, Mixtral-8x7B-Inst, Qwen1.5-MoE-A2.7B-Chat, Qwen2.5-7B-Inst, and Gemma-2-2b, listed in the same order as the enumeration in the architectures paper. The table below is a subset of those, plus Qwen2.5-1.5B, which is not one of the fourteen.

Model	K3V2 pb=0 PPL delta	Notes
Gemma-2-2b	+0.21%	global-only (SWA at fp16)
Mistral-7B-v0.1	+0.274%	main benchmark
Yi-6B	+0.35%	2K prefix
Mistral-Inst-v0.3	+0.33%	1K-prefix
Llama-3.1-Inst	+0.64%	pass rope_scaling; K4V2 recommended
Mixtral-8x7B-Inst	+0.63%	NF4-weights baseline, n=80 (nq_mixtral_stream.json)
Qwen3-8B	+1.130%	strict FP16-weights (nq_q3fp16_iso_strict_fp16_weights.json)
Qwen2.5-1.5B	catastrophic	2 KV heads; +5.04% is a 35% token-removal cell with protect_boundary=1, a different config from this pb=0 column; not recommended

Two of those fourteen checkpoints sit outside the band at protect_boundary=0. Qwen2.5-7B-Inst collapses outright, from a 6.121 FP16 baseline to PPL 3300 at K3V2, and Qwen3-8B lands at +1.130% in the table above. protect_boundary=1 brings both back, to +0.84% on Qwen2.5-7B-Inst and +0.721% on Qwen3-8B. Qwen2.5-1.5B fails at protect_boundary=0 too, but it is not one of those two. It comes from an earlier eviction-rate sweep and is not one of the fourteen. Llama-3.1 is the one model in the table where K3V2 is not the recommendation. Its extended-context RoPE scaling makes 3-bit keys the weak spot: K4V2 measures +0.29% against K3V2's +0.64% on Llama-3.1-Inst under the same n=60 protocol, and on the 1.7K-context NIAH sweep on Llama-3.1-8B, K3V2 recovers 1 of the 4 positions the FP16 model itself passes while K5V2 recovers all 4.

Gemma4 (SWA): use compress_layers="global_only". Encoder-decoder, GPT-NeoX/J, and GGUF are not supported.

How it works

1. (optional) Importance-score tokens and evict the lowest, keeping BOS + a recent sliding window
2. Remove RoPE from keys so they share a common subspace
3. Hadamard-rotate to spread outliers across dimensions
4. Quantize each 8-float group to the nearest E8 lattice point (asymmetric: 3-bit keys + 2-bit values)
5. (optional) Boundary-protect first/last N layers at FP16 (required on Qwen2.5-7B-Inst)

The E8 lattice is the proven densest sphere packing in 8 dimensions and the best known lattice quantizer in 8D (normalized second moment 0.0717 vs 0.0833 for scalar rounding), so the average quantization error per 8-float group is lower than scalar round-to-nearest at the same bit budget above 1 bit per element. At 1 bit the lattice has too few points in range and E8 measured worse than scalar quantization on the one model tested (method paper, Sec. 4.7). Sources: Viazovska, arXiv:1603.04246 (packing); Agrell and Allen, arXiv:2202.09605 (quantizer, Table I).

Papers

The long-form report is broken into six focused, topic-separated papers.

• Method - paper/nexusquant-method.pdf: the E8 lattice quantizer, the six-stage pipeline, and the quantizer ablations (E8 vs uniform vs Lloyd-Max).
• Architectures - paper/nexusquant-architectures.pdf: does it ruin the language model? Cross-architecture PPL (fourteen checkpoints), prefix-length scaling, mixture-of-experts, multi-turn KV reuse.
• Retrieval - paper/nexusquant-retrieval.pdf: does it remember facts? The 2-bit iso-bpe wedge, FP16 NIAH dead zones, RULER, LongBench, and the eviction cliff.
• Downstream tasks - paper/nexusquant-downstream.pdf: function-call accuracy, HumanEval pass@1, multi-document QA, and GSM8K/MATH reasoning under compression.
• Competitors - paper/nexusquant-competitors.pdf: iso-protocol head-to-heads against KIVI, CommVQ, and AQUA-KV, the non-iso Block-GTQ system comparison, and why the KVTC row was retracted.
• System and negative results - paper/nexusquant-system.pdf: honest byte accounting, rANS entropy coding, latency, memory, the K/V bit grid, and the twelve approaches that did not work.

Limitations

• Eviction hurts factual recall: NIAH degrades at or below 35% eviction, and the cliff position is model- and scorer-dependent (PPL hides this). Use quant-only when factual accuracy matters.
• Qwen2.5-7B-Inst collapses without boundary protection: K3V2 perplexity goes from 6.121 to 3300 at protect_boundary=0, and its GSM8K and MATH-500 drop to 0.00%. Keep boundary protection on for Qwen2.5 and test your own model.
• E8 quantization is CPU-bound; the Triton GPU kernel is written but not yet latency-benchmarked.
• nexusquant_evict does not support batch size > 1 or multi-turn chat; vLLM runtime integration is stubbed (NotImplementedError).

Citation

The work is broken into six focused papers. Cite the one(s) you draw from:

@techreport{nexusquant_method_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant}: Training-Free {E8} Lattice Vector Quantization for {KV} Cache Compression},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-method.pdf},
}
@techreport{nexusquant_arch_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant} Cross-Architecture Perplexity: Does {KV} Cache Compression Preserve Language Modeling?},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-architectures.pdf},
}
@techreport{nexusquant_retrieval_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant} Factual Retrieval: Does {KV} Cache Compression Preserve Needle Recall?},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-retrieval.pdf},
}
@techreport{nexusquant_downstream_2026,
  author = {Marques, Jo\~{a}o},
  title  = {Downstream Task Retention Under {KV} Compression: Function Calls, Code, {Q\&A}, and Reasoning under the {E8} Lattice},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-downstream.pdf},
}
@techreport{nexusquant_competitors_2026,
  author = {Marques, Jo\~{a}o},
  title  = {Calibration-Free {E8} {KV} Cache Quantization versus Competitors: Iso-Protocol Head-to-Head Comparisons},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-competitors.pdf},
}
@techreport{nexusquant_system_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant}: System Cost and Negative Results for Training-Free {E8} {KV} Cache Compression},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-system.pdf},
}

License

Apache 2.0. See LICENSE.