NexusQuant K3V2 pb=0 repro artifact (Mistral-7B-v0.1)

Training-free KV cache compression via E8 lattice vector quantization applied to RoPE-removed keys and Hadamard-rotated values. This artifact ships the frozen evaluation indices and reproducer scripts for the Mistral-7B-v0.1 headline row.

Headline number

NQ K3V2 pb=0 on Mistral-7B-v0.1, +0.276% PPL on wikitext-2 (n=60 chunks, 61,380 scored tokens, KIVI-iso protocol, 1024-token prefix, 1024-token continuation, relative PPL delta vs FP16).

Config	bpe (honest)	wikitext-2 delta vs FP16 (n=60, KIVI-iso)
FP16	16.000	0.000%
K4V2 pb=0 †	3.125	+0.15%
K3V2 pb=0	2.625	+0.276%
K2V2 pb=0	2.125	+0.96%

The honest bpe figures include the per-(head, row) fp16 scale at scale_bpe = 16 / head_dim = 16 / 128 = 0.125, so bpe = (key_bits + value_bits)/2 + 0.125. Source: rank116_kivi_iso_bpe.json.

That is the analytic frame: nominal bits plus the fp16 scale, with no entropy coder assumed. Nothing is hidden inside that frame, but it is not the only frame this project publishes. Measured end to end against a real coder the same configs cost more: K3V2 pb=0 is 2.743 bpe and K2V2 pb=0 is 2.303 bpe all-overhead on Mistral-7B-Instruct-v0.3 (nq_e8_bpwmatrix.json), which is the frame the demo Space card and nexusquant/__init__.py quote.

† The K4V2 pb=0 row (+0.15%) is not in rank116_kivi_iso_bpe.json, which holds only the FP16, K3V2 pb=0, and K2V2 pb=0 cells. It traces to the original Kaggle run (kernel rankLL_kivi_h2h_llama_inst) whose result JSON sidecar was not persisted and ships as run-log provenance, matching rank116_kivi_iso_bpe.json.

Paired noise-floor PPL gate

git clone https://github.com/jagmarques/nexusquant
cd nexusquant
git checkout edabb56bbbeacd96fcc6e7be0b33452ba54cdc7c
pip install -e . "torch>=2.1" "transformers>=4.46" accelerate datasets
export HF_TOKEN=hf_...
python scripts/freeze_wikitext_indices.py
python scripts/repro_headline.py

repro_headline.py replays the frozen 161-chunk protocol and checks the K3V2 pb=0 mean per-chunk delta against a fixed +0.31% reference (the n=161 overall-PPL delta on Mistral-7B-v0.1; nq_mistral_subbpe_frontier.json is the separate Mistral-Inst-v0.3 +0.43% cell). The 0.276% figure is the per-chunk sigma, a separate quantity that also coincides with the n=60 KIVI-iso headline above; it is not the mean. No local script reproduces the n=60 headline: that value is Kaggle-kernel provenance backed by rank116_kivi_iso_bpe.json. kivi_iso_repro.py below is the separate KIVI Table-3 task-accuracy gate, not a PPL reproducer. Wall-clock on a single T4 (16 GB) is roughly 25 minutes.

Related task-level reproducer

python scripts/kivi_iso_repro.py

Runs Mistral-7B-v0.1 on the three KIVI Table-3 tasks (CoQA F1, TruthfulQA MC1, GSM8K EM) under FP16 and NQ K3V2 pb=0, on subsets sized for a single T4 session (CoQA n=100, TruthfulQA n=200, GSM8K n=50). Gate: each task within 5 absolute points of the in-run FP16 baseline.

Pinned commit

edabb56bbbeacd96fcc6e7be0b33452ba54cdc7c

git rev-parse HEAD inside the cloned repo should return the same SHA before running the scripts.

Paper

arXiv:TBD (preprint pending).

License

Apache-2.0. See LICENSE in the repository root.

Files in this artifact

• scripts/freeze_wikitext_indices.py: generates scripts/frozen_wikitext_indices.json (161 chunk indices over wikitext-2 test, Mistral-7B-v0.1 tokenizer) when run; this JSON is build output, not a file shipped in the repo or this card
• scripts/repro_headline.py: PPL paired-delta gate
• scripts/kivi_iso_repro.py: CoQA / TruthfulQA / GSM8K iso-task gate
• scripts/README.md: setup and expected output

Hardware

Validated on Kaggle T4 x2 (sm_75) with torch==2.10 and transformers==4.46. repro_headline.py runs comfortably on a single T4 (16 GB). Larger GPUs (L4, L40S, A100) will be faster but produce the same number within fp16 noise (paired-NF sigma is 0.276% across 161 chunks; see the paired noise-floor paragraph in paper/nexusquant-method.pdf).