WASTE Container Format v0

Status: v0, frozen against the released weights. This started as a pre-release design; the TBDs were settled on 2026-07-27 when K3 dropped and are recorded as answers below. Two things in it are specified and not implemented — the shared low-rank block and the SUB1 substitute bank — and both say so where they appear. A third, the per-expert bit allocator, was measured instead of built and then dropped: design goal 5.

format_version is enforced: a container from another version, or one without the field, is refused rather than read against the wrong rules.

Design goals

1. One coalesced read per expert. An expert's gate/up/down matrices are adjacent on disk and loaded with a single pread — measured as the difference between usable and unusable NVMe throughput.

2. Placement decides speed, never precision. The invariant: output is bit-identical whether an expert came from RAM cache or disk. The substitute path (below) is the one deliberate, bounded exception, and it is off by default.

3. O_DIRECT-friendly. Every independently-readable record is aligned to 4 KiB and sized in 4 KiB multiples, so the page cache can be bypassed (measured wins from F_NOCACHE/O_DIRECT on some drives).

4. Sub-4-bit without collapse. Per-expert weights are vector-quantized: multi-stage (residual) VQ over 8-dim vectors with per-channel scales. Gate 3 measured this on real Kimi experts — VQ decisively beats RTN below 4 bits, and 3 bits (19.4% error) is the operating point; 2-bit VQ (33%) is more than double the known-good production int4 baseline.

   The KBVQ-MoE shared-low-rank component (arXiv 2602.11184) is specified but NOT implemented in v0 — see "Shared low-rank: on probation" below.

5. Non-uniform bits. Measured and dropped. The design goal was global per-expert bit allocation à la GEMQ (arXiv 2605.23078) — important experts at 3 bits, unimportant at 2. It assumes experts differ in how much the third bit buys them, and on this family they do not: the spread in that quantity is 1.06–1.15x between experts in a layer, 1.01x between layers, and 1.09–1.30x between gate, up and down, on both K3 and Kimi-Linear. An optimal allocator and a coin flip therefore write the same container, and the only importance signal that is not flat — routing frequency — buys disk footprint and no I/O. Every expert in a container stays at one width. The measurement, and the criterion that would revive this, are in LEARNED.md §20; the instrument is tools/bitalloc_lab.py.

   Attention / router / norms / shared experts / MTP head stay at 4–8 bit (asymmetric recipe; the MTP head must be int8+). That part stands.

File layout

A WASTE model is a directory (not a single file — shard-friendly, resumable conversion, multi-drive splitting):

model.waste/
  manifest.json        # config, trunk tensor index, expert bank index
  trunk.bin            # resident dense part
  experts-L{layer}.bin # one expert bank per layer
  codebooks.bin        # VQ codebooks, resident
  tokenizer.model      # the model's tiktoken rank file, copied in
  specials.json        # special-token ids and their exact strings
  vision.json          # the tower's shape, if the container has one
  chat.json            # conversation format — hand-written, see examples/
  chat_template.jinja  # the release's own template, when it ships one
  usage.waste          # runtime-appended routing stats / learned hotlist
  subs-L{layer}.bin    # 1-bit substitute bank — specified, NOT written in v0
  lowrank.bin          # shared low-rank factors — specified, NOT written in v0

Everything up to usage.waste is produced by tools/convert.py, except chat.json: neither Kimi release distributes a chat template, so that one is written by hand. See examples/.

manifest.json

JSON (hardened parser, treat as untrusted — cfg_sane() bounds every dimension, tensor sizes are bounded by the file that holds them, and the fuzzer in tools/fuzz_container.py exists for this file). Keys:

key	what it holds
format_version	0. Absent or different ⇒ refused
arch	descriptive; the engine derives its own from config
tensor_prefix	language_model. on K3, "" on Kimi-Linear
config	the release's own config verbatim, with the multimodal wrapper under _outer
expert_quant	stages, vec_dim, entries, index_block, index_bits, bits_per_weight
layers	per MoE layer: file, experts, bytes, codebook_base
trunk	per tensor: name, fmt, off, shape, group, scale_off, bytes

Two fields from the original design are not here. There is no bits[]: the GEMQ-style per-expert bit allocator was specified, measured (design goal 5 above, LEARNED.md §20) and dropped, so every expert in a container is at the same width — deliberately, now, rather than for want of the code. And there is no blake3 — the only checksum in the format is the per-expert crc32 in the record header.

What the engine checks on the read path

Two different things happen to every record that comes off the disk, and only one of them is optional.

Always: the header. A record must carry the right magic, be the expert the bank index asked for at that offset, be the bank's stride long, be a VQ format with lowrank_id == 0, name a codebook that exists, and have ordered offsets that fit. All of it is derivable from the manifest, so it is derived rather than believed. This is O(1) per record and it is not a integrity feature — it is what keeps an offset out of a damaged header from reaching the arithmetic downstream, which is also why the checksum could not be bolted on without it: the offsets that say where the payload ends live in the header, so deciding how much to checksum from an unvalidated one would be a read past the buffer rather than a check. A short read is caught here too.

On request: the checksum. waste_cfg.verify_records, --verify on the CLI, or WASTE_VERIFY=1 in the environment. It is off by default, and that is a throughput decision: it is a pass over every record on every cache miss, about 5% on Kimi-Linear (4.7% and 5.5% in two sittings — LEARNED.md §21) and about 1% on K3, where 11.83 MB records make the read dominate — 7,287 misses at 0.376 ms of CRC each is 2.7 s of a 268 s run. Turn it on for a container that was copied, downloaded, or has sat on a disk you do not trust; leave it off for the one you converted locally and have been reading all week. A cache hit is never re-checked either way: the unit being verified is bytes entering RAM.

The crc32 covers the payload from the end of the header to the end of the per-channel scales, excluding the 4 KiB padding — bit for bit what zlib.crc32 returns, since tools/convert.py is what writes it, and tests/test_container.c pins the two together against known values. The implementation is src/crc32.c: 33 GB/s where the ARMv8 CRC extension is available, 2.7 GB/s from the slice-by-8 table everywhere else.

Either check failing ends the generation with WASTE_E_IO, and waste_error_detail names the record — "expert 412 of layer 37: checksum mismatch" — rather than answering with whatever the damaged bytes decode to. tools/verify_container.py checks every record's checksum unconditionally and is the right tool for auditing a container once, as against paying for it on every token.

Not checked at all: the trunk, which has no checksum in the format, and the codebooks. Both are read once at load rather than per token, so the argument for a per-record check does not carry over — it is simply not built.

Quantization formats (fmt)

fmt	name	bits/weight	use
0	F32	32	norms, router, e_score_correction_bias
1	F16/BF16	16	codebooks; low-rank factors if they ever land
2	Q8G	8 (+f16 scale /g128)	the embedding table and the LM head on every container, the vision tower on K3, and anything under --trunk8 / --trunk-bits 8
3	Q4G	4 (+f16 scale /g128)	the trunk default, on every model
4	VQ3R	3.00 (3 stages x 256 entries, dim 8)	default for experts (Gate 3)
5	VQ2R	2.00 (2 stages x 256 entries, dim 8)	only where Gate 3 quality allows
6	SUB1	~1.0 direct VQ	cache-miss substitutes — specified, not written
7	Q3G	3 (+f16 scale /g128)	implemented, default for nothing — see below
8	VQ4P	3.00 (4 stages x 64 entries, dim 8, 6-bit packed)	experts, where the arithmetic is the budget — see below

The bits/weight column for VQxR is exact — one byte of index per 8-dim vector per stage — plus one f16 scale per output row, i.e. 16/n_in amortized. VQ4P instead spends index_bits per stage, so its rate is stages * index_bits / vec_dim = 3.00. A reader must take stages/vec_dim/entries/index_bits from the manifest rather than from the format id; index_bits is absent in containers written before VQ4P and means 8 there.

Q3G exists and is not recommended. The trunk is the RAM floor and the floor is what the expert cache does not get, so a 3-bit trunk looks like free cache. It was built and measured, twice. It does get the better hit rate — 29% against 12% at the same budget — and it is 1.4x slower anyway, because the scalar 3-bit unpack costs more in the trunk matvecs than the cache saves in I/O. Worse, generation collapses: K3's QAT covered the expert weights only, so the trunk has no trained tolerance for being squeezed, and the logits land 36% off. Vectorizing the unpack would not save it — the quality wall sits in front of the speed wall. LEARNED.md §13 has both measurements.

VQxR record (per expert matrix): N stages of 8-dim VQ indices into per-layer codebooks of 256 entries each (N=3 for VQ3R, N=2 for VQ2R), plus one FP16 scale per output channel:

W_expert ≈ scale_per_channel * sum_{s=1..N} codebook_s[index_s]

Residual (multi-stage) VQ: each successive codebook quantizes what the previous stages left over. Bits/weight = N, plus 16/n_in for the channel scale. Measured on real Kimi experts in Gate 3.

Index layout is blocked by 64 rows (index_block in the manifest): [row_block][vector_position][row_in_block][stage]. The engine walks a tile of rows for one vector position at a time; in plain row-major order those rows sit n_in/8 * stages bytes apart, so each is a separate cache line. Blocked, a tile's indices for one position are contiguous. The block size matches VQ_TILE in the engine; a reader must honour index_block (0 = plain row-major), because it changes where the bytes are, not merely how fast they are read.

The speed argument for it was refuted. Blocking measured 1.44x on the gather loop in isolation and changed nothing in the real engine — the microbenchmark did not model 12 threads sharing L2 — and finding that out cost a full reconversion. The layout stays because containers are written in it and it is not worse; do not repeat the 1.44x as a result. LEARNED.md §7.

VQ4P record: the same residual VQ and the same blocked layout, at the same 3.00 bits/weight and the same record size, with four stages of 64 entries instead of three of 256. Only the trailing per-row run changes, from stages whole bytes to three packed ones:

[row_block][vector_position][row_in_block][3 bytes]

Four 6-bit fields, little-endian, LSB of stage 0 at bit 0:

byte0 = s0 | s1<<6      byte1 = s1>>2 | s2<<4      byte2 = s2>>4 | s3<<2

so s0 = b0 & 0x3f, s1 = (b0>>6) | (b1&0x0f)<<2, s2 = (b1>>4) | (b2&0x03)<<4, s3 = b2>>2. The engine's unpack is a shift and a mask per stage, which is why the order is this one and not a tidier big-endian packing.

Why it exists. A 64-entry stage table is 64 bytes — exactly one NEON vqtbl4q — where VQ3R's 256-entry table is sixteen vector registers on a machine with thirty-two and cannot be held at all. That is the whole reason the VQ3R gather is scalar. Measured on K3's gate shape, the kernel is 3.32x the scalar one; the cost is +2.7% perplexity on Kimi-Linear, all of it from the smaller codebook. The int8 quantization of the runtime lookup table, which is what lets a byte shuffle index it at all, measured free. LEARNED.md §41.

It is a distinct fmt on purpose. A VQ4P payload is byte-for-byte the same size as a VQ3R one for the same matrix, so a reader that took these three bytes for three one-byte indices would decode silently and wrongly — no bounds check would fire. The engine additionally refuses a record whose fmt byte disagrees with the manifest's index_bits.

Where it is worth it. Where the apply is dispatch-bound and small: 1.18x on Kimi-Linear. On K3 it is 1.09x — measured, both containers on the same disk — which is real and is not a reason to reconvert 982 GB. The kernel is 3.88x in isolation even against a gigabyte of indices; it does not scale with threads inside the engine and why is an open question. LEARNED.md §46.

Shared low-rank: on probation — specified, NOT implemented in v0

lowrank.bin and the lowrank_id field exist in the spec but the converter does not emit them and the engine does not read them; lowrank_id must be 0 in v0.

Why parked (Gate 3 plus a follow-up subspace measurement, 2026-07-27, on real Kimi-Linear-48B experts):

• at rank N/128 the shared basis costs 0.12 bits/weight and reduces error by 0.3 pp — noise;
• at equal budget it loses badly: kbvq2 at 4.01 bits = 28.87% error, plain per-row INT4 at 4.01 bits = 15.20%;
• structurally, Kimi's experts are nearly mutually orthogonal — pairwise overlap of their rank-72 dominant subspaces is 0.046 against a random baseline of 0.031 (identical = 1.0); a shared basis captures 7.1% of energy vs 3.1% for random directions and 20.0% for each expert's own.

Why not deleted. Those measurements are in the unweighted weight metric. "KLT-guided" in the paper most likely means a basis chosen after whitening by the activation covariance, and there is a credible mechanism by which that flips the result: every expert sees the same hidden-state distribution, and LLM hidden states concentrate in a few dominant directions, so in the activation-weighted metric the useful directions may be shared by construction even though the weights are orthogonal here.

What settles it: rerun the Gate 3 comparison with an importance matrix from real activations, in the same rented GPU session as Gate 2 and the Gate 4 oracle. Revive if a whitened shared basis buys >1.5 pp of error at ≤0.15 bits/weight; otherwise delete the section and the field. No data has been written in this format yet, so either way it is a cheap change.

Expert bank record (experts-L{n}.bin)

[4 KiB-aligned]
ExpertRec {
  u32 magic 'WEXP', u16 layer, u16 expert_id
  u8  fmt (VQ3R|VQ2R), u8 flags, u16 codebook_id
  u32 gate_off, up_off, down_off, correction_off   // within record
  u32 record_4k_blocks
  -- gate indices | up indices | down indices | per-channel corrections --
}

One pread of record_4k_blocks * 4096 bytes yields the whole expert. On K3 that record is 12 406 784 bytes, exactly 3029 pages — which is what makes O_DIRECT possible, and why bank_open checks the alignment rather than assuming it: a record that is not a page multiple makes every read fail EINVAL instead of merely running slow.

Records for the same layer are contiguous and sorted by expert id. subs-L{n}.bin would mirror the layout at SUB1 precision (~5× smaller reads, used only when the engine's miss-latency budget is exceeded, HOBBIT-style, arXiv 2411.01433 — it would need a flag, because it breaks bit-exactness). Specified, not implemented: the converter writes no substitute bank, so there is nothing to substitute. waste_cfg carried an allow_substitutes flag for this until 0.6.0; it was removed, because a switch the engine never read described a capability the engine did not have.

trunk.bin

Everything needed for a forward pass with zero expert reads: KDA/MLA attention weights, routers, shared experts, the latent MoE projections, norms and the LM head — plus the embedding table and, on a multimodal container, the vision tower.

The last two are in the file but not in the resident set. embed_tokens is 1.11 GB of which one 7 KB row is read per token, so it stays on disk and the row is pread on use. The tower is loaded only when a caller asks for images: 434 MB of weights, and 1.12 GB reserved once the bounded source decode, the tower's activations and the queued image embeddings are counted, all of it otherwise straight out of the expert cache. Everything else is touched in full on every token, so streaming it would cost more I/O than the freed cache could save.

Measured on K3: 27.28 GB resident out of a 29.06 GB floor at 4K context. The pre-release target was ≤ 25 GB; the real trunk missed it, and that overshoot is most of why decode sits at 0.5 tok/s rather than 1.5. (0.3 before read-ahead — EFFICIENCY.md.)

usage.waste

Append-only runtime log: per-(layer, expert) hit counts + decayed recency for the LFRU policy, written by --learn and preloaded on the next open so a run starts warm instead of empty. Measured on Kimi-Linear at a 5 GB budget: 1602 misses cold against 1175 warm, 61% → 72%. The cross-layer routing pairs for a pilot/COUPLE prefetcher have a field in the entry struct and no code behind them, and the converter cannot yet bake an initial hotlist from a calibration corpus.

Converter pipeline (tools/convert.py)

1. Stream release shards one at a time — never needs the full 1.42 TB locally beyond the shard in flight plus the output.
2. Dequant MXFP4 → f32 blocks (tools/mxfp4.py, verified bit-identical to compressed_tensors' own unpacker). Only the routed experts are packed; the whole trunk, latent projections and shared experts included, ships as plain bf16. No shared-basis pass — see "Shared low-rank: on probation".
3. Fit VQ codebooks by k-means in 8-dim space on a sample of experts (--cb-sample, default 12), then load, quantize and write each expert on its own, so peak memory is a few hundred MB regardless of model size. Layers convert in separate processes (--jobs), each with its own codebook file, merged by concatenation.
4. Write the trunk, the tokenizer, the special tokens, the vision config and the manifest. Verify with tools/verify_container.py, which dequantizes records back and diffs them against the source weights.

Step 3 of the original design is not missing, it is refused: there is no GEMQ-style per-expert bit allocation, because the importance it would allocate against does not vary — 1.01x between layers, 1.15x between experts. Every expert in a container is at --stages bits. LEARNED.md §20 has the numbers and the revive criterion.

Questions the weights drop answered

• Per-layer expert count and shape. 896 flat, top-16, 92 MoE layers of 93 — but operating on a 3584-wide latent, not the 7168 hidden, which halves both the expert size and the per-token I/O.
• MXFP4 → VQ requantization error compounding. It does not compound badly: requantizing an already-4-bit source down to 3 costs about 2 pp over rtn4-row (20.3% against 18.4%), close to the 19.4% measured going to 3 bits from bf16 on Kimi-Linear. QAT-from-MXFP4 does appear to leave the weights tolerant.
• Codebook granularity. Per layer, 256 entries per stage, fitted on 12 sampled experts. Per-expert-group was never needed.

Still open

• Whether SUB1 substitutes measurably hurt K3 quality — untestable until the substitute bank is written at all.
• Whether the shared low-rank basis survives an activation-weighted metric (the revive-or-delete criterion above).
• Whether per-expert importance spreads under that same metric. Both questions want the same rented GPU session and the same importance matrix, so they are one experiment, not two.
• A read-path integrity check that does not cost a pass per miss.