GGUF parity report

August 4, 2026 · View on GitHub

Differential results from tools/diff_gguf.c, which runs modelvet and a real upstream GGUF loader over the same inputs and buckets the outcomes. Every divergence below is triaged. Nothing here is a claim about a codebase that was not actually executed.

What was measured, and against what

Harness: make diff-gguf. Upstream runs in a forked child, so an upstream abort is recorded as a bucket rather than ending the run.
Upstream build tested: the ggml tree vendored in a local whisper.cpp checkout, commit 21411d81ea736ed5d9cdea4df360d3c4b60a4adb (2026-02-19), linked as libggml-base.so.
Upstream call: gguf_init_from_file with no_alloc = true, the metadata-only mode a scanner would use. Some checks upstream performs only when materializing tensor data do not run in this mode; that is called out per finding below and is not counted against upstream.
This is NOT the pinned llama.cpp revision. tools/ggml-pin.env pins llama.cpp 0ef6e55e for the dtype-table guard; building it was out of scope for this local run. The differential job in .github/workflows/fuzz-nightly.yml runs this same harness against the pinned llama.cpp revision, and that job is what confirms or refutes the upstream-side findings below for llama.cpp specifically. CONFIRMED 2026-08-04: that job has now run against the pinned revision; see "Pinned-revision confirmation" below. Every vendored-build caveat in this report is resolved there.

Two input sets were measured, 249 files total.

Set A: the curated CVE corpus (11 GGUF files)

bucket	count
both accept	0
both reject	6
we accept / they reject	0
we reject / they accept	4
we reject / they crash	1
we accept / they crash	0

Set B: fuzzer-generated inputs (238 GGUF files)

Generated by build/gguf_fuzz seeded from the corpus.

bucket	count
both accept	9
both reject	194
we accept / they reject	0
we reject / they accept	34
we reject / they crash	1
we accept / they crash	0

The headline

Zero inputs in either set landed in we-accept-they-reject. Across 249 files, modelvet was never more permissive than the tested upstream loader. That is the property a verify-before-load boundary has to have: it may be stricter, it must never wave through what the loader itself would refuse.

Pinned-revision confirmation (2026-08-04)

The CI differential job built the exact pinned llama.cpp revision (tools/ggml-pin.env) and ran this harness over the curated corpus (now 11 GGUF files):

bucket	count
both accept	0
both reject	10
we accept / they reject	0
we reject / they accept	1
we reject / they crash	0
we accept / they crash	0

What this resolves, item by item:

Both abort findings: vendored-build artifacts, nothing to report upstream. The pinned revision cleanly rejects the CVE-2024-23496 string-length shape (and no corpus input crashes it). The aborts observed locally were properties of the Feb-2026 vendored ggml tree, not of current llama.cpp.
The size-overflow divergence (310): resolved. The pinned revision, which postdates the CVE-2026-33298 fix, rejects the reconstructed shape. The local acceptance was the unpatched vendored tree, as suspected.
The alignment gap (221): CONFIRMED LIVE. The single remaining divergence on the pinned build is live-alignment-gap.gguf: llama.cpp still accepts general.alignment = $2^{3}$1. This is now confirmed against the audited revision, not just a vendored build.

Triage: we reject / they crash (2 inputs)

Upstream did not reject these files. It threw an uncaught C++ exception that reached std::terminate and aborted the process. For a scanner or server that embeds the loader, an abort on attacker-controlled input is an availability failure, not a rejection.

tests/corpus/gguf/cve-2024-23496.gguf (our code 210, MVET_V_STR_LEN_CAP). A KV string declaring length UINT64_MAX. Upstream path: gguf_reader::read(std::string&) → std::__throw_length_error → std::terminate → abort. This is the reconstructed shape of the 2024 string-length advisory.
One fuzzer input (our code 220, MVET_V_ALIGN_TYPE), same abort signature.

Action (resolved 2026-08-04): the pinned revision rejects both shapes cleanly (see "Pinned-revision confirmation"). The aborts were vendored-build artifacts; there is no upstream abort claim to report.

Triage: we reject / they accept (38 inputs)

Every member falls into one of four classes. None is a modelvet false positive; three are documented deliberate strictness and one is upstream under-validation in the mode tested.

class	code	count	verdict
trailing bytes after the data region	400	21	deliberate policy
declared tensor data past end of file	311	13	upstream gap in `no_alloc` mode
bool payload byte not 0 or 1	208	1	deliberate policy
zero tensor dimension	304	1	deliberate policy (upstream SIGFPE hazard)
`general.alignment` above the cap	221	1	the live upstream gap
tensor byte-size arithmetic overflow	310	1	needs pinned-revision confirmation

Trailing bytes (400). modelvet requires a file to end exactly at the data region; upstream ignores anything after it. This is the anti-polyglot rule, a locked decision, and the largest single source of divergence. Expected and intended.
Data extent (311). Upstream with no_alloc = true does not check that the declared tensor data fits inside the file. This is the same class as CVE-2026-7482 in Ollama's independent Go loader. Our rejection is correct; the count is inflated by the harness mode, so it is reported as a mode caveat rather than an upstream bug.
Bool byte (208) and zero dimension (304). Both are documented strictness deltas. The zero-dimension case is the one where upstream then divides by the dimension, the live SIGFPE hazard modelvet refuses to mirror.
Alignment cap (221). live-alignment-gap.gguf carries general.alignment = $2^{3}$1: nonzero and a power of two, so upstream accepts it and feeds it into padding and seek arithmetic. This run is the empirical confirmation of that gap on a real build.
Size overflow (310). Upstream accepted the reconstructed ggml_nbytes overflow shape (CVE-2026-33298). The tested build predates the advisory's fix, so this most likely reflects an unpatched vendored tree rather than a live gap in current llama.cpp. Resolved 2026-08-04: the pinned revision rejects it (see "Pinned-revision confirmation").

Reproducing

make diff-gguf \
  GGML_DIR=/path/to/llama.cpp \
  GGML_LIB=/path/to/libggml-base.so \
  DIFF_INPUTS=tests/corpus/gguf

Open work

~~Confirm both abort findings and the size-overflow divergence against the pinned llama.cpp revision, then decide what to report upstream.~~ Done 2026-08-04 ("Pinned-revision confirmation"): nothing to report except the live alignment gap, which is confirmed.
Re-run with no_alloc = false to separate upstream's metadata-only checks from its full-load checks.
~~Extend the harness to safetensors against the canonical Rust crate.~~ Done: PARITY-ST.md, measured against the exact pinned canonical commit.