Parakeet TDT multilingual weight-type accuracy harness

Answers one question the other two test tiers cannot: does changing parakeet_tdt.matmul_weight_type actually change what the model transcribes, across the languages it claims to support?

The existing tiers are both too narrow for that:

• test_golden_transcription.cpp asserts the decoded text of one English clip. Greedy/argmax decoding is robust to a lot of numerical drift, so this passes under changes that measurably move the encoder — the parity harness's README documents a real bug it failed to catch.
• parity/ compares intermediate encoder activations against NeMo, which is the right tool for "is the math right", but it also runs on one clip and reports cosine similarity, not anything a user experiences.

Cosine similarity is not a transcription. enc_out cosine of 0.9964 for q8_0 sounds harmless; what it actually costs is measured here.

Why this matters

The completed run covers 120 FLEURS clips across 24 languages. q8_0—about 1.79x faster than F32 on the tested CPU—matches the F32 transcript exactly on 91.7% of clips. Its aggregate absolute WER is effectively flat on this sample (0.1331 versus F32's 0.1338), but the 8.3% transcript churn proves that it is not byte-for-byte interchangeable with F32. On the single English golden clip it is identical, which is exactly the trap: one clip in one language cannot characterize a quantization tradeoff.

Corpus

FLEURS test split, which is read speech with human reference transcripts and per-language configs. fetch_fleurs.py pulls N clips for each of the 24 European languages parakeet-tdt-0.6b-v3 supports that FLEURS covers (it has no Maltese), decodes them to the 16 kHz mono WAV the engine expects, and writes a manifest with the reference transcript.

Each FLEURS test parquet is 150–400 MB and only a handful of clips are kept from each, so the script deletes every parquet immediately after extracting and records per-language progress in done_<lang>.json, making it resumable and bounded in disk use. Expect the download to take a while; it is entirely network-bound.

pip install pyarrow soundfile numpy
python3 tests/parakeet_tdt/multilingual/fetch_fleurs.py 5   # 5 clips/language

Running the comparison

cmake --build build/<preset> --target parakeet_warm_bench
python3 tests/parakeet_tdt/multilingual/compare_weight_types.py native f16 bf16 q8_0

All clips for a given weight type go through a single process via --audio-sequence, so the model and its quantized weights are built once rather than once per clip. The first named type is the reference everything else is compared against, so put native first.

Environment overrides: FLEURS_DIR (corpus location, default tmp/fleurs), PARAKEET_BENCH (bench binary), PARAKEET_THREADS.

The checked-in results under results/ were produced from the upstream safetensors package with each storage type selected at session creation. The standalone-GGUF regression checks cover package loading and the golden paths; they are not a second 120-clip corpus run.

Reading the output

Three numbers per weight type:

• exact vs native — fraction of clips whose decoded text is byte-identical to the f32 run. This is the number that matters for "is this change free".
• WER vs native — how far the text actually moved. Distinguishes "differs on a few clips by one word" from "differs badly".
• WER vs FLEURS — absolute quality against the human reference, so a weight type that merely agrees with f32 while both are bad cannot hide, and so you can see whether a difference from f32 is actually worse or just different.

Then a per-language WER breakdown, because quantization damage is not spread evenly — it concentrates in particular languages, and an aggregate hides that.

WER uses word-level Levenshtein over NFKC-normalized, lowercased, punctuation-stripped text. That normalization is deliberately crude: it is meant for comparing weight types against each other on identical audio, not for publishing absolute WER figures comparable to other papers' numbers.