Granite Speech

September 12, 2026 · View on GitHub

Status: research

Identity

Family key: granite
Upstream architecture strings: granite_speech, granite_speech_plus
Hugging Face repos:
- ibm-granite/granite-4.0-1b-speech (pinned bd87ab862416353633ea431fe49b1614003623c5)
- ibm-granite/granite-speech-4.1-2b (pinned 8f4bb5f31ae98971bd218169f00065a041d20058)
- ibm-granite/granite-speech-4.1-2b-plus (pinned edd3bf54fbb06d8e263aa0c1939321d67b073f86)
License: Apache-2.0 (all variants)
Variants:
- granite-4.0-1b-speech: audio-llm, AR; supports en/fr/de/es/pt/ja transcription and bidirectional translation.
- granite-speech-4.1-2b: audio-llm, AR; same supported languages; improved punctuation/casing and keyword biasing over 4.0-1b.
- granite-speech-4.1-2b-plus: audio-llm, AR; en/fr/de/es/pt only (no ja); adds speaker diarization tags ([Speaker N]:) and word-level timestamps ([T:N] centisecond format) in the text stream. Encoder concatenates mid-layer (idx 3) and final-layer hidden states (cat_hidden_layers=[3]).

Sibling family: granite_nar (the non-autoregressive ibm-granite/granite-speech-4.1-2b-nar editor model). Same underlying Conformer encoder dims and Granite-4.0-1b-base LLM weights, but the bidirectional-editor decoding pipeline is structurally distinct enough to warrant its own family. See docs/porting/families/granite_nar.md.

Note: IBM publishes an official LLM-only GGUF at ibm-granite/granite-4.0-1b-speech-GGUF. It ships the inner granite-4.0-1b-base LLM weights as architecture=granite but does not include the Conformer audio encoder or the Q-Former projector, so it is not reusable as a transcribe.cpp speech model. We will produce a fused encoder+projector+LLM GGUF (granite_speech architecture) ourselves.

References

Canonical reference: transformers mainline for the AR variants (GraniteSpeechForConditionalGeneration, GraniteSpeechPlusForConditionalGeneration).
Instrumented reference: same as canonical. To be configured under scripts/envs/granite/ at Stage 2.
Cross-check references:
- ibm-granite/granite-4.0-1b-speech-GGUF (LLM-only; useful for tensor-name and quant-ladder conventions on the Granite LLM half).
- Open ASR Leaderboard reproduction harness on HuggingFace (hf-audio/open-asr-leaderboard). IBM's published WER numbers come from this.

Transformers version pinning:

granite_speech (4.0-1b, 4.1-2b): transformers>=4.52.1. Model cards pin 4.54.0 / 4.57.6 respectively.
granite_speech_plus (4.1-2b-plus): requires transformers>=5.8 (config pins 5.6.0.dev0); the PyPI release may lag, so the env may need a pinned commit SHA off HEAD.

The reference env (scripts/envs/granite/pyproject.toml) must cover both granite_speech and granite_speech_plus. Stage 2 picks a transformers SHA new enough to register granite_speech_plus.

Architecture

The AR variants share a near-identical skeleton:

Audio encoder: Conformer with block-attention (model_type granite_speech_encoder / granite_speech_plus_encoder).
- 16 layers, hidden_dim=1024, 8 heads at dim_head=128, depthwise conv_kernel_size=15, conv_expansion_factor=2, feedforward_mult=4, input_dim=160 (= 80 logmels with a 2-frame stack at input), output_dim=348, max_pos_emb=512 (Shaw relative-position embedding), context_size=200 (block-attention boundary).
- Macaroni FFN halves (0.5 residual scaling on both half-FFNs), GLU-gated convolution module, self-conditioned CTC bypass from a middle layer.
- 4.1-2b-plus adds cat_hidden_layers=[3]: encoder hidden state at layer 3 is concatenated along the feature dim with the final-layer output; projector input becomes 2*1024=2048.
Projector: BLIP-2 Q-Former (model_type=blip_2_qformer). 2 layers, hidden_size=1024, intermediate_size=4096, 16 heads, absolute position embedding, cross_attention_frequency=1, layer_norm_eps=1e-12. encoder_hidden_size=1024 for base, 2048 for -plus. Trainable queries cross-attend to encoder output windows of window_size=15. The projector vocab_size=30522 is vestigial from BLIP-2 and unused.
- downsample_rate=5. Effective audio-token rate into the LM is ~10 Hz.
Text LM: granite-4.0-1b-base (model_type=granite). 40 layers, hidden_size=2048, intermediate_size=4096, 16 query heads / 4 KV heads (GQA), vocab_size=100353, max_position_embeddings=4096, rope_theta=10000 (standard RoPE, no scaling), rms_norm_eps=1e-05, SiLU activation. Granite scalar multipliers embedding_multiplier=12, logits_scaling=8, attention_multiplier=0.0078125 (=1/128), residual_multiplier=0.22 apply on every forward pass; missing any one of them silently degrades accuracy.
- tie_word_embeddings: false for 4.0-1b and 4.1-2b, true for 4.1-2b-plus and 4.1-2b-nar.
Fusion: the projector output is scattered into the LM input embedding at positions where input_ids == audio_token_index (100352).

Frontend

All AR variants use GraniteSpeechFeatureExtractor:

sample_rate=16000, mono.
n_mels=80, hop_length=160 (10 ms), win_length=400 (25 ms), n_fft=512 (window zero-padded to 512 before FFT). The asymmetry between win_length=400 and n_fft=512 is a classic mismatch trap.
Window/center/padding/mel-norm not declared in preprocessor_config.json; values follow torchaudio.transforms.MelSpectrogram defaults: Hann periodic window, center=True, reflect padding, htk-style mel scale with no filter normalization. Stage 2 (oracle) must confirm these by running the reference processor end-to-end.
After the projector's downsample_rate=5, the effective audio-token rate into the LM is ~10 Hz for the AR variants.

The encoder consumes 80 mels x 2-frame stack = input_dim=160. The 2-frame stack is implicit in the encoder definition; the converter must preserve it on the path from mel to encoder input.

The granite-speech-4.1-2b-plus repo is missing preprocessor_config.json. The processor is registered as granite_speech_plus in HEAD transformers and presumably reuses GraniteSpeechFeatureExtractor defaults; the oracle must confirm.

Commands

Reference WER run (per-variant prompt picked automatically from --model):

uv run --project scripts/envs/granite \
  scripts/wer/run_reference_granite_transformers.py \
    --model ibm-granite/granite-4.0-1b-speech \
    --manifest samples/wer/test-clean.manifest.jsonl \
    --out reports/wer/granite-4.0-1b-speech-REF.test-clean.jsonl

Reference tensor dumps (per-variant):

uv run --project scripts/envs/granite \
  scripts/dump_reference_granite_transformers.py encoder \
    --model ibm-granite/granite-4.0-1b-speech \
    --audio samples/jfk.wav \
    --out build/validate/granite/granite-4.0-1b-speech/jfk/encoder/ref

uv run --project scripts/envs/granite \
  scripts/dump_reference_granite_transformers.py decode \
    --model ibm-granite/granite-4.0-1b-speech \
    --audio samples/jfk.wav \
    --out build/validate/granite/granite-4.0-1b-speech/jfk/decode/ref

Conversion (preserves source dtypes; one HF repo per variant):

uv run --project scripts/envs/granite \
  scripts/convert-granite.py ibm-granite/granite-4.0-1b-speech \
  --repo-id ibm-granite/granite-4.0-1b-speech

Validation (ref-dtype, CPU strict; --variant selects per-variant manifest):

uv run scripts/validate.py all \
  --family granite --variant granite-4.0-1b-speech

Or step by step:

uv run scripts/validate.py ref     --family granite --variant granite-4.0-1b-speech
uv run scripts/validate.py cpp     --family granite --variant granite-4.0-1b-speech
uv run scripts/validate.py compare --family granite --variant granite-4.0-1b-speech

Benchmarks (per-variant):

uv run scripts/perf/bench.py \
  --model models/granite-4.0-1b-speech/granite-4.0-1b-speech-Q8_0.gguf \
  --backend metal \
  --machine "$(uname -srm | tr ' ' _)"

Full WER sweep (Stage 7):

for PRESET in BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do
  uv run scripts/wer/run.py \
    --model models/granite-4.0-1b-speech/granite-4.0-1b-speech-${PRESET}.gguf \
    --manifest samples/wer/test-clean.manifest.jsonl \
    --out reports/wer/granite-4.0-1b-speech-${PRESET}.test-clean.jsonl
  uv run scripts/wer/score.py reports/wer/granite-4.0-1b-speech-${PRESET}.test-clean.jsonl
done

Capability Validation

One row per advertised capability per variant. Stage 1 drafts the rows with Status: TODO; Stage 4 fills in observed statuses after running each command.

Allowed statuses: PASS, SKIP - not exposed by runtime, ACCEPTED GAP - <reason>.

Capability	Variant	Mode	Command / test	Expected observable	Status
Transcribe	granite-4.0-1b-speech	explicit language hint	`build/bin/transcribe-cli -m models/granite-4.0-1b-speech/granite-4.0-1b-speech-BF16.gguf --language en samples/jfk.wav`	non-empty plausible English transcript	PASS (matches reference exact)
Transcribe	granite-4.0-1b-speech	auto / no language hint	`build/bin/transcribe-cli -m models/granite-4.0-1b-speech/granite-4.0-1b-speech-BF16.gguf samples/jfk.wav`	non-empty plausible transcript on the auto-detect path	PASS (same transcript; --language is a no-op for granite which has one chat prompt)
Translate (X->En)	granite-4.0-1b-speech	non-English source audio	`build/bin/transcribe-cli -m models/granite-4.0-1b-speech/granite-4.0-1b-speech-BF16.gguf --translate --target-language en samples/german.wav`	non-empty English transcript on non-English audio	PASS ("at the beach, the bathing suit, ...")
Translate (En->X)	granite-4.0-1b-speech	English source audio + target language hint	`build/bin/transcribe-cli -m models/granite-4.0-1b-speech/granite-4.0-1b-speech-BF16.gguf --translate --target-language de samples/jfk.wav`	non-empty non-English transcript on English audio	PASS ("und so meine amerikanischen freunde: fragen sie nicht ...")
Keyword biasing	granite-4.0-1b-speech	hotword prompt	n/a	hotword preserved verbatim in transcript	SKIP - not exposed by runtime (no --prompt / hotword flag)
Transcribe	granite-speech-4.1-2b	explicit language hint	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b/granite-speech-4.1-2b-BF16.gguf --language en samples/jfk.wav`	non-empty plausible English transcript with punctuation/casing	PASS (matches reference exact)
Transcribe	granite-speech-4.1-2b	auto / no language hint	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b/granite-speech-4.1-2b-BF16.gguf samples/jfk.wav`	non-empty plausible transcript	PASS
Translate (X->En)	granite-speech-4.1-2b	non-English source audio	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b/granite-speech-4.1-2b-BF16.gguf --translate --target-language en samples/german.wav`	non-empty English transcript on non-English audio	PASS
Translate (En->X)	granite-speech-4.1-2b	English source audio + target language hint	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b/granite-speech-4.1-2b-BF16.gguf --translate --target-language de samples/jfk.wav`	non-empty non-English transcript on English audio	PASS ("und so, meine amerikaner, fragen sie nicht ...")
Keyword biasing	granite-speech-4.1-2b	hotword prompt	n/a	hotword preserved verbatim with position-aware decoding	SKIP - not exposed by runtime
Transcribe	granite-speech-4.1-2b-plus	explicit language hint	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b-plus/granite-speech-4.1-2b-plus-BF16.gguf --language en samples/jfk.wav`	non-empty plausible English transcript	PASS (granite-4 system-role template; transcript matches reference within one comma)
Transcribe	granite-speech-4.1-2b-plus	auto / no language hint	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b-plus/granite-speech-4.1-2b-plus-BF16.gguf samples/jfk.wav`	non-empty plausible transcript	PASS
Translate	granite-speech-4.1-2b-plus	n/a	n/a	runtime rejects `--translate` (`supports_translate=false`)	SKIP - not exposed by runtime (upstream model card lists -plus as ASR-only; the base granite-speech-4.1-2b is the AST variant. The fused LLM can be prompt-coerced to translate — the smoke produced "Ich bitte meine Mitbürger, ..." — but the converter advertises `stt.capability.translate=false`, so the dispatcher rejects the task.)
Word timestamps	granite-speech-4.1-2b-plus	structured per-word timestamps	`build/bin/transcribe-cli -m models/granite-speech-4.1-2b-plus/granite-speech-4.1-2b-plus-BF16.gguf --timestamps word samples/jfk.wav`	clean transcript + per-word t0/t1, parsed from the model's `[T:N]` end-of-word centisecond markers	PASS (model emits `[T:N]` markers, e.g. "_ [T:30] and [T:57] so [T:95] my [T:125] fellow [T:160] americans ..."; the runtime drops the `_` placeholders, unwraps the mod-1000 / 10 s rollover, and returns structured words. Verified against IBM's transformers reference with the model-card timestamps prompt; the earlier `[SS:N]` reading was an artifact of an incorrect prompt string.)
Speaker diarization	granite-speech-4.1-2b-plus	multi-speaker audio	n/a	`[Speaker N]:` tags preceding speaker turns in the text	SKIP - not exposed by runtime (user-deferred to later)
Incremental decoding	granite-speech-4.1-2b-plus	prefix_text continuation	n/a	transcript continues from supplied prefix	SKIP - not exposed by runtime

Shaw positional bias: skew path (shared with granite5_ctc)

granite::precompute_pos_rows replaced precompute_attention_dists, and shaw_block_attn is called with the pos_rows argument. Instead of materializing the full [head_dim, context_size, context_size] lookup per layer (at context_size=200, head_dim=128 that is 20 MB per layer) and feeding a batched GEMM with N = n_heads = 8, the encoder now gathers only the 2*context_size - 1 distinct relative offsets, does one fat GEMM against q, and rotates the result into key columns with conformer::rel_shift. See granite_conformer/shaw_attn.h.

This is valid because every entry of the reference's compute_attention_dists is a function of (query - key) alone, clamped or not. The identity was verified host-side in numpy before any C++ changed.

Correction recorded while doing this. The granite5_ctc port originally documented granite 4.x's Shaw index as having the OPPOSITE sign (key - query). That was wrong. granite 4.x writes dists[c * ctx + r] = clamp(c - r) with c the OUTER (query) index and r the inner (key) index; granite 5.0 writes dists[q * ctx + k] = clamp(q - k). Renaming makes them the same code, and the arrays are bit-identical for any (context_size, max_pos_emb). Both families now share one precompute_pos_rows formula.

Verified numerically neutral on all four variants: validate.py all output is identical to the pre-change build on every tensor, to every printed digit (granite-4.0-1b-speech 15/15, granite-speech-4.1-2b 15/15, granite-speech-4.1-2b-plus 10/15, granite-speech-4.1-2b-nar 2/15 — the last two fail identically before and after; see below).

Pre-existing validate failures (NOT caused by the skew change)

Both were confirmed by running validate.py all on the pre-change build and diffing: every failing tensor reports the same value before and after.

granite-speech-4.1-2b-plus: 10/15. Five decoder tensors fail (dec.token_emb, dec.audio_injected, dec.block.0.out, dec.block.20.out, dec.out_before_head). Every encoder tensor passes. dec.token_emb is a pure embedding lookup, and its first_diff is at element 20480 = row 10, so the prompt token ids diverge from the reference at position 10 rather than any arithmetic drifting. Consistent with the known add_generation_prompt handling on this variant. dec.logits_raw and dec.block.39.out still pass, which is why the transcript is right.
granite-speech-4.1-2b-nar: 2/15. Nine ENCODER tensors fail, starting at enc.block.0.post_ff1 (6.065e-02) and growing to enc.block.15.out (2.297e+01, mean 1.167e+00) and enc.ctc_logits (1.114e+01, mean 4.275e+00). Only enc.mel.in and enc.input_linear.out pass. The failure begins before attention runs in block 0, so it is upstream of anything Shaw-related. This one is large enough to warrant its own investigation.

Notes

The IBM-published ibm-granite/granite-4.0-1b-speech-GGUF strips the speech encoder; our port produces fused-stack GGUFs (encoder + projector + LM in one file).
Stage 4 WER sanity on LibriSpeech test-clean (BF16 ref-dtype):

Variant REF full 2620 REF 512 C++ 512 Δ (C++ - REF on 512)
granite-4.0-1b-speech 1.42% 1.24% 1.26% +0.02 pp
granite-speech-4.1-2b 1.35% 1.61% 1.35% -0.26 pp
granite-speech-4.1-2b-plus 1.48% 1.26% 1.29% +0.03 pp

All three pass (C++ within bootstrap CI of REF on the same 512 subset).
- 1b: 95% CIs overlap identically with REF; the 0.02 pp gap is 5 utterances where a single token's argmax flips a character on BF16 noise (e.g. "chiaroscuroists" -> "chiaroscurosts").
- 2b: looks "better than REF" entirely because of one utterance (1995-1836-0004) where the HF reference enters an "and missus vanderpool" repetition loop (reproduced on both MPS and CPU PyTorch) and never recovers; C++ also loops but breaks out one token sooner on BF16 noise. Excluding that utt, C++ and REF score identically at 1.085%.
- -plus: requires add_generation_prompt=True (assistant role marker appended) on the prompt. With False (the original Stage-2 dumper default) both REF and C++ produce 25-27 empty hypotheses on short test-clean clips, blowing up to a ~26% WER. With True the model opens the assistant turn explicitly and the empties disappear (1.29% on C++ 512). The Stage-2 dumper, the WER reference runner, and the C++ prompt builder were all updated to use True; the family-doc Capability Validation rows reflect that prompt. The reference dumps in build/validate/granite/granite-speech-4.1-2b-plus/jfk/ref/ were regenerated so the 15/15 tensor checks remain valid.
Upstream LibriSpeech test-clean WER targets (porting-7-wer gate, ref-dtype C++ must score <= REF + 0.01): the published 4.0-1b number is 1.42 (matches our REF full exactly); 4.1-2b is 1.33 (we measure REF full 1.35 — within bootstrap CI); 4.1-2b-plus published is 1.44 (we measure REF full 1.48 — within bootstrap CI). The granite_nar sibling is tracked separately.

Variant	REF full 2620	REF 512	C++ 512	Δ (C++ - REF on 512)
granite-4.0-1b-speech	1.42%	1.24%	1.26%	+0.02 pp
granite-speech-4.1-2b	1.35%	1.61%	1.35%	-0.26 pp
granite-speech-4.1-2b-plus	1.48%	1.26%	1.29%	+0.03 pp