Forward map - parakeet

Reference: NVIDIA NeMo (ASRModel.from_pretrained -> EncDecRNNTBPEModel / EncDecCTCModelBPE / EncDecHybridRNNTCTCBPEModel) @ NeMo v2.7.x Closest in-tree analog: src/arch/parakeet/ itself — this stage extends the existing TDT v1 port to cover RNNT, CTC, the 1.1B FastConformer-XL geometry, and encoder.use_bias=true. No new in-tree analog is needed.

A compact map from the reference forward pass to the C++ port. Family-level because all 10 parakeet variants share a Conformer encoder; per-head differences (TDT vs RNNT vs CTC) and per-variant geometry deltas go in "Variant Notes" at the bottom.

Frontend

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
Mel / log-mel filterbank	model.preprocessor (AudioToMelSpectrogramPreprocessor -> FilterbankFeatures, n_fft=512, win=400, hop=160, hann_periodic, preemph=0.97, per-feature CMVN)	[T_mel, n_mels]	enc.mel.in	transcribe::MelFrontend constructed once at load() from stt.frontend.* KV	unchanged from existing parakeet

n_mels is 80 for every new variant except unified-en-0.6b (128). Existing tdt-0.6b-v2/v3 use 128. Frontend is otherwise identical across the family.

Encoder

Shared FastConformer encoder. The two existing geometry knobs are subsampling_factor (always 8) and n_mels (80 or 128). The new knobs introduced by Stage 4 are n_layers (17/24/42), d_model (512/1024), subsampling_channels (256), and use_bias (true/false).

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
Pre-encode subsampling (factor 8 = 3x stride-2 conv)	model.encoder.pre_encode (dw_striding)	[T_enc, d_model]	enc.pre_encode.out	conf::build_pre_encode over PreEncodeView. pre_encode_freq = subsampling_channels * (n_mels / 8) — 256·10=2560 (n_mels=80) or 256·16=4096 (n_mels=128). Final linear projects to d_model.	existing parakeet
Relative positional embedding	RelPositionalEncoding	[2*T_enc-1, d_model]	enc.pos_emb	host-built sin/cos table, uploaded as graph input	unchanged
Speaker-kernel injection (multitalker variants)	SpeakerKernelMixin._get_spk_kernel_hook_pre_layer (forward_pre_hook on encoder.layers[0])	[T_enc, d_model]	(folded into enc.block.0.out)	apply_spk_kernel_injection in encoder.cpp, on the block-0 input (post pre-encode + xscale): x += W3·relu(W0·x+b0)+b3 then x += W3_bg·relu(b0_bg)+b3_bg (bg input is all-zeros → constant broadcast). Gated on hp.has_spk_kernel (KV stt.parakeet.encoder.spk_kernel_layers); no-op on every other variant.	new — multitalker-parakeet-streaming-0.6b-v1 only
Conformer block (FF1 → rel-pos MHSA → ConvModule(BN) → FF2 → LN)	model.encoder.layers[i]	[T_enc, d_model]	enc.block.{0, n/2, n-1}.out	conf::build_conformer_block over BlockView. Bias slots populated only when enc_use_bias=true (11 biases per block: ff1.linear1/linear2, attn.q/k/v/out, conv.pw1/dw/pw2, ff2.linear1/linear2).	existing parakeet — extended for biases
Final encoder LN	implicit in layers[-1] (norm_out)	[T_enc, d_model]	enc.final	conf::named on the final tensor	unchanged
Prompt MLP (multilingual variants only)	model.prompt_kernel (NeMo EncDecRNNTBPEModelWithPrompt): concat(enc[d_model], one_hot(prompt_id)[num_prompts]) → Linear(d_model+P → prompt_hidden) → ReLU → Linear(prompt_hidden → d_model)	[T_enc, d_model]	enc.prompted (encoder-side) + dec.enc_out_prompted (decoder-side; same tensor, separate dump for drift attribution)	host-built one-hot broadcast over T frames, then two-layer MLP via ggml_mul_mat with bias adds and ggml_relu between. Loader reads stt.parakeet.prompt.* KVs and the four prompt.mlp.{0,2}.{weight,bias} tensors. When the KVs are absent the path is a no-op (English-only variants).	new — nemotron-3.5-asr-streaming-0.6b is the first variant exercising the prompt-conditioned RNN-T joint feed

Block-spot dump indices vary by encoder depth: layers in {0, n/2, n-1}. For n=17: {0, 8, 16}. For n=24: {0, 12, 23}. For n=42: {0, 21, 41}. The docs/porting/families/parakeet.md Stage-2 entry already lists these.

Decoder

Three head kinds dispatched on stt.parakeet.head_kind (default tdt when KV missing, for the legacy v2/v3 GGUFs). All three host-side implementations live in src/arch/parakeet/decoder.cpp.

TDT (v2, v3, tdt-1.1b, tdt_ctc-110m, tdt_ctc-1.1b)

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
Predictor embed	model.decoder.prediction.embed	[1, pred_hidden] (per token)	dec.embed.0	ggml_get_rows-equivalent (host fp32 lookup)	existing parakeet
Predictor LSTM (1 or 2 layers)	model.decoder.prediction.dec_rnn	[pred_hidden] per layer (h, c)	dec.lstm.{layer}.{h,c}.0	hand-rolled fp32 LSTM step in decoder.cpp	existing parakeet — pred_n_layers=1 first exercised by tdt_ctc-110m
Joint enc/pred projections + activation + out	model.joint (TDTJoint)	[vocab+1+num_extra] per (t, u)	dec.joint.0	fp32 matmul + relu/sigmoid/tanh + matmul on host	existing parakeet
TDT decode loop (greedy)	RNNTGreedyDecodeBatched with TDT mode	per-emit token + duration	(tokens)	decode_tdt_greedy	existing parakeet

RNNT (rnnt-0.6b, rnnt-1.1b, unified-en-0.6b)

Predictor + joint identical in shape to TDT, but the joint emits vocab+1 logits (no duration extra-output channels). Decode loop is the same shape minus duration:

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
Predictor embed	model.decoder.prediction.embed	[1, pred_hidden]	dec.embed.0	reuse TDT predictor mirror	TDT path above
Predictor LSTM	model.decoder.prediction.dec_rnn	[pred_hidden]	dec.lstm.{layer}.{h,c}.0	reuse TDT path	TDT path above
Joint	model.joint (RNNTJoint, no extras)	[vocab+1]	dec.joint.0	reuse TDT path with joint_num_extra_outputs=0. Allowed by existing hp validation.	TDT path above
RNNT decode loop (greedy)	RNNTGreedyDecodeBatched (no TDT mode)	per-emit token	(tokens)	decode_rnnt_greedy: blank → advance 1 frame, non-blank → emit + stay (max_symbols cap, default 10)	new — sibling of decode_tdt_greedy

CTC (ctc-0.6b, ctc-1.1b)

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
CTC head (1×1 Conv1d)	model.decoder.decoder_layers.0	[T_enc, vocab+1]	dec.ctc.logprobs (full), dec.ctc.logprobs.0 (frame 0)	head.ctc.weight @ enc[t] + head.ctc.bias per frame, then log_softmax. Computed host-side from the encoder readback.	new — minimal host head
CTC greedy decode (per-frame argmax + run-length collapse + drop blank)	model.decoding.greedy_search (CTC)	token sequence	(tokens)	decode_ctc_greedy	new

Generation / KV Path

Parakeet has no autoregressive transformer KV cache. The TDT/RNNT predictor LSTM has a step-pair (h, c) per layer that is committed on every non-blank emit and rolled back on blank — host-managed in decoder.cpp. CTC has no recurrent state. The Stage 4 mid-generation-gate-tensor requirement (decoder KV correctness past n_past>0) does not apply to this family — there is no transformer KV cache to mis-index.

Capabilities And Language Controls

Capability	Reference behavior	C++ API behavior	Family-doc Capability Validation row
Transcribe (en)	transcribe() returns greedy text	transcribe-cli -m <gguf> --language en <wav> produces non-empty English transcript	one row per variant
Punctuation / casing (tdt_ctc-1.1b, tdt_ctc-110m, unified-en-0.6b)	upstream training data preserves PnC	non-empty transcript with capital letters and ,.?!	one row per PnC variant
Streaming (unified-en-0.6b)	shared offline+streaming weights	n/a — offline only in v1	ACCEPTED GAP — streaming infra deferred
Word timestamps	NeMo word-aligner	derived host-side from emit-frame indices (TDT/RNNT) or per-frame argmax (CTC)	PASS — same code path as the existing v2/v3 word-timestamp gate, no per-variant differences
Translation, lang-detect, VAD	not advertised	n/a	n/a

Deviations From Closest Analog

• head_kind dispatch: same family handler covers TDT (predictor+joint+durations), RNNT (predictor+joint, no durations), and CTC (single 1×1 conv head). Loader reads stt.parakeet.head_kind (string KV); legacy v2/v3 GGUFs lack the KV and default to "tdt". Predictor/joint/tdt-durations KV reads are conditional on the resolved head_kind so a CTC GGUF does not fail at "predictor.hidden missing".
• encoder.use_bias per-variant: existing v2/v3 ship with use_bias=false (every linear/conv biased); the 8 new variants ship with use_bias=true, contributing 11 biases per block. The shared transcribe::conformer::BlockView already exposes nullable bias slots; loader populates them only when the hparam is true.
• n_mels=80 path: every new variant except unified-en-0.6b uses 80-bin mels (vs 128 on v2/v3/unified-en). The pre-encode chain handles arbitrary even multiples of 8 mels — the existing "only 8/128 implemented" guard in encoder.cpp was a safety check from Phase 4 step 3a, now removed.
• CTC head storage: a single (vocab+1, d_model, 1) conv tensor flattened to head.ctc.weight + (vocab+1,) head.ctc.bias in the GGUF. Host decoder mirrors only these two tensors when head_kind=ctc; predictor and joint mirrors stay empty.
• TDT-CTC hybrids (tdt_ctc-110m, tdt_ctc-1.1b): the upstream checkpoints carry both heads, but the converter emits TDT-only (the auxiliary CTC head is silently dropped — pure CTC variants cover that path). At Stage 4 these load through the standard TDT path with head_kind="tdt".
• Streaming attention (att_context_style = "chunked_limited"): introduced by nemotron-speech-streaming-en-0.6b. NeMo's cache-aware streaming model trains with chunked attention but keeps the full RelPositionalEncoding (pos_emb buffer at 2T-1, not the shortened LocalAttRelPositionalEncoding size). The "regular" style is the existing path used by every other variant. Loader reads stt.parakeet.encoder.att_context_style as optional with default "regular" so legacy GGUFs are unaffected; only att_context_style == "chunked_limited" selects the new mask, computed as chunk_size = att_context_right + 1, left_chunks = att_context_left / chunk_size (every k-frame whose chunk index is in [q_chunk - left_chunks, q_chunk] is allowed). The mask is built host-side and threaded as a graph input that broadcasts across heads; added to matrix_bd before flash_attn_ext.
• Causal depthwise conv (conv_context_left, conv_context_right): streaming-friendly Conformer convolution. NeMo's offline default is centered (k-1)/2 on both sides; nemotron-streaming uses [k-1, 0] (left=8, right=0 for k=9) so the depthwise convolution does not look ahead. Loader reads both as optional with default -1; when -1, the conformer uses centered (k-1)/2. Otherwise depthwise conv padding becomes (left, right).
• Speaker-kernel FF injection (spk_kernel_layers, multitalker): introduced by multitalker-parakeet-streaming-0.6b-v1 (NeMo EncDecMultiTalkerRNNTBPEModel / SpeakerKernelMixin). An always-on forward_pre_hook on encoder.layers[0] injects a per-speaker FF kernel and a background FF kernel into the block-0 input; it fires unconditionally, so it is load-bearing for the in-scope single-speaker path, not just multi-speaker decoding. In single-speaker mode it reduces to x += W3·relu(W0·x+b0)+b3 (spk, per frame) then x += W3_bg·relu(b0_bg)+b3_bg (bg input masked to zeros → a per-channel constant broadcast over T). Loader reads stt.parakeet.encoder.{spk_kernel_type,spk_kernel_layers,add_bg_spk_kernel} (optional; absent → no-op) and binds enc.spk_kernel.0.{0,3}.{weight,bias} + enc.bg_spk_kernel.0.{0,3}.{weight,bias}. Only layer-0 injection is implemented (the checkpoint ships spk_kernel_layers=[0]); the loader rejects any other layer index. Applied in both the offline and the cache-aware streaming encoder graphs.
• Variable-length batched pre-encode masking (causal variants) — found + fixed during the multitalker Stage 4 via tensor bisection of batched-vs-single encoder output. Two shared-code bugs in the offline var-len batched path, both specific to the cache-aware streaming (causal chunked_limited) variants and invisible to same-length / text-only parity:
  1. Off-by-one frame count — model.cpp::pre_encode_t_out used the non-causal subsample formula (in-1)/2+1; causal pre-encode needs in/2+1. Undercounted real_tenc (dropped the last encoder frame → truncated hyps) and mis-sized the pad masks. Fixed by making pre_encode_t_out causal-aware (mirrors weights.cpp::pre_encode_F_prime).
  2. Unmasked causal subsampling — masked subsampling was gated var_len_masks && !causal_pre_encode, so for causal variants the zero-padded mel tail passed through conv0, whose bias made it relu to a NON-zero constant; the causal convs' right-context (right = stride-1) then pulled that constant into the last valid encoder frame (a large boundary-frame drift vs single-shot). Fixed by enabling masked subsampling for the causal path too (the per-stage valid lengths are now causal-aware, so the same masks are correct). Result: batched var-len encoder output is bit-exact (0.0) to single-shot on the production flash path across arbitrary length mixes; LibriSpeech test-clean batch-8 WER matches batch-1 exactly (2.19%). Also fixes the sibling nemotron-3.5 / nemotron-speech-streaming (0.098 → 0.0).
  3. Manual-path all-masked-query NaN — in a var-len batch a padded query row (past the real length) can have its entire chunked-attention window fall on padded keys; with a -inf key-pad sentinel the manual (TRANSCRIBE_NO_FLASH) softmax computes 0/0 = NaN, which then poisons real frames at the next layer via 0*NaN. Fixed in transcribe-batch-util.cpp::fill_keypad_mask by using a large FINITE negative sentinel (-1e30) instead of -inf: bit-exact with -inf for any row that has a non-masked key (expf underflows to exactly 0), casts to -inf in F16 so flash is unchanged, but makes a fully-masked row softmax to a finite value instead of NaN. Manual diverse-length batched output now matches single-shot to f32 reduction-order noise (~1e-7); flash is exactly 0.0.
• LayerNorm in conv module (conv_norm_type = "layer_norm"): streaming-stable replacement for BatchNorm. Same tensor names (conv.bn.weight / conv.bn.bias) but the running_mean / running_var tensors are absent and the per-batch statistics are computed online (it really is a LayerNorm over d_model, not a fused affine). Loader reads stt.parakeet.encoder.conv_norm_type as optional with default "batch_norm"; when "layer_norm", the BN-fusion step is skipped and conv_module applies an unfused LayerNorm using the same scale/bias tensors.

Variant Notes

• tdt-0.6b-v2, tdt-0.6b-v3 (existing baseline): 24 layers, d_model=1024, n_mels=128, use_bias=false, TDT head, pred_n_layers=2. v3 multilingual (25 languages) — capability difference only; same forward graph.
• tdt-1.1b: 42 layers, d_model=1024, n_mels=80, use_bias=true, TDT head, pred_n_layers=2. First exercise of the 1.1B FastConformer-XL geometry, the 80-mel pre-encode, and the encoder bias path.
• tdt_ctc-1.1b: 42 layers, d_model=1024, n_mels=80, use_bias=true, TDT head, pred_n_layers=2. Same encoder geometry as tdt-1.1b, but only variant in the family using local attention (att_context_size=[128,128], NeMo's LocalAttRelPositionalEncoding). Drives the pos_emb buffer length (257 instead of 2T-1) and adds a -INF band pad before rel_shift so out-of-window keys drop out of softmax.
• tdt_ctc-110m: 17 layers, d_model=512, n_mels=80, use_bias=true, TDT head, pred_n_layers=1. First family member with 1-layer predictor and the smaller "Medium" Conformer config (d_model=512, channels=256).
• rnnt-0.6b: 24 layers, d_model=1024, n_mels=80, use_bias=true, RNNT head, pred_n_layers=2. First exercise of the RNNT decode loop (no durations).
• rnnt-1.1b: 42 layers, d_model=1024, n_mels=80, use_bias=true, RNNT head, pred_n_layers=2. Sibling of rnnt-0.6b on the 1.1B encoder.
• unified-en-0.6b: 24 layers, d_model=1024, n_mels=128 (the only new variant that keeps the v2/v3 mel geometry), use_bias=true, RNNT head, pred_n_layers=2. Streaming-shared weights but offline-only in v1.
• ctc-0.6b: 24 layers, d_model=1024, n_mels=80, use_bias=true, CTC head. First exercise of CTC head loading and CTC greedy decode.
• ctc-1.1b: 42 layers, d_model=1024, n_mels=80, use_bias=true, CTC head. Sibling of ctc-0.6b on the 1.1B encoder.
• nemotron-speech-streaming-en-0.6b: 24 layers, d_model=1024, n_mels=128, use_bias=false, RNNT head, pred_n_layers=2. First variant exercising att_context_style="chunked_limited", causal depthwise conv (conv_context_left=8, conv_context_right=0 for kernel=9), conv_norm_type="layer_norm", and fe_normalize="none" (NeMo's normalize="NA" no-op, canonicalised at convert time). Frontend feature stats (per-feature mean/std) are baked into training rather than computed at inference, so the C++ mel frontend emits unnormalised log-mel and the loader accepts normalize="none" alongside "per_feature". Streaming runtime knobs (lookahead, encoder chunk size, decoder context) are deferred to the streaming bring-up — offline-only in v1, same as unified-en-0.6b.
• multitalker-parakeet-streaming-0.6b-v1: 24 layers, d_model=1024, n_mels=128, use_bias=false, RNNT head, pred_n_layers=2. Same cache-aware streaming FastConformer as nemotron-speech-streaming-en-0.6b (chunked_limited [70,13], causal depthwise conv, layer_norm conv, fe_normalize="none"); first and only variant exercising the always-on layer-0 speaker-kernel FF injection (see the Deviations bullet). Ported in SINGLE-SPEAKER mode; the full multitalker path (external Sortformer diarization, N-instance orchestration, SegLST) is OUT OF SCOPE (see the multitalker integration brief). The +8 speaker-kernel tensors are the only weight-set delta from the English predecessor.
• nemotron-3.5-asr-streaming-0.6b: 24 layers, d_model=1024, n_mels=128, use_bias=false, RNNT head, pred_n_layers=2. Same FastConformer geometry as the English predecessor; first variant exercising the multilingual prompt-conditioning MLP (EncDecRNNTBPEModelWithPrompt, num_prompts=128, prompt_hidden read from the GGUF; encoder output passes through prompt_kernel before the RNN-T joint sees it). Left context shrinks 70 → 56 frames in the chunked_limited mask (training-time att_context_size=[56,13]); the runtime mask code reads the GGUF KV, so this is a data-only delta. Vocab grows 1024 → 13087 with 39 explicit <lang-XX> SPM tag tokens used by target_lang=auto. Auxiliary CTC head present in the upstream checkpoint but dropped at convert time per family Open-Decision #1.