Whisper

OpenAI's Whisper family ported to transcribe.cpp. All twelve OpenAI checkpoints — tiny, base, small, medium, large, large-v2, large-v3, large-v3-turbo, plus the four English-only *.en siblings — share the same encoder-decoder transformer architecture and 30-second windowing, so most of the porting and runtime contract is the same across variants. Pick a size based on the accuracy/cost tradeoff you want; pick a .en variant only if you know the audio is English.

For the architecture deep-dive, validation contract, and porting notes, see the family doc at docs/porting/families/whisper.md.

Choosing a variant

• English-only audio? Prefer the .en checkpoint at your chosen size. They are typically a touch more accurate than the multilingual sibling and slightly cheaper to decode (no language-detection step). They cannot transcribe other languages and cannot translate.
• Other languages, or auto-detect across languages? Use the multilingual checkpoints (no .en). They cover 99 languages (100 in the v3 family, which adds Cantonese), do automatic language identification, and can produce English translations of non-English audio when invoked with task=translate.
• Throughput vs accuracy. WER drops as you go up the size ladder, but decode latency scales with parameter count. tiny and base run in near-realtime on CPU; large typically wants Metal or a recent CUDA GPU. large-v3-turbo is large-v3 quality with a much smaller decoder — the best accuracy/speed tradeoff for most multilingual workloads.
• v3 family quirks. large-v3 and large-v3-turbo use a 128-bin mel input (the rest use 80) and add a Cantonese (yue) language token. The reference dtype shipped is F16, not F32 — they were released in F16 upstream, so transcribe.cpp follows suit.

All variants

WER is on LibriSpeech test-clean for the Q8_0 preset (the default recommended quant), measured by transcribe.cpp's WER pipeline with segment timestamps enabled. See each per-variant doc for the full quant matrix (F32/F16/Q8_0/Q6_K/Q5_K_M/Q4_K_M) and a discussion of how our numbers compare to OpenAI's self-reported figures. Numbers come from single Metal-backed runs; Metal's non-deterministic parallel reductions add ~0.1pp run-to-run variance on the noise floor.

Variant	Languages	Q8_0 size	WER (Q8_0)	Doc
whisper-tiny	99 + auto-detect	44 MB	7.53%	whisper-tiny.md
whisper-tiny.en	English only	44 MB	5.72%	whisper-tiny.en.md
whisper-base	99 + auto-detect	81 MB	5.12%	whisper-base.md
whisper-base.en	English only	81 MB	4.16%	whisper-base.en.md
whisper-small	99 + auto-detect	257 MB	3.33%	whisper-small.md
whisper-small.en	English only	257 MB	3.09%	whisper-small.en.md
whisper-medium	99 + auto-detect	793 MB	2.64%	whisper-medium.md
whisper-medium.en	English only	793 MB	2.72%	whisper-medium.en.md
whisper-large	99 + auto-detect	1.55 GB	2.74%	whisper-large.md
whisper-large-v2	99 + auto-detect	1.55 GB	2.65%	whisper-large-v2.md
whisper-large-v3	100 + auto-detect	1.55 GB	1.82%	whisper-large-v3.md
whisper-large-v3-turbo	100 + auto-detect	845 MB	2.01%	whisper-large-v3-turbo.md

Pre-built GGUFs for every variant and quant are hosted under handy-computer on Hugging Face; each per-variant doc has direct download links.

Input limits

No practical per-call length limit (transcribe_capabilities.max_audio_ms == 0): Whisper slices long audio into 30-second windows internally and stitches the results, so you can pass arbitrarily long recordings. See the input-length contract.

Quick start

Pick a variant and run:

cmake -B build
cmake --build build

build/bin/transcribe-cli \
  -m models/whisper-base.en/whisper-base.en-Q8_0.gguf \
  samples/jfk.wav

The repo doesn't ship the GGUFs — pull them from the corresponding handy-computer/<variant>-gguf repo on Hugging Face, or convert from the upstream OpenAI checkpoint via the per-variant doc's reproduction section.

Capabilities

All Whisper variants support:

• Transcription of 16 kHz mono WAV input.
• Long-form audio via 30-second chunked decoding with the prev-context window assembly described in the family doc.
• Segment timestamps — the finest granularity the library emits for Whisper (max_timestamp_kind = segment). Word-level timestamps are not currently exposed.
• Translation (any supported language → English) on multilingual checkpoints — .en variants are transcribe-only.

What's not supported (consistent across the family): real-time streaming (whisper is not streaming-first; chunked 30-second windows only), VAD, speaker diarization. See the family doc for the full runtime contract.