Voxtral Small 24B (2507)

June 27, 2026 · View on GitHub

Mistral's mistralai/Voxtral-Small-24B-2507 ported to transcribe.cpp. An offline audio-LLM: a Whisper-large-v3 bidirectional audio encoder (32 layers, d_model=1280, 20 heads) feeds a 4-frame-group projector (375 audio tokens per 30 s chunk) into a Mistral-Small-24B causal LM (40 layers, hidden_size=5120, intermediate_size=32768, GQA 32 q / 8 kv heads, NEOX RoPE, SwiGLU) via audio-token injection at the audio_token_id=24 positions in the prompt.

It is the larger sibling of Voxtral Mini 3B: the audio encoder, projector pattern, log-mel frontend, and tekken tokenizer are identical — only the text decoder is scaled up (Mistral-Small-24B in place of Ministral-3B).

What it's for

Offline speech-to-text and speech-to-text translation. Takes a 16 kHz mono WAV and produces a transcript via greedy decoding.

Transcription — auto language detection, or an explicit --language hint. Voxtral advertises English, French, German, Spanish, Italian, Portuguese, Dutch, and Hindi.
Translation — --translate --target-language <code> runs the mistral-common instruct template ("Translate this to {Language}.") to translate non-English speech into the target language's text.

See Mistral's model card for training data, intended use, and upstream evaluation.

Licensed Apache-2.0. Ported from upstream commit da5b424, pinned 2026-06-05.

Download

Quantization	Download	Size	WER (LibriSpeech test-clean)
BF16	Voxtral-Small-24B-2507-BF16.gguf	48.54 GB	1.56%
F16	Voxtral-Small-24B-2507-F16.gguf	48.55 GB	1.57%
Q8_0	Voxtral-Small-24B-2507-Q8_0.gguf	25.81 GB	1.56%
Q6_K	Voxtral-Small-24B-2507-Q6_K.gguf	19.94 GB	1.58%
Q5_K_M	Voxtral-Small-24B-2507-Q5_K_M.gguf	17.14 GB	1.60%
Q4_K_M	Voxtral-Small-24B-2507-Q4_K_M.gguf	14.30 GB	2.11%

WER measured on the full LibriSpeech test-clean split (2620 utterances) with the Whisper-style English text normalizer, batch size 8 on an NVIDIA A100 80 GB. The same-split HuggingFace transformers reference run (VoxtralForConditionalGeneration, BF16, greedy) lands at 1.57%, and the BF16 GGUF matches it at 1.56%.

Quick Start

cmake -B build
cmake --build build

# transcription (auto language)
build/bin/transcribe-cli \
  -m models/Voxtral-Small-24B-2507/Voxtral-Small-24B-2507-Q8_0.gguf \
  samples/jfk.wav

# transcription with an explicit language hint
build/bin/transcribe-cli \
  -m models/Voxtral-Small-24B-2507/Voxtral-Small-24B-2507-Q8_0.gguf \
  --language de samples/german.wav

# speech translation (non-English audio -> English text)
build/bin/transcribe-cli \
  -m models/Voxtral-Small-24B-2507/Voxtral-Small-24B-2507-Q8_0.gguf \
  --translate --target-language en samples/german.wav

If your audio is not already 16 kHz mono WAV, convert it first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

CLI flags:

--language <code> — BCP-47 hint. Omit for auto-detection.
--translate --target-language <code> — speech translation via the instruct template.
--batch-size <N> — batched offline transcription. Use N ≤ 8 for this model (see Notes).

Performance

Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over realtime in parentheses. Units: ms below 1 s, s above (2 decimal places).

Apple M4 Max

Backend	Sample	Q8_0	Q4_K_M
Metal	jfk (11.0s)	3.36 s (3.3×)	2.62 s (4.2×)
Metal	dots (35.3s)	11.20 s (3.2×)	8.95 s (3.9×)

A 24B is a GPU-class model; on Apple Silicon it runs at ~3–4× realtime on Metal (the 3B sibling is ~15–18×). CPU is impractical at this size and is not benchmarked. transcribe.cpp 96adddb.

Benchmark reproduction:

uv run scripts/bench/run.py \
  --models Voxtral-Small-24B-2507 \
  --quants q8_0,q4_k_m \
  --samples jfk,dots \
  --backends metal \
  --iters 3 --warmup 1 \
  --name voxtral-small-24b-2507-publication

Notes

Batched inference: use batch size ≤ 8. transcribe.cpp ships a true parallel run_batch() fast path for the Voxtral family, and it is WER-neutral (byte-identical to single-stream). At the 24B's memory footprint, however, a batch composed entirely of long (>30 s, multi-chunk) clips can exceed the compute headroom on an 80 GB GPU and fail to allocate — so the practical ceiling is batch size 8. Smaller batches and single-clip transcription are unaffected. The failure is loud (the run errors rather than silently producing a partial result), so an oversized batch never yields a quietly-wrong transcript.
BF16 and F16 require ~50 GB of memory; on an 80 GB GPU they run comfortably at batch ≤ 8.

Numerical Validation

This variant is validated end-to-end by word error rate against the HuggingFace transformers reference, not by a separate tensor-by-tensor sweep. The full-set BF16 WER (1.56%) matches the same-machine transformers reference (1.57%) within rounding on all 2620 test-clean utterances.

The family's per-tensor numerical correctness is established by the Voxtral Mini 3B sibling, which passes a strict CPU tensor-parity sweep against the reference. The 24B shares that exact architecture (same encoder, projector, fusion, RoPE, and tokenizer; only the decoder is scaled), and given its size it was accepted on the WER match rather than re-running tensor parity — a deliberate scoping decision.

Field	Value
Reference	HuggingFace `transformers` v4.57.6 (`mistralai/Voxtral-Small-24B-2507`)
Acceptance dataset	LibriSpeech `test-clean` (2620 utterances), Whisper English normalizer
Reference WER	1.57%
BF16 GGUF WER	1.56%
Tolerances (family)	`tests/tolerances/voxtral.json`