GGUF Q8 Performance
July 25, 2026 ยท View on GitHub
This report compares tested 16-bit GGUF packages against Q8_0 GGUF packages on CUDA. In the measured routes, Q8_0 improves wall time by up to 1.53x and lowers peak VRAM by up to about 37% compared with the matching 16-bit GGUF package.
Notes:
- Offline session rows exclude the warmup request.
- "Speed vs real time" means generated or processed audio duration divided by wall time.
- "Q8 vs 16-bit" means 16-bit wall time divided by Q8 wall time; values above
1.0xmean Q8 was faster. qwen3_ttsuses the regeneratedq8_v2GGUF, which keeps speaker-sensitive tensors in 16-bit storage.higgs_audio_tts16-bit long-form usestext_chunk_size=256.
Summary
Q8 gives the clearest end-to-end wins on the larger AR-style models:
higgs_audio_tts: Q8 is 1.38x-1.53x faster on warmed requests and lowers peak VRAM from 9.1 GiB to 5.9 GiB.fish_audio: Q8 is 1.26x-1.34x faster on warmed requests and lowers peak VRAM from 11.8 GiB to 7.5 GiB.voxtral_realtime: Q8 is 1.31x-1.38x faster in offline ASR and lowers peak VRAM from 10.7 GiB to 7.6 GiB.
Smaller or already memory-light models still load and run with Q8, but the speed gain can be modest. Treat Q8 as a measured route choice, not a guaranteed win for every model.
Voxtral Q4_K Quick Check
Voxtral also has a q4_k GGUF package. In this quick CUDA path-test check,
Q4_K was smaller and faster than Q8_0 while producing effectively the same
transcripts. Four requests matched exactly; one differed only by capitalization
(Mother Nature vs mother nature).
| Route | Q8 RTF | Q4_K RTF | Q4_K vs Q8 |
|---|---|---|---|
| Offline short | 0.0862 | 0.0629 | 1.37x |
| Offline medium | 0.0643 | 0.0476 | 1.35x |
| Offline longer | 0.0576 | 0.0439 | 1.31x |
| Offline sampled | 0.0630 | 0.0500 | 1.26x |
| Streaming path | 0.1036 | 0.0904 | 1.15x |
TTS Offline Long-Lived Session
| Model | 16-bit speed vs real time | Q8 speed vs real time | Q8 vs 16-bit | 16-bit peak VRAM | Q8 peak VRAM |
|---|---|---|---|---|---|
pocket_tts | 68.5x-99.0x | 78.7x-102.0x | 1.01x-1.21x | 2060 MiB | 1932 MiB |
chatterbox | 6.2x-8.2x | 6.2x-8.7x | 1.01x-1.11x | 4333 MiB | 4309 MiB |
omnivoice | 7.2x-24.4x | 8.2x-29.0x | 1.08x-1.22x | 3250 MiB | 3157 MiB |
qwen3_tts | 5.3x-7.0x | 5.8x-7.9x | 1.01x-1.12x | 7873 MiB | 6397 MiB |
fish_audio | 2.5x-2.6x | 3.1x-3.4x | 1.26x-1.34x | 12093 MiB | 7669 MiB |
higgs_audio_tts | 6.1x-6.7x | 8.8x-10.1x | 1.38x-1.53x | 9326 MiB | 6024 MiB |
TTS Offline Long-Form
| Model | 16-bit speed vs real time | Q8 speed vs real time | Q8 vs 16-bit | 16-bit peak VRAM | Q8 peak VRAM |
|---|---|---|---|---|---|
pocket_tts | 82.6x | 85.5x | 1.04x | 2098 MiB | 2298 MiB |
chatterbox | 7.9x | 8.5x | 1.06x | 5033 MiB | 4554 MiB |
omnivoice | 38.6x | 43.7x | 1.13x | 3175 MiB | 3069 MiB |
qwen3_tts | 5.9x | 6.6x | 1.12x | 9412 MiB | 8138 MiB |
fish_audio | 2.5x | 3.3x | 1.27x | 12261 MiB | 9228 MiB |
higgs_audio_tts | 6.1x | 8.5x | 1.41x | 11878 MiB | 9129 MiB |
ASR Offline Long-Lived Session
| Model | 16-bit speed vs real time | Q8 speed vs real time | Q8 vs 16-bit | 16-bit peak VRAM | Q8 peak VRAM |
|---|---|---|---|---|---|
voxtral_realtime | 11.1x-12.5x | 14.7x-16.7x | 1.31x-1.38x | 10909 MiB | 7754 MiB |
nemotron_asr | 277.8x-384.6x | 285.7x-400.0x | 1.03x-1.12x | 5125 MiB | 4028 MiB |
ASR Streaming Long Audio
| Model | 16-bit server TTFT | Q8 server TTFT | 16-bit client TTFT | Q8 client TTFT | 16-bit speed vs real time | Q8 speed vs real time | 16-bit peak VRAM | Q8 peak VRAM |
|---|---|---|---|---|---|---|---|---|
voxtral_realtime | 207.308 ms | 179.896 ms | 550.526 ms | 530.558 ms | 4.7x | 5.4x | 12616 MiB | 8972 MiB |
nemotron_asr | 205.007 ms | 214.822 ms | 488.629 ms | 499.453 ms | 31.6x | 33.4x | 2816 MiB | 2497 MiB |