GGUF Models

August 4, 2026 · View on GitHub

audio.cpp can load audio.cpp-native GGUF checkpoints for model families that have a package spec. GGUF is a container for tensors and sidecar files; it is not a universal adapter for arbitrary llama.cpp or whisper.cpp GGUF files. The tensor names and embedded metadata still have to match the selected --family.

Package specs are maintained as model_specs/*.json. New GGUFs contain the selected spec in audiocpp.model_spec.* metadata, so a standalone GGUF does not depend on a model_specs directory or on the binary having that family compiled into its spec catalog.

Runtime resolution is deterministic:

explicit --model-spec-override
              |
              v
package spec embedded in the selected GGUF
              |
              v
compiled catalog (AUDIOCPP_DEPLOYMENT_BUILD=ON)
              |
              v
external model_specs/<family>.json discovery

An explicit override is useful for testing a modified layout without rebuilding or reconverting:

audiocpp_cli --inspect --family qwen3_asr --model /path/to/model.gguf \
  --model-spec-override /path/to/qwen3_asr.json

The override may also be a directory containing <family>.json. The server supports the same command-line option and a model_spec_override field either at the top level or inside an individual model entry. A per-model field takes precedence over the server-wide value.

Support And Test Status

Status labels:

Label	Meaning
`Done`	Package-spec refactor is in place for this family.
`No`	Package-spec refactor is not done, or the tested format is not usable.
`Skip (...)`	Package-spec refactor is intentionally skipped.
`Pass`	Covered by the path-test matrix with acceptable output.
`Pass (TTS + clone)`	Both no-reference TTS and reference-audio voice cloning run successfully.
`Pass (drift)`	Loads and runs, with known acceptable output drift.
`Pass (ASR match, drift)`	TTS output has similarity/frame drift but ASR transcript remains usable.
`No (...)`	Known unsupported, failing, or too much output drift.
`---`	Not tested in the current GGUF path-test matrix.

Family	Package-spec refactor	Safetensors tested after refactor	`orig` GGUF tested	16-bit GGUF tested	`q8_0` GGUF tested
`ace_step`	Done	Pass	---	Pass (drift)	No (planner sampling can fail)
`bs_roformer`	Done	Pass	---	---	Pass
`chatterbox`	Done	Pass	---	Pass (ASR match, drift)	Pass (ASR match, drift)
`citrinet_asr`	Done	Pass	---	---	Pass
`fish_audio`	Done	Pass	---	Pass	Pass
`fun_asr_nano`	Done	Pass	---	Pass	Pass
`glm_tts`	Done	Pass (TTS + clone)	---	---	Pass (ASR match, drift)
`heartmula`	Done	Pass	---	Pass (drift)	Pass (drift)
`higgs_audio_stt`	Done	Pass	---	Pass	Pass
`higgs_audio_tts`	Done	Pass	---	Pass	Pass
`htdemucs`	Done	Pass	---	Pass	Pass (drift)
`hviske_asr`	Done	Pass	---	---	Pass
`inflect_v2`	Done	Pass	Pass	---	---
`index_tts2`	Done	Pass	Pass	Pass (drift)	Pass (ASR match, drift)
`irodori_tts`	Done	Pass	---	Pass	Pass (ASR match, drift)
`kroko_asr`	Done	Pass	---	---	Pass
`marblenet_vad`	Bundled (tiny model)	Pass	---	---	---
`mel_band_roformer`	Done	Pass	---	Pass (drift)	Pass (drift)
`miocodec`	Done	Pass	Pass	Pass (drift)	Pass (drift)
`miotts`	Done	Pass	Pass	Pass (drift)	Pass (ASR match, drift)
`moss_tts_local`	Done	Pass	---	Pass	Pass (ASR match, drift)
`moss_tts_nano`	Done	Pass	---	Pass	Pass (ASR match, drift)
`nemotron_asr`	Done	Pass	---	Pass	Pass (minor filler drift)
`omnivoice`	Done	Pass	---	Pass (drift)	Pass (drift)
`outetts`	Done	Pass (TTS + clone)	---	---	Pass (TTS + clone)
`parakeet_tdt`	Done	Pass	Pass	Pass	Pass
`pocket_tts`	Done	Pass	---	Pass	Pass (drift)
`qwen3_asr`	Done	Pass	---	Pass	Pass
`qwen3_forced_aligner`	Done	Pass	---	Pass	Pass
`qwen3_tts` base	Done	Pass	Pass	Pass (ASR match, drift)	Pass (ASR match, drift)
`qwen3_tts` custom voice	Done	Pass	---	Pass (ASR match, drift)	Pass (ASR match, drift)
`qwen3_tts` voice design	Done	Pass	---	Pass (ASR match, drift)	Pass (ASR match, drift)
`rvc`	Done	---	---	Pass	---
`seed_vc`	Done	Pass	---	Pass (drift)	Pass (drift)
`silero_vad`	Skip (tiny model)	---	---	---	---
`sortformer_diar`	Done	Pass	---	Pass	Pass
`stable_audio`	Done	Pass	---	Pass (drift)	Pass (drift)
`supertonic`	Done	Pass	Pass	---	No (unsupported weight dtype)
`vevo2`	Done	Pass	Pass	Pass (drift)	No (mixed route drift; speech ASR match)
`vibevoice`	Done	Pass	---	Pass	Pass (drift)
`vibevoice_asr`	Done	Pass	---	Pass	Pass
`voxcpm2`	Done	Pass	Pass	Pass (ASR match, drift)	Pass (ASR match, drift)
`voxtral_realtime`	Done	Pass	---	Pass	Pass

Additional lower-bit checks:

Family	Format	Tested
`voxtral_realtime`	`q4_k`	Pass (quick CUDA check; transcripts match Q8 except one capitalization-only difference)

Q8 packaging notes:

chatterbox Q8 is intentionally mixed type. Graph-sensitive scalar, norm, bias, and side tensors stay in non-Q8 types while matmul-compatible weights are quantized.
pocket_tts Q8 keeps the four flow_lm.flow_net.time_embed.*.mlp.{0,2}.weight tensors in Q8 in addition to the default converter selection. conditioner.embed, cond_embed, and Mimi conv tensors are not forced to Q8 because tested outputs drifted or the current conv path casts quantized conv weights back to F32.
qwen3_tts Q8 should keep speaker-sensitive components in their original 16-bit type. The tested Base Q8 package quantizes the talker transformer and projections, talker code-predictor heads, and speech-tokenizer encoder/decoder projection or linear weights, while leaving the speaker encoder, lookup, and codebook-sensitive tensors unquantized. Quantizing those speaker-side tensors can produce long-form quality problems such as large silence.
voxtral_realtime also has a tested q4_k package. In a quick CUDA path check it was smaller and faster than Q8_0, while transcript output matched Q8_0 except for one capitalization-only difference.

Build The Converter

cmake --build build/debug --parallel --target audiocpp_gguf

Normal builds leave AUDIOCPP_DEPLOYMENT_BUILD off. Enable it when one binary must also carry fallback specs for safetensors packages or legacy GGUFs that predate embedded spec metadata:

cmake -S . -B build/deploy -DAUDIOCPP_DEPLOYMENT_BUILD=ON
cmake --build build/deploy --parallel

audiocpp_gguf always carries the conversion catalog. This is separate from the optional CLI/server deployment catalog, and keeps a copied converter executable usable when the source checkout and its model_specs directory are not present.

Check the converter interface:

audiocpp_gguf --help

Current shape:

audiocpp_gguf --input [namespace=]<weights> [--input namespace=<weights> ...] \
  --output <weights.gguf> \
  --type <orig|f16|bf16|q8_0|q2_k|q3_k|q4_k|q5_k|q6_k> \
  [--family <family>] \
  [--model-spec <json-or-directory>] \
  [--root <model-dir>] \
  [--sidecar <source>=<destination>] \
  [--bnb-nf4-type q8_0] \
  [--exclude-prefix <logical-prefix>] \
  [--keep-type <tensor-prefix>*=<type>] \
  [--overwrite] \
  [--no-sidecars] \
  [--allow-missing-model-spec]

audiocpp_gguf --inspect <model.gguf>

Convert A Single Tensor Source

Standalone conversion is the default. The converter embeds non-weight files recursively from the first tensor source's directory, or from --root when it is supplied. Use --root when the model has tokenizer, config, processor, or other non-weight files in a different model root. It also finds, validates, and embeds the package spec. Conversion fails before writing when the tensor namespaces or required sidecars do not match that spec.

audiocpp_gguf \
  --input /path/to/model/model.safetensors \
  --root /path/to/model \
  --output /path/to/model-gguf/model.gguf \
  --type f16 \
  --overwrite

Safetensors shard indexes are accepted directly:

audiocpp_gguf \
  --input /path/to/model/model.safetensors.index.json \
  --root /path/to/model \
  --output /path/to/model-gguf/model.gguf \
  --type q8_0 \
  --overwrite

Convert A Multi-Component Model

Use repeated namespaced --input entries when a model has multiple tensor components. The namespace must match the model's package spec.

audiocpp_gguf \
  --input model_weights=/path/to/model/model.safetensors \
  --input codec_weights=/path/to/model/codec/model.safetensors \
  --root /path/to/model \
  --output /path/to/model-gguf/model.gguf \
  --type f16 \
  --overwrite

Convert BitsAndBytes NF4 Sources

Some upstream packages store tensors as BitsAndBytes NF4 data in U8 safetensors plus helper tensors. Use --bnb-nf4-type q8_0 for these sources. The converter decodes the NF4 payload, uses the quant-state shape for GGUF metadata, re-quantizes the decoded weights to GGML Q8_0, and skips the BNB helper tensors from the output.

--keep-type only overrides the GGUF output type for normal tensors. It does not decode raw BNB NF4 U8 tensors by itself.

Use --exclude-prefix <logical-prefix> to omit a tensor subtree that the audio.cpp model does not load, for example an unused vision tower in a shared language-model checkpoint.

audiocpp_gguf \
  --input audio=models/Dramabox/dramabox-audio-components.safetensors \
  --input dit=models/Dramabox/dramabox-dit-v1.safetensors \
  --input gemma=models/gemma-3-12b-it-bnb-4bit/model.safetensors.index.json \
  --input silence=models/Dramabox/assets/silence_latent_frame.safetensors \
  --root build/debug/dramabox_gguf_sidecars_spec \
  --output models/Dramabox-GGUF/dramabox-q8_0.gguf \
  --type q8_0 \
  --bnb-nf4-type q8_0 \
  --exclude-prefix gemma/vision_tower \
  --family dramabox \
  --model-spec model_specs/dramabox.json \
  --overwrite

Add External Sidecars

Use --sidecar <source>=<destination> when a runtime file is needed but does not live under --root, or when it should be embedded at a different path inside the GGUF.

audiocpp_gguf \
  --input /path/to/model/model.safetensors.index.json \
  --root /path/to/model \
  --sidecar /path/to/shared/preprocessor_config.json=preprocessor_config.json \
  --output /path/to/model-gguf/model.gguf \
  --type q8_0 \
  --overwrite

If the default pipeline cannot find any sidecars, conversion fails instead of silently creating a tensor-only file. Supply the correct --root and any required external --sidecar mappings. Pass --no-sidecars only when you intentionally want a tensor-only container; place that GGUF and all package-spec-required sidecars together in one model directory when loading it. --no-sidecars does not remove the embedded package spec and does not disable build-time validation.

Package Spec Discovery During Conversion

The converter selects the first valid source at the highest available priority:

--model-spec <json-or-directory> (also accepted as --model-spec-override).
A spec object, JSON string, or relative path in the model's config.json.
model_spec.json or model_specs/*.json below the model root.
A discovered model_specs/*.json directory from the working directory upward.
The converter's bundled source catalog.

Higher-priority inputs are authoritative. If an explicit override, model-config spec, or local spec is present but does not match the tensor namespaces and required files, the converter reports that error instead of silently falling back to a lower-priority layout.

Use --family <family> to disambiguate models whose configuration does not identify the audio.cpp family. A model configuration can declare it directly:

{
  "audiocpp_family": "qwen3_asr",
  "audiocpp_model_spec": "model_spec.json"
}

audiocpp_model_spec may instead be a JSON string or a path relative to config.json. The nested forms audiocpp.family, audiocpp.model_spec, and audiocpp.package_spec are also accepted. The converter additionally recognizes known upstream model_type values.

--allow-missing-model-spec is an explicit escape hatch for creating a tensor archive that audio.cpp is not expected to load as a model. It is not recommended for deployable GGUFs.

Inspect And Run

Inspect the finished package before using it:

audiocpp_gguf --inspect /path/to/model-gguf/model.gguf

If the GGUF embeds all required sidecars, it can be passed directly as --model:

audiocpp_cli --task asr --family qwen3_asr --model /path/to/model-gguf/model.gguf --backend cuda --audio speech.wav

A directory is also accepted by supported package specs. It resolves to model.gguf when that name is present, otherwise to the single *.gguf inside it — so a downloaded package directory works under its release name without renaming anything:

audiocpp_cli --task tts --family qwen3_tts --model /path/to/model-gguf --backend cuda --text "Hello." --out out.wav
audiocpp_cli --task vc --family vevo2 --model models/Vevo2-GGUF --backend cuda --audio source.wav --voice-ref target.wav --out converted.wav

A directory holding several GGUFs and no model.gguf is ambiguous and is rejected with the candidates listed; pass one of them directly as --model, or keep a single GGUF per directory.

Compatibility summary:

Format	Where its package spec comes from	Other model files
Safetensors	Override, compiled deployment catalog, or external discovery	Required
New standalone GGUF	Embedded in GGUF	None
New tensor-only GGUF (`--no-sidecars`)	Embedded in GGUF	Required sidecars
Legacy GGUF without embedded spec	Compiled deployment catalog or external discovery	Depends on embedded sidecars

Compatibility with older binaries:

After PR #53 refers to tag release-0.3-gguf-v2, commit bf1ac678758aee4caafa7bb25fc0e6db9c25228f.

Build	GGUF package	Runtime context	Result
Before PR #53 (`14e9258`)	Legacy GGUF without embedded spec	Repo checkout or external `model_specs` visible	Pass
Before PR #53 (`14e9258`)	New standalone GGUF	Repo checkout or external `model_specs` visible	Pass
After PR #53 (`release-0.3-gguf-v2`), normal build	Legacy GGUF without embedded spec	Repo checkout or external `model_specs` visible	Pass
After PR #53 (`release-0.3-gguf-v2`), normal build	New standalone GGUF	Repo checkout or external `model_specs` visible	Pass
Before PR #53 (`14e9258`)	Legacy GGUF without embedded spec	No `model_specs` visible	Fail
Before PR #53 (`14e9258`)	New standalone GGUF	No `model_specs` visible	Fail
After PR #53 (`release-0.3-gguf-v2`), normal build	Legacy GGUF without embedded spec	No `model_specs` visible	Fail; use `--model-spec-override`, a deployment build, or external specs
After PR #53 (`release-0.3-gguf-v2`), normal build	New standalone GGUF	No `model_specs` visible	Pass
After PR #53 (`release-0.3-gguf-v2`), deployment build	Legacy GGUF without embedded spec	No `model_specs` visible	Pass through compiled package specs

Type Notes

Type	Meaning
`orig`	Preserve the original safetensors storage type where possible.
`f16`	Convert eligible tensors to FP16.
`bf16`	Convert eligible tensors to BF16. Useful for BF16 source models.
`q8_0`	Quantize eligible tensors to Q8_0; unsupported tensors remain in a backend-safe type.
`q2_k`/`q3_k`/`q4_k`/`q5_k`/`q6_k`	Lower-bit quantized formats. Treat as experimental per model and backend.

Quantized GGUF support is model- and route-specific. A model may load successfully but still drift in length, waveform similarity, or recognized text, so validate the exact route you plan to ship.

For measured 16-bit vs Q8 speed and peak VRAM results, see GGUF Q8 performance.