audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CUDA, HIP/ROCm, Vulkan, Metal, and CPU backends behind shared CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project.

News

2026-06-25 to 2026-07-08: audio.cpp grew from the first released model wave into broad TTS, ASR, music generation, source separation, VAD, diarization, codec, and voice-conversion coverage, with VibeVoice 1.5B/7B, LoRA adapter loading, initial streaming support, and major CUDA Conv1DTransp speedups.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, ASR speech recognition, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, Music music/song generation, SFX sound effects, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags: safetensors is the default model loading path. GGUF 16/Q8/Q4 means those GGUF precision or quantization paths are tested; GGUF Q8 means only q8_0 is tested; GGUF F32 means the original-F32 GGUF path is tested. See docs/gguf.md for precision/status details. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path.

Family	Task	Lang	Variants	Runtime
ace_step	Music, Edit	50+ langs	ACE-Step 1.5 Turbo and Base with acestep-5Hz-lm-1.7B	GGUF 16
bs_roformer	Sep	lang agnostic	BS-RoFormer vocal separation checkpoints	GGUF Q8
chatterbox	TTS, Clone, VC	ar, da, de, el, en, es, fi, fr, hi, it, ko, ms, nl, no, pl, pt, sv, sw, tr	Chatterbox with 0.5B backbone	GGUF 16/Q8
confucius4_tts	Clone	zh, en, ja, ko, de, fr, es, id, it, th, pt, ru, ms, vi	Confucius4-TTS multilingual voice cloning	GGUF F32, Stream
citrinet_asr	ASR	en	Citrinet-256	GGUF Q8
dramabox	TTS, Clone	en	DramaBox expressive TTS and voice cloning	GGUF Q8
fish_audio	TTS, Clone, Ctrl	auto, en, zh	Fish Audio S2 Pro	GGUF 16/Q8
fun_asr_nano	ASR	auto, zh, en, ja	Fun-ASR-Nano-2512	GGUF 16/Q8
heartmula	Music	zh, en, ja, ko, es	HeartMuLa-oss-3B with HeartCodec-oss	GGUF 16/Q8
higgs_audio_stt	ASR	en	Higgs Audio v3 STT	GGUF 16/Q8, Stream
higgs_audio_tts	TTS, Clone, Ctrl	auto	Higgs Audio v3 TTS 4B	GGUF 16/Q8
htdemucs	Sep	lang agnostic	HTDemucs, HTDemucs_ft	GGUF 16/Q8
hviske_asr	ASR	da	Hviske v5.3	GGUF Q8
marblenet_vad	VAD	lang agnostic	MarbleNet VAD	Bundled
mel_band_roformer	Sep	lang agnostic	Mel-Band RoFormer MLX vocal separation variants	GGUF 16/Q8
miocodec	Codec, VC	lang agnostic	MioCodec v2, 25 Hz, 44.1 kHz	GGUF 16/Q8
miotts	TTS, Clone	en, ja	MioTTS-1.7B	GGUF 16/Q8
omnivoice	TTS, Clone, Design, Ctrl	646+ langs	OmniVoice, Qwen3-0.6B based	GGUF 16/Q8, Stream
pocket_tts	TTS, Clone	en, de, it, pt, es	PocketTTS-100M	GGUF 16/Q8
nemotron_asr	ASR	100+ ASR prompt codes incl. auto	Nemotron 3.5 ASR Streaming 0.6B	GGUF 16/Q8, Stream
qwen3_asr	ASR	zh, en, yue, ar, de, fr, es, pt, id, it, ko, ru, th, vi, ja, tr, hi, ms, nl, sv, da, fi, pl, cs, fil, fa, el, ro, hu, mk	Qwen3-ASR-0.6B, Qwen3-ASR-1.7B-hf	GGUF 16/Q8, Stream
qwen3_forced_aligner	Align	zh, yue, en, de, es, fr, it, pt, ru, ko, ja	Qwen3-ForcedAligner-0.6B	GGUF 16/Q8
qwen3_tts	TTS, Clone, Design, Ctrl	zh, en, fr, de, it, ja, ko, pt, ru, es	Qwen3-TTS-12Hz-0.6B-Base, Qwen3-TTS-12Hz-1.7B-Base, Qwen3-TTS-12Hz-1.7B-CustomVoice, Qwen3-TTS-12Hz-1.7B-VoiceDesign	GGUF 16/Q8
rvc	VC	lang agnostic	RVC F16 GGUF with packaged v1/v2 voices and optional retrieval blending	GGUF 16
seed_vc	VC	lang agnostic	SeedVC XLS-R + HiFT, SeedVC Whisper-small + BigVGAN	GGUF 16/Q8
silero_vad	VAD	lang agnostic	Silero VAD	Bundled, Stream
sortformer_diar	Diar	en	Sortformer-4spk-v1	-
stable_audio	Music, SFX, Edit	en	Stable Audio 3 Small Music, Stable Audio 3 Small SFX, Stable Audio 3 Medium	GGUF 16/Q8
vevo2	TTS, Music, VC, Edit	en, zh	Vevo2 with Qwen2.5-0.5B AR model	GGUF 16
vibevoice	TTS, Dialogue	en, zh	VibeVoice-1.5B, VibeVoice-7B	GGUF 16/Q8
vibevoice_asr	ASR	auto	VibeVoice ASR	GGUF 16/Q8
voxtral_realtime	ASR	auto	Voxtral-Mini-4B-Realtime-2602	GGUF 16/Q8/Q4, Stream
voxcpm2	TTS, Clone, Design, Ctrl	ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh	VoxCPM2-2B, 48 kHz	GGUF 16/Q8, Stream
index_tts2	TTS, Clone, Ctrl	zh, en	IndexTTS-2	GGUF 16/Q8
irodori_tts	TTS, Clone, Design, Ctrl	ja	Irodori-TTS-v4-Small, Irodori-TTS-500M-v3, Irodori-TTS-600M-v3-VoiceDesign	GGUF 16/Q8
moss_tts_nano	TTS, Clone	auto	MOSS-TTS-Nano-100M	GGUF 16/Q8
moss_tts_local	TTS, Clone, Ctrl	auto, optional language hint	MOSS-TTS-Local-Transformer-v1.5	GGUF 16/Q8
supertonic	TTS	en, ko, ja, ar, bg, cs, da, de, el, es, et, fi, fr, hi, hr, hu, id, it, lt, lv, nl, pl, pt, ro, ru, sk, sl, sv, tr, uk, vi, na	Supertonic 3	GGUF F32, Stream

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See docs/community_models/models.md for community-model expectations and current entries.

Family	Task	Lang	Runtime	Contributor	What They Added
glm_tts	TTS, Clone	zh, en	GGUF	Mirek @mirek190	GLM-TTS zero-shot synthesis and voice cloning support
inflect_v2	TTS	en	GGUF FP32	Jan @JanWerder	Inflect Micro v2 and Nano v2 native offline synthesis
kroko_asr	ASR	de, en, es, fr, it, he, nl, pt, sv, tr	Safetensors, GGUF Q8	Mirek @mirek190	Kroko Community ASR native offline/streaming Zipformer2/RNN-T transcription with word timestamps
moss_tts_local	TTS, Clone, Ctrl	auto, optional language hint	GGUF	@justinjohn0306	MOSS-TTS-Local Transformer v1.5 support
outetts	TTS, Clone	en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk	GGUF	Mirek @mirek190	Llama-OuteTTS-1.0-1B TTS and voice cloning support
parakeet_tdt	ASR	auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk	GGUF F32/16/Q8, Stream	@dleiferives	Parakeet-TDT 0.6B v3 offline, long-form, and buffered-streaming ASR support
vietneu_tts	TTS, Clone	vi, en	GGUF	Phuoc @phuocnguyen90	VieNeu-TTS-v3-Turbo TTS and voice cloning support

Docker

Docker CUDA and CPU images are available for both CLI and server use. See docker.md for available images, build commands and working Docker examples.

Model Manager and GGUF Downloads

Use tools/model_manager_v2.py for normal model downloads. It reads model_specs/*.json and installs the default package for each family, preferring ready-to-use GGUF packages when they are available.

The old safetensors/converter catalog has been renamed to tools/model_manager_deprecated.py. Use it only for legacy model layouts that have not moved to spec-backed GGUF packages yet.

GGUF downloads:

• Released model packages: audio-cpp/audio.cpp-gguf
• Community model package: mirek190/audio.cpp

See the Model Manager guide for model-manager usage and package notes.

WebUI

audio.cpp includes a Gradio WebUI for trying local models from the browser, managing downloads, and running common TTS/ASR/audio workflows without writing CLI commands.

The WebUI lives in webui/. See webui/README.md for setup, launch commands, and model-download notes.

Huge thanks to @kigner for the original audio.cpp-webui, and to @patrickjchen for porting and integrating it into audio.cpp.

Prebuilt Binaries

• Windows (CUDA / CPU): official packages on the Releases page.
• Windows (HIP/ROCm, AMD GPUs): community-maintained packages with the ROCm runtime bundled — no HIP SDK installation required. Published from @IIIIIllllIIIIIlllll's fork Releases in two tracks: ROCm 6.4 (full coverage incl. RX 7600 / gfx1102) and ROCm 7.1 (recommended for RDNA4). Version numbers follow the upstream releases; see docs/build/windows-hip-distribution.md for details.

Build

OS	Requirements
Linux	GCC 13 or newer, CMake, plus the backend toolchain for the build you want: NVIDIA CUDA Toolkit for CUDA, Vulkan SDK for Vulkan, ROCm for HIP
Windows	Visual Studio Build Tools 2022 or newer with C++ desktop workload, MSVC x64 compiler, Windows SDK, CMake, Ninja, MSVC OpenMP components; official NVIDIA CUDA Toolkit for CUDA builds, AMD HIP SDK for HIP builds
macOS	Xcode or Xcode Command Line Tools with the Metal compiler available through xcrun

Homebrew Install

On macOS, audio.cpp can be installed from the Homebrew tap:

brew tap 0xShug0/audio-cpp
brew trust 0xShug0/audio-cpp
brew install audio-cpp

For Nix and NixOS builds, see docs/build/nixos.md.

Composite Builds

Composite builds let you compile only the model families you need. full is the default and is what release/Docker builds should use. custom registers only the requested loaders while still linking required internal dependencies; core builds the runtime without the optional model-family set.

The helper scripts expose this as --model-set and --models on Linux/macOS, and -ModelSet and -Models on Windows:

scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli

Direct CMake builds use the same underlying variables:

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_tts,pocket_tts,qwen3_asr
cmake --build build/debug --target audiocpp_cli -j 8

Linux Build

Use the Linux helper script for CPU, CUDA, Vulkan, or HIP builds:

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server

The script writes to aligned build directories such as build/linux-cuda-release, build/linux-vulkan-release, build/linux-hip-release, and build/linux-cpu-release.

Composite examples:

scripts/build_linux.sh --backend cuda --model-set full --target audiocpp_cli
scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli
scripts/build_linux.sh --backend cpu --model-set core --target audiocpp_cli

For portable CPU kernels on machines where native ISA flags are not suitable:

scripts/build_linux.sh --backend cuda --native-cpu OFF --target audiocpp_cli --target audiocpp_server

For deployment builds with compiled package specs:

scripts/build_linux.sh --backend cuda --deployment-build --target audiocpp_cli --target audiocpp_server

For direct CMake commands, see docs/build/linux.md.

Windows Build

Use the Windows PowerShell build script:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

Common presets:

.\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models "qwen3_tts,pocket_tts,qwen3_asr" -Target audiocpp_cli

From cmd.exe, use the wrapper:

scripts\build_windows.cmd

For deployment builds with compiled package specs:

.\scripts\build_windows.ps1 -DeploymentBuild -Target audiocpp_cli

For requirements, CPU profiles, CUDA packaging, and release zips, see docs/build/windows.md.

Metal Build

On macOS, use the Metal helper script to build against ggml's Metal backend:

scripts/build_metal.sh --target audiocpp_cli

The script configures build/macos-metal-release by default, enables ENGINE_ENABLE_METAL=ON, disables CUDA and Vulkan, embeds the Metal shader library, and builds static libraries plus the requested target.

Useful variants:

scripts/build_metal.sh --target audiocpp_server
scripts/build_metal.sh --build-type Release --archs arm64 --target audiocpp_cli
scripts/build_metal.sh --model-set custom --models qwen3_tts,pocket_tts --target audiocpp_cli
scripts/build_metal.sh --with-tests --target audio_dsp_test
scripts/build_metal.sh --openmp auto --target audiocpp_cli
scripts/build_metal.sh --native-cpu OFF --target audiocpp_cli
scripts/build_metal.sh --deployment-build --target audiocpp_cli

The built CLI is written to:

build/macos-metal-release/bin/audiocpp_cli

HIP/ROCm Build

On Linux and Windows, HIP builds compile ggml's CUDA backend sources as HIP code for AMD GPUs. ENGINE_ENABLE_HIP and ENGINE_ENABLE_CUDA are mutually exclusive — configure with exactly one of them.

Linux (the helper script auto-detects ROCm via ROCM_PATH/HIP_PATH/hipconfig and local GPU targets via amdgpu-arch, falling back to rocminfo; pass --gpu-targets to build for other architectures, or when no AMD GPU is visible, e.g. in a VM or container):

scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --gpu-targets "gfx1100;gfx1103" --target audiocpp_cli

Direct CMake:

cmake -S . -B build_hip \
  -DENGINE_ENABLE_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DCMAKE_C_COMPILER="$(hipconfig -l)/clang" \
  -DCMAKE_CXX_COMPILER="$(hipconfig -l)/clang++" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build_hip -j$(nproc)

Windows (the helper script auto-detects ROCm, GPU targets, cmake, and ninja):

powershell -ExecutionPolicy Bypass -File scripts\build_windows_hip.ps1

Run with --backend hip (rocm is accepted as an alias). For GPU target selection, hipBLASLt GEMM notes, iGPU tuning, and known limitations, see docs/build/HIP.md.

Build Options

Option	Meaning	Default
ENGINE_ENABLE_CUDA	Enable the ggml CUDA backend. Required for --backend cuda.	OFF
ENGINE_ENABLE_HIP	Enable the ggml HIP backend (AMD GPUs). Required for --backend hip; mutually exclusive with ENGINE_ENABLE_CUDA.	OFF
ENGINE_ENABLE_VULKAN	Enable the ggml Vulkan backend. Required for --backend vulkan.	OFF
ENGINE_ENABLE_METAL	Enable the ggml Metal backend. Required for --backend metal.	OFF on most platforms, ON on Apple
ENGINE_ENABLE_LLAMAFILE	Enable llamafile SGEMM support in ggml CPU builds.	ON
ENGINE_ENABLE_CUDA_GRAPHS	Enable ggml CUDA graphs support when CUDA is enabled.	ON
ENGINE_ENABLE_NATIVE_CPU	Build ggml CPU kernels with native host ISA flags such as -march=native. Disable this for portable CPU kernels or toolchains that reject generated CPU instructions.	ON
ENGINE_ENABLE_OPENMP	Enable OpenMP for host-side parallel work.	ON
ENGINE_BUILD_EXAMPLES	Build example binaries.	OFF
ENGINE_BUILD_TESTS	Build framework unit tests.	OFF
ENGINE_BUILD_WARMBENCH	Build warmbench helper binaries.	OFF
AUDIOCPP_DEPLOYMENT_BUILD	Compile package specs into CLI/server binaries for standalone GGUF and package-spec fallback loading. Script builds expose this as --deployment-build on Linux/macOS and -DeploymentBuild on Windows.	OFF
AUDIOCPP_MODEL_SET	Model composite to build: full, core, or custom. Script builds expose this as --model-set on Linux/macOS and -ModelSet on Windows.	full
AUDIOCPP_MODELS	Comma or semicolon separated model target names when AUDIOCPP_MODEL_SET=custom, such as qwen3_tts,pocket_tts,qwen3_asr. Script builds expose this as --models on Linux/macOS and -Models on Windows.	empty

Usage

For full setup, CLI, server, and workflow examples, see docs/usage.md.

CLI

The main CLI binary is:

build/bin/audiocpp_cli

High-level command shape:

audiocpp_cli --task <task> --model <path> [--family <family>] [--backend <backend>] [--mode <mode>] [options]

Core selectors:

• --task vad|asr|diar|sep|gen|tts|clon|vc|s2s|align|vdes|spk|svc
• --model <path>
• --family <name> optionally narrows model-loader selection when a model path could match more than one family
• --backend cpu|cuda|vulkan|metal|best
• --mode offline|streaming; streaming is available for models whose docs list streaming support

Common interface options:

• --load-option key=value passes model-load options, such as PocketTTS language selection
• --session-option key=value passes session/runtime options, such as backend-specific weight controls
• --request-option key=value passes per-request model options
• --config <id> selects a discovered config asset
• --weight <id> selects a discovered weight asset
• --device <n> selects the backend device
• --threads <n> sets backend and OpenMP worker threads

Examples:

Text-to-speech:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/model \
  --backend cuda \
  --text "audio.cpp is running PocketTTS locally." \
  --voice-ref assets/resources/sample.wav \
  --out build/out/pocket_tts.wav

PocketTTS with another language and a built-in voice:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/models/pocket-tts \
  --backend cuda \
  --load-option language=spanish \
  --text "Hola, esta es una prueba corta de Pocket TTS." \
  --voice-id alba \
  --out build/out/pocket_tts_spanish.wav

ASR:

build/bin/audiocpp_cli \
  --task asr \
  --family qwen3_asr \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/sample_16k.wav

Voice conversion:

build/bin/audiocpp_cli \
  --task vc \
  --family seed_vc \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/a.wav \
  --voice-ref assets/resources/b.wav \
  --out build/out/seed_vc.wav

Useful CLI features:

• --help with --task shows task-oriented help
• --help with --model <path> and optional --family <family> shows model-owned request, session, and load options
• --inspect prints discovered configs, weights, and capabilities
• --list-loaders prints registered model families (--json for the machine-readable contract)
• python tools/model_manager_v2.py list --json prints installable packages from model_specs/*.json
• --batch-text-file <txt> runs one offline request per non-empty line
• --batch-text-dir <dir> runs one offline request per .txt, .md, or .json file, normalizing each file as one paragraph
• --batch-audio-dir <dir> runs one offline request per .wav
• --audio-chunk-mode auto lets ASR/alignment models choose their safe long-audio policy; expert users can override with fixed, vad, or none where supported
• --request-sequence <json> runs a multi-request offline session
• --batch-merge-audio none|concat controls batch audio merge behavior
• --batch-manifest-out <json> writes a batch output manifest
• --metrics prints compact offline wall time, audio duration, RTF, realtime speed, sample rate, and channel metrics
• Use --request-sequence <json> --metrics for per-request metrics from one long-lived offline session
• --pipeline <json> runs a workflow instead of a raw task
• --list-pipelines prints registered workflows
• --workflow-input key=value overrides pipeline inputs
• --log streams framework logs to stdout
• --log-file <path> streams framework logs to a file in real time
• --segments-out, --turns-out, and --words-out write structured JSON outputs
• --vad-chunks-out writes offline VAD-based chunk windows; tune them with --vad-chunk-max-seconds, --vad-chunk-merge-gap-seconds, and --vad-chunk-padding-seconds

Server

The server binary is:

build/bin/audiocpp_server

Build:

cmake --build build -j$(nproc) --target audiocpp_server

Create a config file with your own model paths:

cat > server.json <<'JSON'
{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "pocket-tts",
      "family": "pocket_tts",
      "path": "/path/to/models/pocket-tts",
      "task": "tts",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english"
      }
    },
    {
      "id": "qwen3-asr",
      "family": "qwen3_asr",
      "path": "/path/to/models/Qwen3-ASR-0.6B",
      "task": "asr",
      "mode": "offline"
    }
  ]
}
JSON

Set "lazy_load": true to register configured model ids at startup while loading each model only on first use. Use per-model "lazy": true or "lazy": false to override that default.

Set top-level "backend" to "cuda", "cpu", "vulkan", "metal", or "hip". CUDA is the optimized path for audio.cpp; CPU, Vulkan, Metal, and HIP are intended for portability and testing when the binary is built with that backend, but performance and model coverage may be lower.

Start:

build/bin/audiocpp_server --config server.json

The server exposes:

• GET /health
• GET /v1/models
• POST /v1/audio/speech
• POST /v1/audio/transcriptions
• POST /v1/tasks/run

More server examples are in app/server/README.md.

Pipelines

Pipelines are an experimental JSON workflow feature for chaining multiple model and audio-processing steps behind one CLI command. A pipeline can define default inputs, let users override them with --workflow-input key=value, split long media into model-sized chunks, merge text or audio outputs back together, write intermediate artifacts under --out-dir, and copy the declared final_audio to --out.

This is the higher-level layer for production-style audio jobs: redubbing, batch cleanup, long-form narration, voice conversion, source-separation workflows, transcription-plus-alignment, and future workflows that combine translation, diarization, denoise, enhancement, or review steps as those model surfaces are wired into the framework.

The included same-language speech redub pipeline transcribes long speech in chunks with Qwen3 ASR, merges the transcript, then regenerates the speech in a target reference voice with Qwen3 TTS. The default test input assets/resources/speech.wav is about 418 seconds long and was generated from an 8,091-character speech text, so it exercises long-audio split and merge behavior rather than a short one-shot request:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Override the source speech or target voice without editing the JSON:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --workflow-input source_audio=/path/to/speech.wav \
  --workflow-input target_voice=/path/to/voice.wav \
  --workflow-input language=English \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Tests

The repository includes both framework-level parity validation and app-level end-to-end path checks. At a high level, the flow is:

The main harness under tests/ is tests/warmbench.py. It is used for long-lived multi-request validation, parity checks against Python references, and performance-oriented session reuse scenarios. The tests/ tree also contains model-specific C++ and Python warmbench entrypoints that warmbench.py coordinates.

The main app-facing test tooling under tools/ is tools/audiocpp_cli/run_audiocpp_cli_path_tests.py. It drives audiocpp_cli through cataloged offline and streaming cases, verifies expected outputs such as audio or JSON artifacts, and is useful for checking real user-facing request paths rather than just lower-level model components. Streaming coverage is model-specific and applies to models documented with streaming support.

The Python-reference side of these tests usually requires more time-consuming setup than the C++ path because different models rely on different Python reference repos and dependency stacks. In practice, the framework-side tooling is fast to iterate on once models are installed, while Python parity runs often need extra environment preparation before they are ready.

Projects

Last update: 2026-07-08

Have a project using audio.cpp? Submit a PR or let me know, and I’ll be happy to add it here.

• TranscrIA is a self-hosted meeting transcription platform with diarization and local LLM correction. audio.cpp is integrated as a first-class STT engine in the product.
• Pocket TTS Browser Engine uses audio.cpp to bring fully local PocketTTS voices into Chrome and Edge through the browser TTS API.
• GuideAnts uses audio.cpp as the default local AI stack path for basic ASR and TTS, with planned reusable skills for audio.cpp scenarios and model configurations.

Performance Metrics

All performance metrics in this section were measured on Ubuntu with the CUDA backend on an NVIDIA GeForce RTX 5090. The Python-relative one-shot and long-lived-session comparisons come from direct framework/runtime API benchmark calls, not from audiocpp_cli; CLI path tests are separate and include app-layer request parsing, output writing, and other user-facing overhead.

Absolute RTF depends on the GPU and system setup, but the Python-relative speedups are real because audio.cpp and the matching Python reference paths were measured on the same CUDA setup.

audio.cpp already shows some genuinely exciting wins against the matching Python reference paths, especially on the TTS side, even when using the original model weights without quantization. The headline win is wall time: several TTS paths run 1.8x to up to 10x faster than Python while cutting end-to-end latency by 45%-90%.

• In one-shot runs, several TTS-family models already land far ahead of Python:
  • vevo2: 5.03x faster with 80.11% less wall time
  • pocket tts: 3.68x faster with 72.80% less wall time
  • miotts: 2.73x faster with 63.39% less wall time
  • moss_tts_local: 2.33x faster with 57.07% less wall time
  • qwen3 tts: 1.83x faster with 45.34% less wall time
  • vibevoice: 1.40x faster with 28.75% less wall time
• In long-lived-session runs, where the same loaded session serves multiple requests in sequence, the gains stay strong:
  • pocket tts: 3.22x faster with 68.91% less wall time
  • qwen3 tts: 2.74x faster with 63.47% less wall time
  • moss_tts_local: 2.66x faster with 62.35% less wall time
  • miotts: 2.28x faster with 56.22% less wall time
  • vibevoice: 1.77x faster with 43.55% less wall time
  • vevo2: 1.75x faster with 42.72% less wall time
• In long-form runs on the shared 6,026-character, 1,028-word passage, the strongest Python-relative wins still show up clearly:
  • pocket tts: 3.15x faster with 68.23% less wall time
  • qwen3 tts: 3.06x faster with 67.33% less wall time
  • vibevoice: 2.86x faster with 65.07% less wall time
  • vevo2: 1.77x faster with 43.51% less wall time
  • chatterbox: 1.58x faster with 36.83% less wall time
• These long-lived-session numbers are especially important for real applications, because they reflect the common case where model load, cached state, and reusable runtime setup are amortized across many requests.
• Bars below the 1.0x line are useful too: they spotlight exactly where more optimization work is still worth doing.

The figures report Python wall time / audio.cpp wall time. The 1.0x line means equal wall time; bars above 1.0x mean audio.cpp is faster than Python, and bars below 1.0x mean it is slower.

For TTS-family models, the measured one-shot RTF is:

model	audio len (s)	wall time (s)	RTF	x faster than real time
chatterbox	9.72	2.45	0.252	3.97x
miotts	20.40	3.30	0.162	6.18x
moss_tts_local	9.60	0.97	0.101	9.91x
omnivoice	9.00	1.32	0.146	6.84x
pocket tts	8.08	0.26	0.032	31.09x
qwen3 tts	11.44	4.46	0.390	2.56x
vevo2	8.66	2.47	0.285	3.51x
vibevoice	11.07	5.02	0.454	2.20x
voxcpm2	5.60	3.09	0.551	1.81x

For long-form TTS tests, each run uses the same 6,026-character, 1,028-word input text (vibevoice uses 106,310 chars, 18,052 words, 4 speakers). Rows are CUDA unless marked CPU. The measured RTF is:

model	audio len (s)	wall time (s)	RTF	x faster than real time
chatterbox	391.24	58.57	0.150	6.68x
index tts2	422.12	139.95	0.332	3.02x
miotts	399.16	66.59	0.167	5.99x
moss_tts_nano	391.20	43.16	0.110	9.06x
moss_tts_local	375.44	73.84	0.197	5.08x
omnivoice	357.00	17.77	0.050	20.09x
pocket tts	353.12	7.30	0.021	48.40x
qwen3 tts	327.60	72.65	0.222	4.51x
supertonic	379.32	2.02	0.005	187.62x
supertonic (CPU)	379.40	61.40	0.162	6.18x
vevo2	457.68	52.47	0.115	8.72x
voxcpm2	315.84	72.70	0.230	4.34x
vibevoice	5615.73	1376.84	0.245	4.08x

Runtime Memory Options

Some models expose memory-saver session options such as ace_step.mem_saver=true, dramabox.mem_saver=true, heartmula.mem_saver=true, stable_audio.mem_saver=true, omnivoice.mem_saver=true, and voxcpm2.mem_saver=true. These options keep the default output path unchanged while reducing graph workspace VRAM or releasing staged graph/cache state after request phases; later requests may rebuild released graphs.

Precision/Quantization Support

Many model sessions expose quantization through --session-option <family>.weight_type=<mode>, and some families also expose more specific knobs such as ...conv_weight_type, ...talker_weight_type, or ...speech_decoder_weight_type. The exact supported modes are model-specific rather than global.

audio.cpp also supports standalone GGUF packages.

In practice, lower precision and quantized modes should be treated as model- and route-specific optimizations rather than universally safe defaults.

• Safety. Quantization may not be safe on every path even when a model parser accepts the option. For example, in our ACE-Step 1.5 checks, lower-precision runs could fail at runtime with ACE-Step planner masked decode found no valid token while higher-precision settings completed normally.

• Quality Drop. Output quality can drop a lot. In our VeVo2 checks, non-fp32 outputs showed noticeably weaker similarity to the fp32 reference under the repo's existing waveform and log-mel comparison metrics, and even output length could shift.

• Performance Gain. The performance gain may be minor relative to that quality risk. For example, q8_0 was faster than the default setting by only around 3.8% on Qwen3-TTS and around 3.6% on VeVo2. Other models may benefit more, but the tradeoff should be validated per model and per route rather than assumed.

• Memory Benefit. Lower precision and quantized weights can still be useful for reducing weight memory footprint and making larger models easier to fit within device limits. For example, in our Qwen3-TTS checks, switching from the default setting to q8_0 reduced peak RAM by about 3.7% and peak VRAM by about 25.0%. That benefit is real, but it should be evaluated together with runtime stability, output quality, and end-to-end speed rather than assumed from precision alone.

Notes

• The repo supports multiple backends, but backend and model coverage are model-dependent.
• GGUF is a container, not a universal architecture adapter. Existing llama.cpp or whisper.cpp GGUF files are not automatically compatible unless their tensor names and model metadata are mapped to the audio.cpp family implementation.
• Build_xcframework.sh is outdated; Metal and Apple XCFramework packaging still need to be retested after the framework refactor.