OuteTTS validation

July 31, 2026 · View on GitHub

This procedure exercises OuteTTS 1.0 1B in one long-lived audio.cpp session. It covers normal TTS, framework long-form text chunking, voice cloning, repeated reference-profile cache hits, cached-step graph reuse, stage timings, and default-versus-mem_saver memory behavior.

Model setup

Install the safetensors model, IBM DAC, and Qwen3 Forced Aligner resources:

python tools/model_manager_deprecated.py install outetts_1_0_1b --models-dir models

The model-manager package id is outetts_1_0_1b. It creates these model roots:

models/Llama-OuteTTS-1.0-1B
models/DAC.speech.v1.0
models/Qwen3-ForcedAligner-0.6B

The standalone GGUF command is documented in TTS. The packed file used below contains the OuteTTS language model, IBM DAC, Qwen aligner, tokenizers, configuration, and package specification.

Python reference setup

The parity run used the official OuteTTS repository at commit f5eac6e70d792844c6a6959d900a47af2c061a5b (outetts 0.4.4), Python 3.10, PyTorch 2.5.1+cu121, and Transformers 4.52.3:

git clone https://github.com/edwko/OuteTTS build\reference\OuteTTS
git -C build\reference\OuteTTS checkout f5eac6e70d792844c6a6959d900a47af2c061a5b
python -m venv --system-site-packages build\reference\venv
build\reference\venv\Scripts\python.exe -m pip install -e build\reference\OuteTTS --no-deps
build\reference\venv\Scripts\python.exe -m pip install transformers==4.52.3 llama-cpp-python==0.3.9 polars natsort mecab-python3 unidic-lite uroman openai-whisper ftfy pyloudnorm

The reference loaded models/Llama-OuteTTS-1.0-1B through outetts.Backend.HF with FP32 CUDA weights and loaded the official IBM DAC checkpoint from models/DAC.speech.v1.0/weights_24khz_1.5kbps_v1.0.pth. Both implementations used temperature 0.4, repetition penalty 1.1 over the latest 64 tokens, top-k 40, top-p 0.9, min-p 0.05, the request-file seeds, and the same maximum number of new tokens.

Temperature zero was deliberately not used. OuteTTS 1.0's official generation_config.json, README example, and Python API all select sampled generation with temperature 0.4; changing to greedy generation would test a different inference contract. OuteTTS has no diffusion or latent-noise input. Its only request-time randomness is language-model token sampling.

For a controlled clone comparison, Qwen word timings were generated once and passed to the official Python AudioProcessor.create_speaker_from_dict path:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task align --family qwen3_forced_aligner `
  --model ..\models\Qwen3-ForcedAligner-0.6B_Q8\Qwen3-ForcedAligner-0.6B_Q8.gguf `
  --backend cuda --audio assets\resources\b.wav `
  --text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." `
  --language en --words-out build\reference\b_words.json

Python still performed the official DAC encoding and feature extraction. This keeps different alignment algorithms from contaminating the TTS comparison. The audio.cpp cold-clone timing includes its Qwen alignment, while the Python number starts from the supplied word timings.

Run the committed official-reference driver after creating the alignment JSON:

build\reference\venv\Scripts\python.exe `
  tools\community_models\outetts_reference.py `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --dac ..\models\DAC.speech.v1.0\weights_24khz_1.5kbps_v1.0.pth `
  --alignment-json build\reference\b_words.json `
  --request-file tests\outetts\warm_bench_requests.json `
  --device cuda --dtype fp32 `
  --out-dir build\reference\official_script_fp32

The driver resets both torch.manual_seed and torch.cuda.manual_seed_all immediately before every model.generate call, after all prompt, alignment, and DAC-profile work. It writes native 24 kHz WAVs plus boundaries.json, memory.json, stdout.log, and command.json.

Exact build commands

The Windows validation used the project build script to establish the compiler and backend configuration, then explicitly enabled the warm-benchmark targets.

Windows CPU:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -ConfigureOnly
cmake -S . -B build\windows-cpu-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cpu-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench

Windows CUDA 12.4:

$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ConfigureOnly
cmake -S . -B build\windows-cuda-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cuda-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench

Exact standalone-GGUF path tests

The tested GGUF was deliberately outside the PR worktree and its directory contained only one file:

..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf

Normal TTS:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task tts --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda `
  --text "This is the standalone GGUF path test." `
  --max-tokens 256 --request-option seed=1234 `
  --out ..\outputs\outetts_path_test_tts.wav

Voice cloning using the aligner and DAC embedded in the same GGUF:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda `
  --voice-ref assets\resources\b.wav `
  --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." `
  --request-option reference_language=en `
  --text "This is the standalone GGUF cloning path test." `
  --max-tokens 256 --request-option seed=42 `
  --out ..\outputs\outetts_path_test_clone.wav

Both commands loaded successfully without an external package spec, tokenizer, DAC directory, aligner directory, or other sidecar. The generated 24 kHz mono WAVs passed ffmpeg decoding:

Artifact	Duration	Bytes	SHA-256
`..\outputs\outetts_path_test_tts.wav`	1.399667s	67228	`92DA5D8438D4E6A79FBDDBD8EDA77D1AFD2A9B47F61085ECD3AD45C913102904`
`..\outputs\outetts_path_test_clone.wav`	1.453000s	69788	`E5B46E62C2DB89A654F7EC21A7BE31F575D6F667F8578224386E7C552492B05E`

Exact long-lived-session runs

CUDA default:

build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cuda_default `
  --log-file build\logs\warmbench\outetts-cuda-default.log

Expected trace evidence:

outetts.text_chunk_count is greater than one for tts_longform.
outetts.llama.step.graph_reused=1 appears after the first compatible generation request or chunk.
the second identical reference reports outetts.reference_cache.hit=1.
tts_cold and tts_repeat are byte-identical, as are clone_cold and clone_repeat; this verifies that warm graph/profile reuse does not change deterministic output.
outetts.aligner.runtime_reused=1 is observable for uncached references while the default session retains the aligner.
only one active OuteTTS Llama runtime is retained; switching between native TTS weights and the CUDA F32 cloning fallback replaces the previous runtime.

Run the same request sequence in a fresh process with memory saver enabled:

build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --session-option outetts.mem_saver=true `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cuda_mem_saver `
  --log-file build\logs\warmbench\outetts-cuda-mem_saver.log

CPU default:

build\windows-cpu-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cpu --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cpu_default `
  --log-file build\logs\warmbench\outetts-cpu-default-final.log

Safetensors CUDA parity run (DAC is resolved by the package spec; the aligner path is supplied because it is not embedded in the safetensors directory):

build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --session-option outetts.weight_type=f32 `
  --session-option outetts.aligner_path=..\models\Qwen3-ForcedAligner-0.6B `
  --audio-out-dir build\reference\cpp_f32_final_explicit `
  --log-file build\reference\cpp_f32_final_explicit.log

The memory-saver trace reports a positive outetts.llama.step.released_cache_capacity after generation and outetts.aligner.runtime_released=1 after an uncached reference.

Sample total-device VRAM every 250 ms while each fresh benchmark runs. Ensure that no unrelated CUDA workload is active:

nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -lms 250

Compare peak and final resident VRAM, request wall time, output duration, and RTF between the two runs. Keep model, backend, device, seed, requests, and quantization identical.

Generated artifacts

Each benchmark directory contains:

tts_cold_1.wav
tts_repeat_1.wav
tts_longform_1.wav
clone_cold_1.wav
clone_repeat_1.wav

The exact generated directories and logs were:

..\outputs\outetts_review_cuda_default
..\outputs\outetts_review_cuda_mem_saver
..\outputs\outetts_review_cpu_default
build\logs\warmbench\outetts-cuda-default.log
build\logs\warmbench\outetts-cuda-mem_saver.log
build\logs\warmbench\outetts-cpu-default-final.log
build\logs\warmbench\outetts-ab-torch-sampler\default-{1,2,3}-audio
build\logs\warmbench\outetts-ab-torch-sampler\mem_saver-{1,2,3}-audio
build\reference\cpp_f32_final_explicit
build\reference\cpp_q8_torch_sampler
build\reference\official_script_fp32\outputs
build\reference\official_script_fp32\boundaries.json

These validation artifacts are reproducible local outputs and are not committed to the repository.

Measured validation

The committed five-request sequence was measured with the packed Q8 GGUF on an NVIDIA GeForce RTX 3090 (CUDA 12.4). Each CUDA mode was run in three fresh processes in alternating order. Values below are mean +/- sample standard deviation. The Python HF and audio.cpp safetensors columns are single loaded- session runs; model load is excluded from every per-request measurement.

Request	Python HF FP32 wall / RTF	C++ FP32 wall / RTF	C++ Q8 wall / RTF
`tts_cold`	4599.12 ms / 3.485	5366.24 ms / 4.066	3211.84 +/- 17.30 ms / 2.434
`tts_repeat`	3777.69 ms / 2.863	3864.98 ms / 2.929	3030.79 +/- 18.60 ms / 2.297
`tts_longform`	15129.91 ms / 2.690	15547.80 ms / 2.764	13015.83 +/- 171.29 ms / 2.342
`clone_cold`	5762.51 ms / 3.792	5988.01 ms / 3.940	6432.68 +/- 65.11 ms / 4.233
`clone_repeat`	5522.67 ms / 3.634	4231.19 ms / 2.784	4220.23 +/- 44.07 ms / 2.777

These are per-request timings from a single loaded session; model loading is excluded. Python and C++ FP32 generated 1.31967, 1.31967, 5.62533, 1.51967, and 1.51967 seconds respectively. Q8 generated 5.55867 seconds for the long-form request and matched the other four durations.

The memory-saver A/B is also reported per request rather than inferred from a single aggregate run:

Request	Default CUDA Q8	Memory saver CUDA Q8	Wall delta	Default RTF	Memory-saver RTF
`tts_cold`	3211.84 +/- 17.30 ms	3223.24 +/- 12.89 ms	+0.35%	2.434	2.443
`tts_repeat`	3030.79 +/- 18.60 ms	3044.58 +/- 18.76 ms	+0.45%	2.297	2.307
`tts_longform`	13015.83 +/- 171.29 ms	13070.17 +/- 105.32 ms	+0.42%	2.342	2.351
`clone_cold`	6432.68 +/- 65.11 ms	6467.00 +/- 46.38 ms	+0.53%	4.233	4.256
`clone_repeat`	4220.23 +/- 44.07 ms	4238.27 +/- 17.07 ms	+0.43%	2.777	2.789

Memory saver was slightly slower on every request in this controlled repeat: 0.35% to 0.53%, and 0.44% over the mean five-request wall total. These small differences are near normal run-to-run variance, so there is no evidence that memory saver is a speed optimization. It remains useful because the independently sampled peak VRAM fell from 17653 MiB to 5780 MiB (67.3%) with the same 294 MiB post-session resident VRAM. It is therefore a memory-versus-speed option, not a speed optimization.

The Windows CPU Q8 run is likewise reported per request:

Request	Wall	Audio	RTF
`tts_cold`	10517.2 ms	1.31967s	7.970
`tts_repeat`	10349.9 ms	1.31967s	7.843
`tts_longform`	41604.0 ms	5.67867s	7.326
`clone_cold`	22849.3 ms	1.58633s	14.404
`clone_repeat`	18205.6 ms	1.58633s	11.477

All generated WAV files passed an ffmpeg decode check. The traces confirmed four framework chunks for the long-form request, compatible step-graph reuse in default mode, explicit graph release in memory-saver mode, and a reference- profile cache hit on the repeated clone. In the original instrumented default CUDA run, cold reference preparation took about 890 ms (109 ms alignment, 139 ms DAC encoding, and 214 ms profile construction), while the repeated cached reference took 0.29 ms.

Path and parity results

The standalone path test loaded the one-file GGUF from outside the worktree; no adjacent sidecars or model_specs directory were present.
Within CUDA, cold and repeated TTS were byte-identical with SHA-256 9D1AF0D25212129B3C26880F7C928CFFA813A4B9EF22F12F390A4C372D51D406.
Within CUDA, cold and repeated cloning were byte-identical with SHA-256 EB7E6B5A1330F845F45D3DA21C8AC28490572F95218A6F0D3DA2A83CB797EFE6.
The CUDA default and memory-saver outputs had the same hashes.
Within CPU, cold and repeated TTS were byte-identical with SHA-256 A315C0AD425C99910597FCC4BAB9D80CB5F4C893176EBD9099BA6C6AA7B66BFD.
Within CPU, cold and repeated cloning were byte-identical with SHA-256 1A2009C0D512773A4B5082BF37EDEDD6A52A37CC6D556772C62EA26550A83B4D.
CPU and CUDA WAV bytes are not identical because backend floating-point execution differs; deterministic reuse parity was therefore checked within each backend rather than asserted across backends.

Python parity uses the repository's tools/audiocpp_cli/compare_audiocpp_cli_path_results.py implementation: PCM WAV cosine plus an 80-band log1p mel-spectrogram cosine (n_fft=1024, hop=256). Both sides write the DAC-native 24 kHz signal directly. Raw WAV cosine is highly phase-sensitive; log-mel cosine is the more informative value when the DAC encoders are numerically close but not bit-identical.

The CUDA sampler now uses the framework's PyTorch-compatible exponential-race random stream, equivalent to the torch.multinomial path used by Transformers. Temporary trace-level instrumentation compared prompt strings, prompt token IDs, generated token IDs, codec codes, and WAV frame counts at component boundaries. It was removed after validation to avoid normal-log spam. The permanent Python driver records the Python side in boundaries.json.

Boundary results under the controlled FP32 run:

Request	Chunks	Python/C++ prompt IDs	Generated IDs per chunk	Python/C++ frames
`tts_cold`	1	22 / 22	256	31672 / 31672
`tts_repeat`	1	22 / 22	256	31672 / 31672
`tts_longform`	4	20,22,14,21 / same	256 each	135008 / 135008
`clone_cold`	1	2367 / 2367	256	36472 / 36472
`clone_repeat`	1	2367 / 2367	256	36472 / 36472

The sampled prompt/generated boundary values for all six ordinary-TTS chunks had zero mismatches. Clone prompt construction also matches the official word labels and token count. The C++ DAC reference encoder still differs slightly from Python: among 1046 reference-profile frames, C1 differs on 48 and C2 on 118. That numerical boundary explains why cloning is not waveform-identical.

Primary native/FP32 parity, measured only after the controlled random stream and frame counts matched:

Request	WAV cosine	Log-mel cosine	C++ / Python frames
`tts_cold`	0.999812247	0.999997481	31672 / 31672
`tts_repeat`	0.999812238	0.999997483	31672 / 31672
`tts_longform`	0.997324530	0.999294328	135008 / 135008
`clone_cold`	-0.007753064	0.889403845	36472 / 36472
`clone_repeat`	-0.007753064	0.889403845	36472 / 36472

Only after that path was clean was the packed Q8 model compared with the same FP32 Python reference. Quantization is expected to perturb autoregressive sampling, so this is a secondary characterization rather than the correctness gate:

Request	WAV cosine	Log-mel cosine	C++ Q8 / Python frames
`tts_cold`	-0.016326171	0.696795792	31672 / 31672
`tts_repeat`	-0.016326591	0.696792751	31672 / 31672
`tts_longform`	0.577927575	0.920741352	133408 / 135008
`clone_cold`	-0.033990775	0.882715855	36472 / 36472
`clone_repeat`	-0.033990775	0.882715855	36472 / 36472

Backend coverage

Backend	Coverage
Windows CPU	CLI/server/warm-benchmark build; normal TTS, long-form TTS, cloning, cache reuse, graph reuse, WAV decode, timing, and RSS run
Windows CUDA 12.4	CLI/server/warm-benchmark build; normal TTS, long-form TTS, cloning, standalone-path loading, cache/graph reuse, three-run default/memory-saver A/B, Python HF parity, WAV decode, timing, RTF, and VRAM run
Linux CPU	GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed
Linux Vulkan	GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed
macOS CPU	GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed
Metal	Not enabled or runtime-tested in this validation

Known limitations

OuteTTS is offline-only in this implementation.
Voice cloning requires an accurate transcript of the reference WAV and rejects references longer than 20 seconds.
Quantized CUDA cloning expands the OuteTTS language-model weights to F32 for generation correctness. Default mode can therefore have a high transient peak while the aligner is retained; outetts.mem_saver=true releases the aligner and cached-step graph between phases.
Long-form output concatenates independently generated chunks. max_tokens applies to each chunk; the runtime splits text further to fit an explicit cap and retries a chunk that reaches the cap without EOS/audio-end as smaller chunks rather than returning silently truncated speech. It reports an error only when the remaining text cannot be split any further.
CPU and CUDA outputs are deterministic within each tested backend but are not expected to be byte-identical across backends.
The controlled CUDA sampling stream is PyTorch-compatible. CPU sampling is deterministic for a fixed seed but still uses the C++ standard-library distribution and is not claimed to reproduce PyTorch token-for-token.
Quantized weights can change sampled tokens and output length even when the seed and sampler implementation match, as shown by the Q8 long-form frame drift above.