Voicebox Project Status & Roadmap
July 2, 2026 · View on GitHub
Last updated: 2026-07-02 | Current version: v0.5.0 | 402 open issues | 88 open PRs | 1.3M downloads · 34.8k stars
Table of Contents
- Architecture Overview
- Current State
- Open PRs — Triage & Analysis
- Open Issues — Categorized
- Existing Plan Documents — Status
- New Model Integration — Landscape
- Architectural Bottlenecks
- Recommended Priorities
Architecture Overview
Tauri shell (Rust) hosts a React frontend (app/) that talks over HTTP on localhost:17493 to a FastAPI backend (backend/).
The backend exposes:
TTSBackendProtocol with seven concrete engine implementations:- Qwen3-TTS (PyTorch or MLX depending on platform)
- Qwen CustomVoice (predefined speakers with instruct)
- LuxTTS (fast, CPU-friendly)
- Chatterbox Multilingual (23 languages)
- Chatterbox Turbo (English, paralinguistic tags)
- TADA (1B English, 3B multilingual via HumeAI)
- Kokoro 82M (pre-built voices, CPU realtime)
STTBackendProtocol for Whisper (PyTorch or MLX-Whisper)- Profiles / History / Stories services for persistence and timeline editing
Key Files
| Layer | File | Purpose |
|---|---|---|
| Backend entry | backend/main.py | FastAPI app, all API routes (~2850 lines) |
| TTS protocol | backend/backends/__init__.py:32-101 | TTSBackend Protocol definition |
| Model registry | backend/backends/__init__.py:17-29,153-366 | ModelConfig dataclass + registry helpers |
| TTS factory | backend/backends/__init__.py:382-426 | Thread-safe engine registry (double-checked locking) |
| PyTorch TTS | backend/backends/pytorch_backend.py | Qwen3-TTS via qwen_tts package |
| MLX TTS | backend/backends/mlx_backend.py | Qwen3-TTS via mlx_audio.tts |
| LuxTTS | backend/backends/luxtts_backend.py | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | backend/backends/chatterbox_backend.py | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | backend/backends/chatterbox_turbo_backend.py | Chatterbox Turbo — English, paralinguistic tags |
| TADA | backend/backends/hume_backend.py | HumeAI TADA — 1B English + 3B Multilingual |
| Kokoro | backend/backends/kokoro_backend.py | Kokoro 82M — CPU realtime, pre-built voices |
| Qwen CustomVoice | backend/backends/qwen_custom_voice_backend.py | Qwen CustomVoice — predefined speakers with instruct |
| Platform detect | backend/platform_detect.py | Apple Silicon → MLX, else → PyTorch |
| API types | backend/models.py | Pydantic request/response models |
| HF progress | backend/utils/hf_progress.py | HFProgressTracker (tqdm patching for download progress) |
| Audio utils | backend/utils/audio.py | trim_tts_output(), normalize, load/save audio |
| Frontend API | app/src/lib/api/client.ts | Hand-written fetch wrapper |
| Frontend types | app/src/lib/api/types.ts | TypeScript API types |
| Engine selector | app/src/components/Generation/EngineModelSelector.tsx | Shared engine/model dropdown |
| Generation form | app/src/components/Generation/GenerationForm.tsx | TTS generation UI |
| Floating gen box | app/src/components/Generation/FloatingGenerateBox.tsx | Compact generation UI |
| Model manager | app/src/components/ServerSettings/ModelManagement.tsx | Model download/status/progress UI |
| GPU acceleration | app/src/components/ServerSettings/GpuAcceleration.tsx | CUDA backend swap UI |
| Gen form hook | app/src/lib/hooks/useGenerationForm.ts | Form validation + submission |
| Language constants | app/src/lib/constants/languages.ts | Per-engine language maps |
How TTS Generation Works (Current Flow)
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
→ tts_backend.create_voice_prompt(audio_path, reference_text)
7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
8. Post-process: trim_tts_output() for Chatterbox engines
9. Save WAV → data/generations/{id}.wav
10. Insert history record in SQLite
11. Return GenerationResponse
Current State
Since v0.5.0 — Two-Month Pulse (2026-04-25 → 2026-06-27)
The repo went quiet while demand kept climbing. 0.5.0 (the Capture release) shipped 2026-04-25. In the two months since, only 2 PRs merged (#544 the release itself, #550 a remote-URL fix) while 150 new issues were opened and the open-PR queue more than tripled to 88. The community kept contributing — translations, new engines, GPU fixes — but nothing's been reviewed or merged. This is a review-and-merge backlog, not a build backlog.
| Metric | At v0.4.1 (2026-04-18) | Now (2026-06-27) | Δ |
|---|---|---|---|
| Open issues | 232 | 402 | +170 |
| Open PRs | 12 | 88 | +76 |
| GitHub stars | ~28k | 34.8k | +~7k |
| Downloads | — | 1.3M | — |
What the two months actually produced (all unmerged):
- A flood of community translations — pt-BR, de-DE, Russian (+docs), Arabic+RTL, Spanish (+docs site), French, Cantonese. ~12 i18n PRs sitting on the i18next foundation that landed in 0.5.0.
- GPU coverage PRs — AMD ROCm on Windows (#538), Intel XPU (#539), DirectML for Intel iGPU (#674), Blackwell diagnostic (#653), MLX threading fix (#789).
- A 17-PR hardening dump from one contributor (@neuron-tech-ai, all 2026-05-14): CI pipeline, Biome, OpenAI-compatible
/v1/audio/speech, SQLite WAL + indexes, LIKE-injection / upload-limit / N+1 fixes, platform gating. High value, entirely unreviewed. - New engine PRs — MiniMax cloud, MOSS-TTS-Nano, Fun-CosyVoice3, Parakeet STT.
- Two giant Linux PRs — vendored
taopatch for the Wayland startup panic (#748), and an SRT2Voice workflow (#673).
0.5.0 regressions worth triaging first:
- macOS Apple Silicon — all TTS models crash the server on load (#606, #615), MLX falls back to CPU on M4/M5 (#706, #650).
- Capture cutoffs at 30s for imported audio (#609, #626); paste broken in 0.5.0 (#762).
- MCP rough edges — dotted tool names violate Claude Desktop's name pattern (#790), audio scrambled over MCP (#780).
- Refinement silently translates non-English transcripts to English (#603).
Funding model — $VOICEBOX token (#806): the two-month gap was a solo-dev decision about long-term sustainability, not neglect or a compromise. $VOICEBOX (Solana) is the official, dev-controlled token and the chosen revenue path — donations/sponsors didn't cover full-time work. The app stays 100% free, open-source, local-first, no subscriptions. Dev supply is being bought back and burned (done twice), liquidity locked. It has funded ~2–3 months of full-time work, so the cadence resumes this week. The #806 thread was a community concern, addressed transparently and resolved amicably; keep an eye out for actual impersonator/community tokens, which are a separate thing.
Other trust/security signals: macOS malware-flag reports continue (#369); a DNS-rebinding / Host-header exposure on the local API+MCP server was reported with fixes attached (#778).
What's Shipped (v0.5.0 — the Capture release)
Shipped 2026-04-25 (PR #544). Voicebox went from a voice-cloning studio to a full voice studio — dictation in, agent speech out, a local LLM in the middle.
- Dictation — global hotkey capture (push-to-talk + toggle chords), on-screen pill with live state, auto-paste into the focused field with clipboard save/restore, chord-picker UI. Scoped Accessibility permission (transcripts still land if paste is denied).
- MCP server at
http://127.0.0.1:17493/mcp—voicebox.speak/.transcribe/.list_captures/.list_profiles. Streamable HTTP primary transport, stdio sidecar shim, per-client voice binding viaX-Voicebox-Client-Id. Speaking pill always shows agent-initiated output. - Personality — voice profiles carry an optional ≤2000-char persona. Compose (shuffle an in-character line) and Speak-in-character (rewrite input before TTS), both on a local Qwen3 LLM that doubles as the refinement model.
- Refinement — on-device Qwen3 strips fillers, fixes punctuation, optional self-correction rewrites; Whisper hallucination-loop stripping at a 6-token threshold; per-capture flag snapshots; model picker (0.6B / 1.7B / 4B).
POST /speakREST wrapper and i18next foundation (English + zh-CN) also landed.
What's Shipped (v0.4.x)
New since v0.3.0:
- Kokoro 82M TTS engine + voice profile type system (PR #325)
- Qwen CustomVoice preset engine — predefined speakers with instruct support (PR #328)
- Intel Arc (XPU) GPU support (PR #320)
- Blackwell GPU (sm_120) CUDA support (PR #401)
- Generation cancellation flow (PR #444)
- Frontend quality gates + TypeScript hardening (PR #418)
- macOS Intel (x86_64) PyTorch compatibility (PR #416)
- Frozen-binary import fixes for Kokoro / Chatterbox Multilingual / scipy / transformers (PR #438)
- Linux PipeWire/PulseAudio monitor detection (PR #457)
- Server survives GUI close on Windows (PR #402)
- GPU arch compatibility warning on startup (catches unsupported PyTorch builds)
- cpal Stream playback reliability (PR #405), clip-splitting stability (PR #403)
- torch.from_numpy crash with numpy 2.x in frozen binary (PR #361)
- Async CUDA download lock (PR #428), NUMBA_CACHE_DIR env var (PR #425)
- "Clear failed" history button (PR #412)
- External server GUI startup + data refresh (PR #319)
- Force offline mode for cached Qwen/Whisper models (PR #318)
- macOS 11 ScreenCaptureKit launch crash fix (PR #424)
Core TTS (cumulative):
- Qwen3-TTS voice cloning (1.7B and 0.6B models, MLX + PyTorch)
- Qwen CustomVoice (preset speakers, instruct)
- LuxTTS — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual — 23 languages including Hebrew (PR #257)
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
- HumeAI TADA — 1B English + 3B Multilingual (PR #296)
- Kokoro 82M — CPU-realtime, 8 languages, Apache 2.0 (PR #325)
- Multi-engine architecture with thread-safe backend registry (PR #254)
- Chunked TTS generation — engine-agnostic, removes ~500 char limit (PR #266)
- Async generation queue (PR #269)
- Post-processing audio effects system (PR #271)
- Voice profile type system (preset vs cloned, engine compatibility gating)
- Centralized
ModelConfigregistry — no per-engine dispatch maps - Shared
EngineModelSelectorcomponent
Infrastructure (cumulative):
- CUDA backend swap via binary download (PR #252), cu128 upgrade (PR #316), Blackwell/sm_120 (PR #401)
- CUDA backend split into independently versioned server + libs archives (PR #298)
- Intel Arc XPU support (PR #320)
- Docker + web deployment (PR #161)
- Backend refactor: modular architecture, style guide, tooling (PR #285)
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
- Windows support: CUDA detection, cross-platform justfile, server lifecycle (PR #272, #402)
- Linux audio capture via pactl monitor detection (PR #457)
- macOS Intel x86_64 compatibility (PR #416)
- Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large, turbo variants)
- Model management UI with inline download progress + folder migration (PR #268)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching and cancellation (PR #444)
- Streaming generation endpoint (MLX only)
- Audio player freeze fix + UX improvements (PR #293)
- CORS restriction to known local origins (PR #88)
Abandoned / Backlogged Integrations
| Model | PR / Branch | Reason |
|---|---|---|
| CosyVoice2/3 | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. PR should be closed. |
| VoxCPM 1.5 / VoxCPM2 | voicebox-new-models research (2026-04-18) | Backlogged. See detailed analysis below. |
VoxCPM — Evaluation Notes (2026-04-18)
Project: OpenBMB/VoxCPM — tokenizer-free TTS, 2B params (VoxCPM2), end-to-end diffusion autoregressive architecture, 30 languages, 48 kHz output, Apache 2.0, pip install voxcpm.
Why it looked interesting:
- Clean PyPI install (
pip install voxcpm) - Apache 2.0 — commercially safe
- Voice cloning via
reference_wav_pathwith optionalprompt_wav_path+prompt_textfor "ultimate" cloning - Streaming API via
generate_streaming() - Zero-shot cloning + style control via parenthetical prefixes in text (
(slightly faster, cheerful tone)...) - Relatively high-quality output per demos
Why we backlogged it:
- Effectively CUDA-only. README states
CUDA ≥ 12.0as hard requirement. Source code'sfrom_pretrained(device=None|"auto")claims "preferring CUDA, then MPS, then CPU," but in practice:- MPS (Apple Silicon) broken upstream — OpenBMB/VoxCPM issues #232 (
NotImplementedError: Output channels > 65536 not supported at the MPS device) and #248 (IndexErroron M3 Mac) are both open with no resolution. - CPU unsupported in the Python package — issue #256 shows
voxcpm --device cpurejected withunrecognized arguments. The only CPU path is the third-party VoxCPM.cpp GGML engine, which is a separate ecosystem project, notpip install voxcpm. - macOS source install fails — issue #233 open with no resolution.
- MPS (Apple Silicon) broken upstream — OpenBMB/VoxCPM issues #232 (
- Would require CUDA-only gating in UI (new
requires_cudaflag onModelConfig, lock icon + "Requires NVIDIA GPU" inModelManagement.tsx/EngineModelSelector.tsx) plus a hard error atload_model()as safety net. Doable but adds first-class platform gating that doesn't exist for any other engine today. - Voicebox's user base skews Apple Silicon (MLX is a primary backend). Shipping a CUDA-only model sets a precedent worth a separate scoping discussion (see issues #419 engine sprawl, #420 platform tiers, PR #465).
What would change the decision:
- Upstream fixes MPS crashes (watch issues #232, #248).
- We define an "experimental / CUDA-only" engine tier as part of issue #419 / PR #465, and decide it's acceptable to ship engines that are hidden on non-NVIDIA platforms.
- VoxCPM.cpp matures into a viable CPU path we can wrap (currently separate project, C++/GGML, unclear ergonomics).
Integration shape if we revive it: Zero-shot cloning maps naturally to the Chatterbox-style backend (store ref_audio + ref_text paths in the voice prompt dict, process at generate time). Est. ~250 lines for voxcpm_backend.py + one ModelConfig entry + engine registration in backends/__init__.py. Frontend UI gating is the bigger lift.
Funded Roadmap (2026-H2)
$VOICEBOX funded ~2–3 months of full-time work; cadence resumes the week of 2026-06-27. Direction committed publicly in #806:
| Item | Notes |
|---|---|
| Resume merge/release cadence | Clear the 88-PR backlog, regular commits + releases — this is the immediate focus (see Tier 1) |
| Mobile companion app | New surface; already drawing issues (#773 iPhone logout) |
| Encrypted cloud backup/sync | For voice profiles + generations — first cloud feature; stays opt-in, local-first remains default |
| More TTS models | Engine candidates in the Landscape section below; community PRs #507/#766/#777 in queue |
| Better GPU support | Blackwell/sm_120, ROCm, DirectML, Intel — incl. paying testers for hardware the dev lacks |
| Bug fixes | 0.5.0 regression cluster first (macOS load crash, capture cutoffs, MCP, refinement) |
What's In-Flight
| Feature | Branch/PR | Status |
|---|---|---|
| Platform support tiers | PR #465, issue #420 | Defining tier-1 (supported) vs tier-2 (community) platforms |
| Engine sprawl cleanup | issue #419 | First-class vs experimental TTS backends distinction |
| Frontend tech-debt burn-down | issue #421 | Biome + a11y debt before gating CI |
| Docker registry auto-publish | PR #463, issue #453 | ghcr.io image on tag push |
| New model research | voicebox-new-models branch | Evaluating Fish Speech, XTTS-v2, Pocket TTS, VibeVoice, Fish Audio S2, index-tts2. 2026-06-27 sweep added dots.tts, LongCat-AudioDiT, SoproTTS, NeuTTS, Nemotron/Cohere STT — see Landscape → New Candidate Sweep |
TTS Engine Comparison
| Engine | Model Name | Profile Type | Languages | Size | Key Features | Instruct Support |
|---|---|---|---|---|---|---|
| Qwen3-TTS 1.7B | qwen-tts-1.7B | Cloned | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
| Qwen3-TTS 0.6B | qwen-tts-0.6B | Cloned | 10 | ~1.2 GB | Lighter, faster | None |
| Qwen CustomVoice 1.7B | qwen-custom-voice-1.7B | Preset | 10 | ~3.5 GB | Predefined speakers, instruct support | Yes |
| Qwen CustomVoice 0.6B | qwen-custom-voice-0.6B | Preset | 10 | ~1.2 GB | Predefined speakers, instruct support | Yes |
| LuxTTS | luxtts | Cloned | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | chatterbox-tts | Cloned | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — exaggeration float (0-1) |
| Chatterbox Turbo | chatterbox-turbo | Cloned | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only |
| TADA 1B | tada-1b | Cloned | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | tada-3b-ml | Cloned | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
| Kokoro 82M | kokoro | Preset | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
Multi-Engine Architecture (Shipped)
- Thread-safe backend registry (
_tts_backendsdict +_tts_backends_lock) with double-checked locking - Per-engine backend instances — each engine gets its own singleton, loaded lazily
- Engine field on GenerationRequest — frontend sends
engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro' - Per-engine language filtering —
ENGINE_LANGUAGESmap in frontend, backend regex accepts all languages - Per-engine voice prompts —
create_voice_prompt_for_profile()dispatches to the correct backend - Profile type system — preset vs cloned profiles, UI grays out incompatible engines and auto-switches on selection
- Trim post-processing —
trim_tts_output()for Chatterbox engines (cuts trailing silence/hallucination)
Known Limitations
- HF XET progress: Large files downloaded via
hf-xet(HuggingFace's new transfer backend) reportn=0in tqdm updates. Progress bars may appear stuck for large.safetensorsfiles even though the download is proceeding. This is a known upstream limitation. - Chatterbox Turbo upstream token bug:
from_pretrained()passestoken=os.getenv("HF_TOKEN") or Truewhich fails without a stored HF token. Our backend works around this by callingsnapshot_download(token=None)+from_local(). - chatterbox-tts must install with
--no-deps: It pinsnumpy<1.26,torch==2.6.0,transformers==4.46.3— all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly inrequirements.txt. - Instruct parameter partially shipped (#224, #303): Qwen CustomVoice (PR #328) now provides real instruct support via predefined speakers. Other backends still silently drop the instruct field — the UI exposes the field broadly but most engines ignore it. The floating generate box was patched to restore instruct for CustomVoice (commit
106aec4). - Streaming generation only works for Qwen on MLX. Other engines use the non-streaming
/generateendpoint. - dicta-onnx (Hebrew diacritization) not included — upstream Chatterbox bug requires
model_patharg but callsDicta()with none. Hebrew works fine without it. - Blackwell (RTX 50-series) CUDA: cu128 + sm_120 kernel support shipped (PR #401, #316), but users still report
cudaErrorNoKernelImageForDevice(#417, #400, #396, #395, #390, #362) — likely a stale CUDA binary on upgraded installs. Needs a follow-up diagnostic / forced re-download path. - Long text 50k character limit (#464, #365, #354): Still hit on GPU despite chunking (PR #266). Chunking reliability needs another pass.
- ROCm on RDNA 3/4 (#469):
HSA_OVERRIDE_GFX_VERSIONis hardcoded and harms newer cards. flash-attn is not installedwarning on every platform (cosmetic, common user complaint): Our transformer-based engines (Chatterbox / Qwen) emitWarning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.on every startup, on every platform — we don't pinflash-attnin requirements because installing it is fragile and version-sensitive. Fallback is PyTorch SDPA, which is near-FA2 throughput on Ampere+ and is what actually runs. Per-platform reality: (a) macOS/Apple Silicon — FlashAttention is CUDA-only, irrelevant here; MLX has its own attention kernels. (b) Linux —pip install flash-attn --no-build-isolationworks but takes 20+ min to compile. (c) Windows — no official support (Dao-AILab README still says only "Might work"; source builds routinely fail on recent CUDA/MSVC, issues #1715, #1828, #2395). Windows users can install community prebuilt wheels fromkingbri1/flash-attentionorbdashore3/flash-attention(latest v2.8.3, Aug 2025;win_amd64wheels for CUDA 12.4/12.8, Torch 2.6–2.9, Python 3.10–3.13) matching their exact CUDA/Torch/Python, or use WSL2. Native-Windows alternatives worth considering as a build-time swap: SageAttention (thu-ml, Apache 2.0, claims 2–5× over FA2) and xformers (official Windows wheels). Action for us: troubleshooting doc now covers it (seedocs/content/docs/overview/troubleshooting.mdx), and we should optionally suppress the warning vialogging.getLogger(...).setLevel(ERROR)at backend import since the fallback is functionally fine.- WebAudio playback dies after audio-session interruption (#41, plus an internal repro where the app is backgrounded long enough): WaveSurfer's
AudioContextgets suspended by macOS — either because another app grabs the audio output, or because the WKWebView throttles when backgrounded.play()resolves andtimeupdatecan still fire, but no audio reaches the output. Only app restart fixes it. Things already tried that didn't work: (a) swapping WaveSurfer backend away from WebAudio — introduced more bugs, not an option; (b) remount hook on the player — doesn't help because a freshly-createdAudioContextis born suspended and only resumes on a user gesture. PR #293 was a prior partial fix that doesn't cover this path. Next thing to try (not yet attempted — confirmed via grep ofAudioPlayer.tsx): callwavesurfer.getMediaElement().getGainNode().context.resume()on the play button click (the click itself is a valid user gesture), plus avisibilitychange+statechangelistener as belt-and-suspenders. Thectx.resume()pattern already exists in the codebase atuseStoryPlayback.ts:52— just not wired into the main player.
Open PRs — Triage & Analysis
88 open PRs, only 2 merged since 0.5.0. The queue is the single biggest lever right now — a lot of finished community work is waiting on review. Clustered below by theme. Counts are approximate; a PR can span clusters.
Merged since 0.5.0
| PR | Title | Merged |
|---|---|---|
| #550 | Fix web API URL for remote access | 2026-04-25 |
| #544 | feat: 0.5.0 Capture release — dictation, MCP, personalities | 2026-04-25 |
i18n / translations (~12) — easy wins, unblock a large user segment
The i18next + zh-CN foundation shipped in 0.5.0; these stack on it. Triage as a batch.
| PR | Locale / scope |
|---|---|
| #528 | pt-BR translation |
| #571 | de-DE translation |
| #599 / #600 / #601 | Russian — app, landing routes, docs |
| #569 | Arabic + RTL layout fixes |
| #798 / #799 / #801 | Spanish — locale, README/CONTRIBUTING/SECURITY, docs site (see #800 for approach alignment) |
| #802 | French translation |
| #776 | Cantonese language option |
| #688 | Compose follows the selected language |
New engines / models (~7)
| PR | Engine | Notes |
|---|---|---|
| #507 | MOSS-TTS-Nano (0.1B, 20 langs, CPU realtime) | Matches our cross-platform criteria — top engine candidate |
| #331 / #430 | MiniMax Cloud TTS | Two PRs, same provider — dedupe. External-API direction. |
| #777 | Fun-CosyVoice3 (draft) | We abandoned CosyVoice2/3 once on quality — re-evaluate output before reviving |
| #766 | Parakeet as STT model | Whisper alternative |
| #563 | 4-bit quantized Qwen + Russian abbreviations | Smaller/faster Qwen |
| #225 | Custom HuggingFace voice models | Long-lived; needs rework for multi-engine arch |
| #195 | Per-profile LoRA fine-tuning (draft, +6.2k) | Complex, 15 endpoints — addresses #185/#224 demand |
GPU / hardware (~9)
| PR | Scope |
|---|---|
| #538 | Native AMD ROCm on Windows (+2.8k, resolves #531) |
| #539 | Optional IPEX + native Intel XPU detection for PyTorch 2.9+ |
| #674 | DirectML for Intel iGPU (Iris/UHD/Arc) — pairs with demand in #676, #759 |
| #653 | Blackwell GPU arch-mismatch diagnostic — directly targets the sm_120 cluster |
| #789 | Run MLX load+inference on one thread (fixes #699) — likely fixes the M-series load crashes |
| #560 / #561 | Linux NVIDIA auto-detect + Linux CUDA backend build |
| #736 / #785 / #769 | ROCm HSA_OVERRIDE_GFX_VERSION cleanup (resolves #469) |
| #770 | Fix CUDA downloads on unsupported platforms |
Capture / transcription / refinement (~6) — fixes 0.5.0 regressions
| PR | Fix |
|---|---|
| #602 | Re-encode uploaded audio as PCM WAV before Whisper — likely fixes the 30s import cutoff (#609/#626) |
| #616 | Enable long-form Whisper on the PyTorch path |
| #629 | Preserve source language in refinement (fixes #603 English translation) |
| #637 | MCP/REST generations incorrectly trigger autoplay |
| #712 | Personality LLM respects the selected refinement model |
| #796 | Capture preview placement setting (#698) |
Long-form / stories / streaming (~5)
| PR | Scope |
|---|---|
| #154 | Audiobook tab with chunked generation (predates shipped chunking — reconcile) |
| #673 | SRT2Voice workflow (+14k) — large, subtitle-driven generation |
| #787 | m4b/mp3 story export with auto chapter markers |
| #642 | stream=true immediate-audio mode for GET /tts |
| #804 | Stream MLX TTS audio chunks (draft) |
Linux / Wayland (~5)
| PR | Scope |
|---|---|
| #748 | Vendor-patch tao 0.34.8 for Wayland startup panic (+39k — large vendored diff, verify approach) |
| #624 | Linux tauri schema + build deps (+6.2k) |
| #747 | Avoid abort when hiding the dictate pill |
| #622 / #768 | Linux audio monitor selection / thread-unsafe PULSE_SOURCE |
| #677 | HF cache permissions + startup fallback |
Hardening / CI / perf — the @neuron-tech-ai batch (all 2026-05-14, ~17 PRs)
One contributor opened a large, coherent quality suite in a single day. Review as a group; #662 is the headline.
| PR | Scope |
|---|---|
| #662 | LIKE injection, upload-size enforcement, N+1 queries, SSE reconnect, memory leaks, model display names (+6.8k) |
| #654 | CI pipeline + pre-commit hooks + Biome config + test suite |
| #656 | OpenAI-compatible /v1/audio/speech + /v1/models (addresses #10) |
| #657 | Platform gating on ModelConfig + UI (addresses bottleneck #6 / issue #419) |
| #666 / #667 | DB indexes on hot FKs + SQLite WAL + busy timeout |
| #659 / #660 / #661 / #663 / #665 / #668 | datetime.utcnow→UTC, MediaRecorder crash + sample reorder, fail-fast on missing model, batch story counts, Metal warmup, drop debug logs |
| #652 / #655 / #658 / #664 | AGENTS.md, docs GitHub Pages, avatar size limit, non-fatal actool |
Build / dev tooling / docker (~6)
#764 uv for backend env · #632 docker GPU build + cache + fastmcp (+7.9k) · #630 ROCm docker overlay · #463 ghcr.io auto-publish · #543 / #681 setup-script fixes · #584 docker permission fix
Smaller fixes worth grabbing
#786 remove 50k char limit (#464) · #621 broken-pipe crashes on model load · #743 harden mac generation status + MLX threading · #788 missing male Mandarin Kokoro voices · #794 build mcp shim on Windows · #527 Chatterbox exaggeration + CFG sliders · #253 48kHz speech tokenizer
Stale / low-signal — close or request changes
#91 (draft, Feb, CoreAudio, +6.2k unrebased) · #649 ("fix this errors") · #623 / #782 (badges / package tweaks) · #311-style abandoned engines — verify before merging anything older than ~April against the 0.5.0 codebase.
Open Issues — Categorized
402 open, +150 in the two months since 0.5.0. Demand snapshot from a keyword sweep over all open titles (buckets overlap):
| Theme | ~Open | Signal |
|---|---|---|
| New model / engine requests | ~79 | Largest category. Voxtral, OmniVoice, VibeVoice, VoxCPM2, CosyVoice3, Dramabox, Parakeet, GGUF, ONNX/Piper export |
| CUDA / GPU / Blackwell | ~53 | Still the #1 bug driver — sm_120 "no kernel image", ROCm, DirectML, Intel Arc, VRAM/load times |
| Model download / server startup | ~42 | Stuck downloads, "server process ended unexpectedly", loading_model hangs |
| Capture / dictation / transcribe | ~31 | New surface from 0.5.0 — 30s cutoffs, paste, mic permission, refinement translation |
| Language / locale requests | ~27 | Bengali, Ukrainian, Filipino, Indonesian, Cantonese, zh-TW; plus UI localization |
| Fine-tune / clone quality | ~22 | #185 (top-engagement issue), accent leakage, "finetunes not working" |
| Long-form / chunking / export | ~16 | Pause control, speed control, audiobook export, >50k chars |
| Linux / Wayland | ~11 | Build failures, Wayland panics, CUDA-on-Linux packaging |
| MCP / agent / API | ~9 | Dotted tool names (#790), scrambled audio (#780), OpenAI compat (#10) |
| Security / trust | ~4 | DNS-rebinding (#778), malware flag (#369); funding via official $VOICEBOX token (#806) |
Highest-engagement open issues: #185 Fine-tune instructions (32c) · #98 Connecting to Download (16c) · #301 CUDA generation failure (18c) · #20 Model download failed (13c) · #364 Voxtral-TTS FR (11r) · #341 Arch Linux build · #513 server startup failed (12c) · #138 ONNX/Piper export (9r) · #10 OpenAI API compat.
New since 0.5.0 — clusters to triage first
- macOS Apple Silicon load crashes (regression): #606, #615 — all TTS models crash the server on load; #706, #650 — MLX falls back to CPU / 7-min VRAM load on M4/M5. PR #789 (single-thread MLX, fixes #699) and #743 are the candidate fixes. Highest priority — breaks the primary platform.
- Capture cutoffs & paste: #609, #626 — transcription stops at 30s for imported audio (PR #602 re-encodes WAV); #762 — paste broken in 0.5.0; #698, #577 — capture/output folder locations.
- MCP integration: #790 — dotted tool names violate Claude Desktop's
^[a-zA-Z0-9_-]{1,64}$; #780 — audio scrambled over MCP; #728 — CUDA re-downloads on cold start. - Refinement: #603 — silently translates non-English transcripts to English (PR #629 preserves source language).
- GPU expansion requests: #676 DirectML (AMD/Intel), #759 Intel Arc, #684 RTX 5060 Ti CUDA 13, #774 CUDA 11.x for older cards, #767 Linux CUDA installs Windows
.exe. - New engines/langs: #791 OmniVoice, #633 VoxCPM2, #690 Dramabox, #638 Bengali, #754 zh-TW, #761 Filipino.
- Open plugin interface (#771): request for a community engine/provider plugin API — ties into engine-sprawl (#419) and platform-gating work.
- Trust/security: #806 —
$VOICEBOXis the official dev-backed funding token (concern raised and resolved on-thread; see funding note above); #778 DNS-rebinding/Host-header exposure on local API+MCP (fixes attached) — genuine security item; #369 macOS malware flag (ongoing).
GPU / Hardware Detection — still the top category
RTX 50-series (Blackwell / sm_120) cluster — NEW: #417, #400, #396, #395, #390, #362 all report cudaErrorNoKernelImageForDevice / "no kernel image available." sm_120 support shipped in PR #401 + cu128 in PR #316, but users on upgraded installs still hit it — likely stale CUDA binary. Needs a diagnostic that detects binary/GPU-arch mismatch and prompts re-download.
AMD / ROCm — NEW: #469 HSA_OVERRIDE_GFX_VERSION is hardcoded and breaks RDNA 3/4 cards. #313 DirectML on AMD Ryzen AI Max+ 395 not working.
Intel Arc: PR #320 shipped XPU support — may resolve #119.
General GPU-not-detected (older): #368, #310, #330, #324, #326, #355 (multi-GPU / eGPU).
Fix path: CUDA backend swap (PR #252) + cu128 (PR #316) + sm_120 (PR #401) + GPU-arch warning (73170d0) are all in. Remaining work is diagnostics + re-download prompts for users whose binary predates the kernel updates.
Model Downloads
Still reported. Users get stuck downloads, can't resume, offline mode edge cases.
Key issues: #475 (MAC CustomVoice install error), #449 (infinite loading macOS), #445 (can't download CustomVoice), #462 (Qwen requires internet even when loaded — regression from #150), #434 (infinite retry loop offline — PR #443 open), #432 (storage location change hangs when empty — partly fixed by PR #439/#433), #348 (TADA 3B Multilingual download fails), #336 (TADA model not listed in app), #275 (No module named 'chatterbox' on download), #304 (whisper-base feature extractor load error), #287 (macOS ARM check_model_inputs ImportError on new version), #181, #180.
Fix path: PR #443 addresses infinite offline retry. CustomVoice-specific download failures (#475, #445) need triage — likely related to frozen-binary import fixes in PR #438. TADA cluster (#336, #348) and macOS ARM import regressions (#287, #275, #304) need a dedicated triage pass.
Qwen 0.6B-downloads-1.7B reports: #485 (2026-04-19), #423 (macOS M1), #329. Originally a stale-fallback bug: mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16 wasn't published when MLX support shipped, so the 0.6B slot was aliased to the 1.7B repo. The 0.6B bf16 conversion is live now and both backend/backends/mlx_backend.py and backend/backends/__init__.py point at their correct repos. Qwen CustomVoice is unaffected — it runs via PyTorch on all platforms, both sizes always have dedicated repos.
Language Requests (ongoing)
Strong demand: Hungarian (#479), Indonesian (#458, #247), Thai (#455), Bangla (#454), Arabic (#379), Persian (#162), IndicF5 (#339 — Indian languages), Ukrainian (#109), Chinese UI (#392, #261).
Fix path: Chatterbox Multilingual (PR #257) covers Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Still missing: Hungarian, Indonesian, Thai, Bangla, Ukrainian. Issue #411 offers a PR for UI i18n foundation.
New Model Requests (growing)
| Issue | Model Requested |
|---|---|
| #478 | CosyVoice3 (we tried & abandoned CosyVoice2/3 — see #311) |
| #407, #347 | RVC-style voice-to-voice / seed voice conversion (STS) |
| #385 | Fish Audio S2 |
| #380 | OmniVoice |
| #370 | index-tts2 |
| #364 | Voxtral-TTS |
| #335 | Faster-Qwen-TTS |
| #346 | Multi-model batch request |
| #381 | Microsoft MAI models |
| #339 | IndicF5 |
| #226 | GGUF support |
| #172 | VibeVoice |
| #138 | Export to ONNX/Piper format |
| #132 | LavaSR (transcription) |
| #147 | Facebook Omnilingual ASR |
| #338 | Default voices |
The multi-engine architecture makes integration straightforward — see content/docs/developer/tts-engines.mdx. Platform-specific gating (e.g. VoxCPM CUDA-only) doesn't exist yet and would need design.
Platform Scope & Quality Debt — NEW category
Awareness issues filed this cycle — ties into engine sprawl and platform tier work.
- #419 — Engine sprawl: define first-class vs experimental TTS backends
- #420 — Formalize tier-1 vs tier-2 platform support targets (PR #465 open)
- #421 — Track & burn down frontend Biome + a11y debt before gating CI
- #422 — Code-split web build (main bundle > 1 MB)
Long-Form / Chunking
Still reported despite chunking + queue being merged.
Key issues: #464 (50k char limit on GPU despite 16 GB VRAM — v0.4.0), #365 (FR: >50k chars), #363 (smart chunking to prevent robotic artifacts), #354 (50k limit v0.3.0).
Fix path: Chunking (#266) and queue (#269) shipped. Remaining work is raising/removing the 50k guard and tuning chunk boundaries for prosody.
Feature Requests (ongoing)
Notable:
- #480 — Noise removal on uploaded recordings
- #448 — API for non-Qwen models (external integrations)
- #427 — Task status control
- #407, #347 — Voice-to-voice / audio-to-audio conversion
- #387 — Location of downloaded generated voices
- #383 — Concatenate partial reference audio into generated audio
- #382 — Lightning.ai support
- #376 — Remote mode
- #353 — Audio transcoding
- #317 — Voice pitch control
- #189 — "Auto" language option
- #173 — Vocal intonation/inflection control
- #165, #270 — Audiobook mode (PR #154 open)
- #242 — Seed value pinning
- #228 — Always use 0.6B option
- #235 — Finetuned Qwen3-TTS tokenizer (PR #253 open)
- #144 — Copy text to clipboard
Housekeeping / Triage Needed
| Issue | Reason |
|---|---|
| #431, #408 | Spam — Chinese "free Claude API" promos. Close. |
| #398 ("Excelente") | Non-issue. Close. |
| #357 | Informational — project featured in Awesome MLX. Close after acknowledgement. |
| #374, #377 | Version-release questions, no bug. Close. |
| #306 ("voice model"), #389 ("New model"), #473 ("New functionality") | Title-only issues, no content. Request details or close. |
| #309 | Uninstall/cleanup question. Answer and close. |
| #241 | "How to use in Colab" — support question, not a bug. |
| #423 / #485 / #329 | Stale MLX fallback to 1.7B repo — fixed; 0.6B bf16 conversion now live on mlx-community, registry points at correct repo on both backends. |
| #336 / #348 | TADA download/registration cluster — triage together. |
| #287 / #275 / #304 | macOS ARM import regressions on new version — likely one root cause. |
| #292, #349 | Possibly already fixed by merged PRs (#321/#412 and #345). Verify + close. |
~70 older issues (pre-#170) not individually categorized above. Most are long-tail support questions or duplicates of problems now addressed by the multi-engine / model-registry work. A dedicated backlog-sweep pass is overdue.
Bugs (ongoing)
| Category | Issues |
|---|---|
| Generation failures | #476, #467, #452, #459 (voice clone fetch error), #468 (tada-1b marked error), #437, #300, #301, #282 |
| Audio quality | #456 (clipping errors v0.4.0), #436 (emotion labels), #333 (pitch/echo), #307 (by-model breakdown), #340 (all generations say "www...") |
| Transcription | #371 (fails every time), #291 (extract transcription from generated audio) |
| Effects / presets | #349 ("Failed to save" when creating effects presets — possibly fixed by merged #345) |
| File ops | #477 (spacy_pkuseg dict missing on frozen Windows build), #472 (storage location change), #283 (allow longer files for voice creation + in-app trim), #350 (failed to add sample) |
| History | #292 (can't delete failed generations — possibly fixed by merged #321/#412) |
| Windows | #466 (install problem), #375 (WinError 5 access denied), #273 (port 8000 conflict), #201 (model doesn't stay loaded) |
| Linux | #471 (thread-safe PULSE_SOURCE), #413 (Arch build), #409 (Kubuntu build), #351, #341 |
| macOS | #441 (older macOS), #369 (malware flag), #334 (microphone permission), #287 (check_model_inputs ImportError — regression), #171 (ARM64 binary won't open) |
| Profile/UI | #360 (Kokoro profile hides others — partly addressed by auto-switch), #299 (drag-drop on Win11), #329 (size selector state bug), #393 (stuck loading screen after reinstall to new dir) |
| Integrations | #397 (SAMMI-bot 422 Unprocessable Entity) |
| Audio playback / session | #41 (macOS: Voicebox goes silent after another app takes audio output; restart restores it) — see deep-dive below |
| Database | #174 (sqlite3 IntegrityError) |
Existing Plan Documents — Status
| Document | Target Version | Status | Relevance |
|---|---|---|---|
TTS_PROVIDER_ARCHITECTURE.md | v0.1.13 | Partially superseded by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
CUDA_BACKEND_SWAP.md | — | Shipped (PR #252) | CUDA binary download + backend restart |
CUDA_BACKEND_SWAP_FINAL.md | — | Shipped (PR #252) | Final implementation plan |
EXTERNAL_PROVIDERS.md | v0.2.0 | Not started | Remote server support |
MLX_AUDIO.md | — | Shipped | MLX backend is live |
DOCKER_DEPLOYMENT.md | v0.2.0 | Shipped (PR #161) | Docker + web deployment |
OPENAI_SUPPORT.md | v0.2.0 | Not started | OpenAI-compatible API layer |
PR33_CUDA_PROVIDER_REVIEW.md | — | Reference | Analysis of the original provider approach |
New Model Integration — Landscape
Status Snapshot (2026-04-18)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct | Cross-platform? | Status |
|---|---|---|---|---|---|---|---|---|
| Qwen3-TTS | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None | MLX + PyTorch | Shipped |
| Qwen CustomVoice | Preset speakers | Medium | 24 kHz | 10 | Medium | Yes | PyTorch | Shipped (PR #328) |
| LuxTTS | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | All | Shipped (PR #254) |
| Chatterbox MTL | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — exaggeration | CPU/CUDA | Shipped (PR #257) |
| Chatterbox Turbo | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags | CPU/CUDA | Shipped (PR #258) |
| HumeAI TADA 1B/3B | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), 10 (3B) | Medium | Partial — prosody | PyTorch | Shipped (PR #296) |
| Kokoro-82M | Preset voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | All | Shipped (PR #325) |
| 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Yes | — | Abandoned (PR #311) — poor output quality | |
| Zero-shot | ~0.15 RTF streaming | 48 kHz | 30 | Medium | Partial — parenthetical style | CUDA-only in practice | Backlogged (2026-04-18) — see notes above | |
| Fish Speech | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | Yes — word-level inline | All | Candidate — license TBD |
| Fish Audio S2 | — | — | — | — | — | — | — | Candidate (#385) |
| XTTS-v2 | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref | All | Candidate — CPML license likely blocker |
| Pocket TTS (Kyutai) | Zero-shot + streaming | >1x RT on CPU | — | English + several European (FR/DE/PT/IT/ES added by Feb 2026) | ~100M | None | CPU-first | Candidate — MIT |
| MOSS-TTS-Nano | Zero-shot | Realtime on 4 CPU cores | 48 kHz stereo | 20 | 0.1B | Partial — MOSS-VoiceGenerator companion does text-to-voice design | All (ONNX CPU path dropped 2026-04-17) | Top candidate — Apache 2.0, released 2026-04-13, streaming |
| VibeVoice (Microsoft) | — | — | — | Multi-speaker long-form (up to 90 min, 4 speakers) | 1.5B | — | — | Candidate (#172) — Stories-editor fit |
| index-tts2 | — | — | — | — | — | — | — | Candidate (#370) |
| Voxtral TTS (Mistral) | Zero-shot (short clips) + 20 preset voices | Single-GPU | — | — | 4B (Voxtral-4B-TTS-2603) | Presets + cloning | CUDA (16 GB+ VRAM) | Candidate (#364) — frontier quality claim, open-weight |
| Dia / Dia2 | — | — | — | — | — | — | — | Watch — emotion-forward, but "rough edges" / artifacts per April reviews |
| IndicF5 | — | — | — | Indian languages | — | — | — | Candidate (#339) — fills Indic gap |
| MiniMax Cloud TTS | — | Cloud | — | — | N/A (API) | — | N/A | Community PR #430, #331 — new direction (external API) |
| OmniVoice | — | — | — | — | — | — | — | Candidate (#380) |
| RVC voice conversion | N/A (STS) | — | — | — | — | N/A | All | New modality, not TTS (#407, #347) |
Watch list: MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control), Faster-Qwen-TTS (#335), Orpheus / Sesame CSM (on-device fine-tuning discussions), Fish Audio S2 Pro / Fish Speech V1.5 (benchmark leader but research/non-commercial license — same blocker as Fish Speech).
Deep-research pass (2026-04-18): MOSS-TTS-Nano identified as the freshest high-alignment candidate — verified via OpenMOSS/MOSS-TTS README (0.1B params, Apache 2.0, 48 kHz stereo, 4-core CPU realtime, streaming, released 2026-04-13). Dedicated repo: OpenMOSS/MOSS-TTS-Nano. Voxtral TTS verified on HF as mistralai/Voxtral-4B-TTS-2603.
Active Evaluation Criteria (learned from cycle)
- Cross-platform first. MLX is a primary backend for our Apple Silicon user base. CUDA-only models require platform gating that doesn't exist yet — shipping one sets a precedent (see VoxCPM notes, issues #419/#420).
- PyPI + Apache/MIT licensing preferred. Heavy deps, git-only installs, and
--no-depsworkarounds are expensive to maintain (Chatterbox taught us this). - Output quality is non-negotiable. CosyVoice was abandoned despite the best instruct API.
- Instruct support fills a real gap (#173, #224, #303). Qwen CustomVoice partially addresses it with preset speakers; zero-shot clone-with-instruct is still unmet.
- Long-form + streaming are user-requested (#363, #365, #464). Candidates with native streaming (Pocket TTS, Fish Speech) get extra weight.
New Candidate Sweep (2026-06-27)
A follow-up deep-research pass, filtered against everything already tracked — the shipped engines plus MOSS-TTS-Nano, Pocket TTS, IndicF5, VibeVoice, Voxtral, Fish/Fish Audio, XTTS-v2, index-tts2, VoxCPM2, OmniVoice, MioTTS, Oolel, Faster-Qwen, Orpheus/Sesame, MiniMax, RVC, Parakeet, Qwen3-ASR, Moshi, GLM-4-Voice, Qwen2.5-Omni — kept only where a newer sibling/variant changes the evaluation. Same criteria as the 04-18 cycle: cross-platform, PyPI/clean packaging, permissive license, quality, instruct/style control, long-form, streaming.
Top new TTS candidates
| Candidate | Add as | Why it matters | Caveat |
|---|---|---|---|
| dots.tts (soar / mf) | Top new TTS candidate | 2B fully-continuous end-to-end autoregressive TTS, 48 kHz AudioVAE output, zero-shot cloning via prompt audio/text, Apache-2.0 code+checkpoints, MeanFlow-distilled variant for low latency. Freshest "serious clone engine" not yet on the roadmap. | Git-source install with constraints, not clean PyPI. Needs Windows/macOS packaging + VRAM/CPU smoke test; probably experimental until platform gating exists. |
| MOSS-TTS family / v1.5 / Local-Transformer-v1.5 | Upgrade the MOSS-Nano entry into a MOSS family epic | We track only Nano, but MOSS now spans MOSS-TTS, TTSD (long multi-speaker dialogue), VoiceGenerator (text-prompt voice design), TTS-Realtime, SoundEffect. v1.5 adds broader languages, long-reference cloning, pause control, 48 kHz stereo, MLX/vLLM support, Apache-2.0. | Full 4B/8B variants aren't the lightweight Nano win. Treat as several engines/features, not one checkbox. |
| LongCat-AudioDiT | High-priority Apple Silicon candidate | 3.5B non-autoregressive diffusion TTS in waveform latent space, zero-shot cloning, already has an MLX conversion usable via mlx_audio — unusually aligned with our Apple Silicon base. | zh/en only, not realtime. Quality play, not low-latency agent speech. |
| SoproTTS | Lightweight CPU/streaming cloned TTS | 135M zero-shot cloning, pip install -U sopro, streaming + non-streaming APIs, 3–12s reference, claimed 250 ms TTFA / 0.05 RTF on M3 CPU. Strong local-first/low-maintenance fit. | English-focused, self-described as inconsistent — quality-test before promoting past experimental. |
| NeuTTS Air / Nano | GGUF/on-device cloned TTS | On-device instant cloning, GGUF-ready, ~3s reference, laptop/phone/Pi targets. Air is Apache-2.0. | Needs a GGUF/llama.cpp-style wrapper, not a normal PyTorch backend. Nano has a separate NeuTTS Open License — split needs review. |
| X-Voice | Small multilingual clone | 0.4B multilingual zero-shot cloning, 30 languages, IPA-style unified rep, claims no prompt-transcript requirement — targets a real cloning-UX pain point. | Verify license, packaging, production-readiness of weights/code. |
| FireRedTTS-2 | Stories / podcast / multi-speaker | Apache-2.0 long-form streaming, 3-min / 4-speaker dialogue, cross-lingual code-switching cloning, low first-packet latency. | Stories-editor engine more than a general default. Needs platform/VRAM testing. |
| Maya1 | Expressive English voice-design | 3B Apache-2.0, voice design, streaming, emotion/style tags, vLLM-compatible, 24 kHz, single-GPU. Good "voice personalities" / game-dialogue fit. | English-only, 16 GB+ VRAM — platform gating required. |
MOSS is now a family, not one checkbox. The single MOSS-TTS-Nano row above should become an epic: keep Nano as the CPU-friendly model, and track v1.5 / Local-Transformer-v1.5, Realtime, TTSD, VoiceGenerator, and SoundEffect as siblings under it.
STT / capture candidates (feed the planned streaming-transcription roadmap)
| Candidate | Add as | Why it matters | Caveat |
|---|---|---|---|
| Nemotron 3.5 ASR Streaming 0.6B | Top new STT candidate | Cache-aware streaming FastConformer-RNNT, 40 language-locales, punctuation/caps, language-ID conditioning, MLX conversion path — strongest fit for planned streaming transcription. | NVIDIA-origin; verify license + non-CUDA (MLX/CPU) performance. |
| Cohere Transcribe 03-2026 | High-quality offline STT | 2B Apache-2.0, 14 languages, ONNX/INT8 exports across CPU / Apple Silicon / GPU. Cleanest-looking offline /transcribe + captures candidate. | Less clearly a streaming dictation model than Nemotron. |
| ARK-ASR 3B / 0.6B | Multilingual STT watch | New family, broad European/Asian coverage, strong leaderboard claims, INT8 ONNX for edge. | Very new; likely trust_remote_code. Validate stability first. |
| IBM Granite Speech 4.1 2B / NAR | ASR + speech translation | Compact multilingual ASR + bidirectional speech translation (en/fr/de/es/pt/ja); NAR variant for latency-sensitive work. | More compelling if we expand into translation, not just dictation. |
Watch-list / blocked (license or platform work must land first): LEMAS-TTS, Supertonic 3, KugelAudio, GLM-TTS, KittenTTS, TinyTTS (preset/on-device, not cloning); Sarashina2.2, Higgs Audio v3, T5Gemma-TTS, Step-Audio-EditX, MisoTTS (non-commercial terms or CUDA-heavy); MegaTTS3 (incomplete WaveVAE encoder distribution); PFluxTTS, LongCat-Next (paper-only / too broad). Low-hanging Qwen-family variants: Qwen3-TTS-VoiceDesign (fills text-to-voice-design with minimal churn) and ZipVoice/ZipVoice-Dialog (only if it brings zh-en/dialogue behavior our shipped LuxTTS doesn't already expose).
Roadmap patch from this sweep (reflected in Tier 3 below):
- Replace the
MOSS-TTS-Nanocheckbox with a MOSS-TTS family epic (Nano tracked separately as the CPU model). - New Tier-3 TTS candidates, in order: dots.tts → LongCat-AudioDiT → SoproTTS → NeuTTS → X-Voice → FireRedTTS-2 → Maya1.
- New STT expansion candidates, in order: Nemotron 3.5 → Cohere Transcribe → ARK-ASR → Granite Speech.
- Keep Sarashina2.2, Higgs v3, T5Gemma, Step-Audio-EditX, MisoTTS, MegaTTS3, PFluxTTS blocked/watch-only.
- Do platform gating (bottleneck #6 /
ModelConfig.requires) before shipping GPU-only engines — Maya1, Step-Audio-EditX, MisoTTS, and probably dots.tts stay experimental until it exists.
Adding a New Engine (Now Straightforward)
With the model config registry and shared EngineModelSelector component, adding a new TTS engine requires:
- Create
backend/backends/<engine>_backend.py— implementTTSBackendprotocol (~200-300 lines) - Register in
backend/backends/__init__.py— addModelConfigentry +TTS_ENGINESentry + factory elif - Update
backend/models.py— add engine name to regex - Update frontend — add to engine union type,
EngineModelSelectoroptions, form schema, language map, profile type gating (icons/labels ~9 files per grep ofkokoro)
main.py requires zero changes — the registry handles all dispatch automatically.
Platform gating doesn't exist yet. If we add a CUDA-only model (e.g. VoxCPM), we need a new requires_cuda (or more generally requires: list[device]) flag on ModelConfig, plumbed through /models API and surfaced in ModelManagement.tsx and EngineModelSelector.tsx as a lock icon + "Requires NVIDIA GPU" state. Backend should hard-error at load_model() as a safety net.
Total effort: ~1 day for a well-documented model with a PyPI package, cross-platform. ~2 days if platform gating is required. See content/docs/developer/tts-engines.mdx for the full guide.
Architectural Bottlenecks
1. Single Backend Singleton — RESOLVED
The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
2. main.py Dispatch Point Duplication — RESOLVED
main.py Dispatch Point DuplicationPreviously, each engine required updates to 6+ hardcoded dispatch maps across main.py (~320 lines of if/elif chains). A model config registry in backend/backends/__init__.py now centralizes all model metadata (ModelConfig dataclass) with helper functions (load_engine_model(), check_model_loaded(), engine_needs_trim(), etc.). Adding a new engine requires zero changes to main.py.
3. Model Config is Scattered — RESOLVED
Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the ModelConfig registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in EngineModelSelector.tsx.
4. Voice Prompt Cache Assumes PyTorch Tensors
backend/utils/cache.py uses torch.save() / torch.load(). LuxTTS, Chatterbox, and Kokoro backends work around this by storing reference audio paths (or preset voice IDs) instead of tensors in their voice prompt dicts. Not ideal but functional.
5. Frontend Assumes Qwen Model Sizes — RESOLVED
The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen / Qwen CustomVoice.
6. No Platform Gating on Models — NEW
ModelConfig has no way to express hardware requirements. Every engine is shown to every user, regardless of whether it'll actually load. Users on non-CUDA platforms discover failure at load time (or not at all — some fall back silently to CPU and never complete). Blocks shipping CUDA-only engines (VoxCPM) and would improve the Intel Arc / ROCm / CPU-only UX today. See ModelConfig TODO: add requires: list[Literal["cuda", "mps", "xpu", "cpu", "rocm"]] or equivalent, plumb through /models API, render in ModelManagement.tsx + EngineModelSelector.tsx.
7. Engine Sprawl — NEW
Seven TTS engines shipped, more candidates queued. Issue #419 asks for a first-class vs experimental distinction. Related: issue #420 asks for formalized platform support tiers. Combined, these would let us ship more engines more confidently with clearer expectations for users.
Recommended Priorities
Tier 1 — Ship Now (the next release is mostly a merge-and-fix pass)
The two-month gap means the highest-leverage work isn't new code — it's reviewing the 88-PR queue and shipping the 0.5.0 regression fixes that are already written.
| Priority | PR/Item | Impact | Effort |
|---|---|---|---|
| 1 | macOS Apple Silicon load crash (#606, #615, #706, #650) — review/merge PR #789 (single-thread MLX) + #743 | Breaks the primary platform on 0.5.0 | Low (PRs exist) |
| 2 | Capture 30s import cutoff (#609, #626) — review PR #602; paste-broken #762 | Core 0.5.0 feature degraded | Low–Medium |
| 3 | Refinement translates to English (#603) — merge PR #629 | Silent data loss for non-English users | Low |
| 4 | MCP dotted tool names (#790) — breaks Claude Desktop; scrambled audio #780 | Flagship integration broken for some clients | Low–Medium |
| 5 | Blackwell / sm_120 diagnostic — review PR #653; stale-binary re-download path | Largest GPU bug cluster | Medium |
| 6 | Drain the i18n batch (#528, #571, #599–601, #569, #798–801, #802, #776) | ~12 finished PRs, large user segment | Low (review-bound) |
| 7 | Review the @neuron-tech-ai hardening batch — start with #662, #657 (platform gating), #656 (OpenAI API), #654 (CI) | Security + perf + bottleneck #6 in one sweep | Medium (review-bound) |
| 8 | Remove 50k char limit (#464) — merge PR #786; tune chunk boundaries | Long-standing regression | Low |
| 9 | Housekeeping — dedupe MiniMax #331/#430, re-evaluate CosyVoice #777, close spam/empty issues (#805, #775) | Triage hygiene | Low |
Tier 2 — Feature Work
| Priority | Item | Impact | Effort |
|---|---|---|---|
| 1 | Engine tier system (#419) — first-class vs experimental, platform gating in ModelConfig | Unblocks CUDA-only engines (VoxCPM, etc.) and frontend polish | Medium |
| 2 | Frontend tech-debt burn-down (#421) + code-split (#422) | Before gating CI on Biome | Medium |
| 3 | #154 — Audiobook tab | Long-form users. Chunking + queue shipped. | Medium |
| 4 | UI i18n (#411 PR offer, #392, #261) | Chinese UI + general localization | Medium |
| 5 | #225 — Custom HuggingFace models | User-supplied models. Needs rework. | High |
| 6 | OpenAI-compatible API (plan doc exists) — see also #448 (API for non-Qwen) | Low effort once API is stable | Low |
| 7 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
| 8 | Streaming for non-MLX engines | Currently MLX-only | Medium |
| 9 | Voice-to-voice / RVC (#407, #347) | New modality — different arch shape | High |
Tier 3 — Future Engines (cross-platform preferred)
Committed ordering (04-18 cycle), then the 2026-06-27 sweep additions. See Landscape → New Candidate Sweep for full rationale.
| Priority | Item | Notes |
|---|---|---|
| 1 | MOSS-TTS family (was MOSS-TTS-Nano) | Nano first: 0.1B, Apache 2.0, 4-core CPU realtime, 48 kHz stereo, streaming, 20 langs. Best alignment with our criteria. Then track v1.5 / Realtime / TTSD / VoiceGenerator / SoundEffect as siblings under one epic. |
| 2 | Pocket TTS (Kyutai) | CPU-first 100M model. MIT. Fills streaming gap without CUDA dependency. Several European langs added by Feb 2026. |
| 3 | IndicF5 | Fills Indian-language gap (#339). Closes many language-request issues. |
| 4 | VibeVoice (Microsoft, #172) | 1.5B, long-form multi-speaker (up to 90 min, 4 speakers). Strong Stories-editor fit. |
| 5 | Voxtral TTS (Mistral, #364) | 4B presets+cloning. Frontier quality claim, but 16 GB+ VRAM — would need the platform-tier work first. |
| 6 | Fish Speech / Fish Audio S2 | 50+ langs, word-level instruct. License clarification first. (#385) |
| 7 | XTTS-v2 | 17+ langs, mature pip. CPML likely kills commercial use — verify. |
| 8 | index-tts2 (#370) | Unvetted. |
| — | Backlogged — CUDA-only upstream. Revisit when tier system ships or MPS bugs are fixed upstream. | |
| — | New (06-27 sweep), in order → | |
| 9 | dots.tts | 2B end-to-end AR, 48 kHz, Apache-2.0 + fast MeanFlow variant. Top new candidate. Git-source install — smoke-test packaging + VRAM; likely experimental until platform gating exists. |
| 10 | LongCat-AudioDiT | 3.5B diffusion, has an MLX/mlx_audio path — best Apple Silicon fit. zh/en only, not realtime. |
| 11 | SoproTTS | 135M, pip install sopro, streaming, ~250 ms TTFA / 0.05 RTF on M3 CPU. Quality-test first. |
| 12 | NeuTTS Air/Nano | On-device GGUF cloning, ~3s reference. Needs a GGUF wrapper; Air is Apache-2.0, Nano license split needs review. |
| 13 | X-Voice | 0.4B, 30 langs, no prompt-transcript required. Verify license/packaging. |
| 14 | FireRedTTS-2 | Apache-2.0 long-form multi-speaker/podcast streaming. Stories-editor engine; needs VRAM testing. |
| 15 | Maya1 | 3B Apache-2.0 expressive voice-design, emotion tags. English-only, 16 GB+ VRAM — gate behind platform tiers. |
Tier 3b — STT / Capture Candidates (06-27 sweep)
Feeds the planned streaming-transcription roadmap; Whisper alternatives.
| Priority | Item | Notes |
|---|---|---|
| 1 | Nemotron 3.5 ASR Streaming 0.6B | Cache-aware streaming FastConformer-RNNT, 40 locales, MLX path. Strongest streaming-dictation fit. Verify license + non-CUDA perf. |
| 2 | Cohere Transcribe 03-2026 | 2B Apache-2.0, 14 langs, ONNX/INT8 across CPU/Apple Silicon/GPU. Cleanest offline /transcribe candidate. |
| 3 | ARK-ASR 3B / 0.6B | Broad multilingual, INT8 ONNX for edge. Very new; likely trust_remote_code — validate stability. |
| 4 | IBM Granite Speech 4.1 2B / NAR | ASR + speech translation (en/fr/de/es/pt/ja). Compelling if we expand into translation. |
Previously Prioritized — Now Done
Kokoro 82M — finish integrationShipped (PR #325)Qwen CustomVoiceShipped (PR #328)Intel Arc (XPU) supportShipped (PR #320)Blackwell CUDAShipped (PR #401, follow-up work open)Generation cancellationShipped (PR #444)macOS Intel x86_64Shipped (PR #416)
Branch Inventory
| Branch | PR | Status | Notes |
|---|---|---|---|
voicebox-new-models | — | Active | New model research (Fish Speech, Pocket TTS, VibeVoice, etc.); VoxCPM evaluated & backlogged |
fix/kokoro-pyinstaller-source-files | — | Active | Kokoro frozen-build source bundling (parent of voicebox-new-models) |
feat/cosyvoice-engine | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
feat/kokoro | #325 | Merged | Kokoro 82M + voice profile type system |
feat/qwen-custom-voice | #328 | Merged | Qwen CustomVoice preset engine |
feat/chatterbox-turbo | #258 | Merged | Chatterbox Turbo + per-engine languages |
feat/chatterbox | #257 | Merged | Chatterbox Multilingual |
feat/luxtts | #254 | Merged | LuxTTS + multi-engine arch |
Quick Reference: API Endpoints
All current endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/health | GET | Health check, model/GPU status |
/profiles | POST, GET | Create/list voice profiles |
/profiles/{id} | GET, PUT, DELETE | Profile CRUD |
/profiles/{id}/samples | POST, GET | Add/list voice samples |
/profiles/{id}/avatar | POST, GET, DELETE | Avatar management |
/profiles/{id}/export | GET | Export profile as ZIP |
/profiles/import | POST | Import profile from ZIP |
/generate | POST | Generate speech (engine param selects TTS backend) |
/generate/stream | POST | Stream speech (MLX only) |
/history | GET | List generation history |
/history/{id} | GET, DELETE | Get/delete generation |
/history/{id}/export | GET | Export generation ZIP |
/history/{id}/export-audio | GET | Export audio only |
/transcribe | POST | Transcribe audio (Whisper) |
/models/status | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
/models/download | POST | Trigger model download |
/models/download/cancel | POST | Cancel/dismiss download |
/models/{name} | DELETE | Delete downloaded model |
/models/load | POST | Load model into memory |
/models/unload | POST | Unload model |
/models/progress/{name} | GET | SSE download progress |
/tasks/active | GET | Active downloads/generations (with inline progress) |
/stories | POST, GET | Create/list stories |
/stories/{id} | GET, PUT, DELETE | Story CRUD |
/stories/{id}/items | POST, GET | Story items CRUD |
/stories/{id}/export | GET | Export story audio |
/channels | POST, GET | Audio channel CRUD |
/channels/{id} | PUT, DELETE | Channel update/delete |
/cache/clear | POST | Clear voice prompt cache |
/server/cuda/status | GET | CUDA binary availability |
/server/cuda/download | POST | Download CUDA binary |
/server/cuda/switch | POST | Switch to CUDA backend |