Competitive Analysis

Compiled 2026-06-11 from four parallel research passes (one per competitor repo + a full feature-surface inventory of this codebase). Star counts and versions are as-of-date snapshots. VoiceStudio grades (A–D) come from the self-inventory: code signals, test coverage, TODO density, and open-issue mentions — not marketing.

Expanded 2026-06-11 (second pass, six additional research agents): second-tier landscape survey, source-level deep dives into the flagged components of all three competitors (clean-room functional specs for the GPL one), implementation specs mapping every ranked action onto this codebase, and user-sentiment / market-positioning research. Three grades in the matrix were corrected where the original self-inventory was stale — see the matrix legend.

Expanded again 2026-06-12 (third pass): the Roadmap directions section grounds the feature roadmap announced in discussion #346 — agentic voice workflows, remote GPU backends, audiobook creator, persona gallery, model/env + GPU-compat management — in landscape, license, and regulatory research, with scope ladders and eight new consolidated actions (15–22).

TL;DR

	voicebox	pyvideotrans	Patter
What	Local-first voice studio (ElevenLabs + WisprFlow alt) — our most direct competitor	Desktop video translate/dub pipeline (GUI + CLI)	Telephony voice-agent SDK (self-hosted Vapi/Retell alt) — adjacent, not competing
Stack	Tauri v2 + React + FastAPI (same as us)	PySide6 (Qt) + FFmpeg, 100 % Python	Python + TS dual SDK
Maturity	v0.5.0, ~29.7k★, fast cadence, beta-grade hardware backlog	V4.01, ~17.9k★, 2.5 yrs mature, monthly releases	v0.6.x, ~511★, 2 months old, exceptionally well-engineered
License	MIT	GPL-3.0	MIT
Code reuse for us	✅ Port directly (keep MIT attribution header)	⚠️ Reimplement ideas only — never copy	✅ Port directly (keep MIT attribution header)

Second-pass promotion: KrillinAI / KlicStudio (Go, ~10.3k★, GPL-3.0, v2.0.3 released 2026-06-09, only 28 open issues) is now rated as direct a competitor as voicebox for the dubbing user — desktop+web+CLI form factors and a deliberate "for AI agents" strategy that competes with our MCP angle. GPL-3.0 → study-only. Full profile in the second-tier landscape below; it has not been folded into the per-capability matrix because we have not done a capability-by-capability source pass on it yet.

License ground rule. VoiceStudio is AGPL-3.0-only with a commercial dual-license offering. MIT code can be incorporated (attribution preserved) and stays compatible with selling commercial exceptions. GPL-3.0 code is technically combinable with AGPL-3.0 (GPLv3 §13), but copied GPL files stay GPL-3.0 forever under the original author's copyright — which would break the commercial-license model (we can only sell exceptions for code we own). So pyvideotrans is a design-document, not a code source: study _rate.py, write our own. The same logic applies to third-party AGPL code (e.g. alltalk_tts): even though we are AGPL ourselves, we cannot sublicense someone else's AGPL code under our commercial exception — study-only, same as GPL.

Fun fact discovered en route: pyvideotrans already integrates VoiceStudio as a first-class TTS/clone backend (videotrans/tts/_omnivoice.py, via our Gradio API). We are upstream for 17.9k-star project users. Second-pass update: the integration is verified broken — it speaks Gradio to an endpoint we never exposed. Details in the pyvideotrans deep dive; the action item is now "ship a compat shim or upstream a REST integration", not "verify".

---

Big feature matrix

Legend — Us: A–D maturity grade from the self-inventory. Them: ✅ stable · 🟡 beta/partial · ❌ absent. (bold = they beat us; this is the gap list) Second-pass corrections (marked △): three "Us" cells in the original were stale — the self-inventory missed scripts/validate-install-docs.py, the probe-judge + omnivoice/eval/ stack, and the real FastMCP server at backend/mcp_server.py.

Capability	Us	voicebox	pyvideotrans	Patter	Notes
Generation & cloning
Zero-shot voice cloning	B	✅	✅ (via clone-TTS engines)	❌	Parity; their multi-sample profiles are slightly ahead
Preset voice library (no reference audio)	B+ (20+ archetypes)	✅ 50+ presets (Kokoro/Qwen)	❌	❌	They win on count, we win on curation + degenerate-check
Voice design from text description	B	🟡 (personality descriptors)	❌	❌	We're ahead (#317 shipped a deterministic mapper)
Unlimited-length generation (chunk + crossfade)	❌ (no auto-chunking)	✅ chunked_tts.py	✅ (per-subtitle by design)	❌	Gap. Their crossfade chunker removes the length ceiling
Paralinguistic tags ([laugh], [sigh])	B (13 native reaction tags via ⊕ Insert; no [breath] yet)	✅ (Chatterbox Turbo)	❌	❌	Near-parity — see docs/expressive-speech.md; CosyVoice 3 adds [breath]/[laughter]
Delivery instructions ("whisper", "slowly")	B (instruct field)	✅ (Qwen NL control)	❌	❌	Parity-ish
Generation queue w/ cancel + SSE	B+ (job store, SSE replay)	✅	✅ (9-queue pipeline)	n/a	Parity; our SSE reconnect-replay is ahead of voicebox
Post-processing FX chain (reverb/pitch/comp)	B (effect chain exists)	✅ Pedalboard, per-profile presets	❌	❌	Theirs is richer + has preset UX
Multi-track timeline editor (stories/podcasts)	❌	✅ Stories editor (v0.5.0)	❌	❌	Gap — also the #280-item-3 timeline ask
Audio watermarking (AudioSeal)	B	❌	❌	❌	We're unique here
Dubbing pipeline
Full video dub (ASR→translate→TTS→mux)	A–	❌	✅ (1200-line battle-tested pipeline)	❌	Two-horse race; we're competitive
Incremental re-dub (change 1 line, regen 1 segment)	A– (#281 fixed)	❌	❌	❌	We're unique here
Dub-length fitting (audio speedup + video slowdown)	A– (Smart Fit complete: planner + generate path + fit fingerprints + two-tier video-retime export with drift absorption and fitted subtitles)	❌	✅ _rate.py — the crown jewel	❌	Gap closed — Action 1 reimplemented clean-room (services/fit_planner.py, services/video_retime.py, Spec 1)
Vocal/BGM separation + re-mix	A– (Demucs 4-stem)	❌	✅ (UVR/Spleeter ONNX)	❌	Parity; their ONNX models are lighter than Demucs
Clone refs cut from separated vocals per segment	🟡 (speaker_clone refs 5–15 s/speaker)	❌	✅ per-subtitle-line refs	❌	Their per-line granularity beats our per-speaker. Action 4
Speaker diarization → multi-voice dub	B+ (pyannote)	❌	✅ (4 backends incl. CAM++)	❌	Parity; their backend choice is wider
Second-pass ASR on dubbed audio (regenerate exact subtitle timings)	❌	❌	✅	❌	Gap — clever QC step. Action 5
Subtitle styling / burn-in / dual-language	A– (#309 fixed)	❌	✅	❌	Parity
Batch processing (N videos)	B (50-job queue)	❌	✅ (wave control, multi-GPU scaling)	❌	Their batch_nums waves + per-GPU thread scaling is ahead
Translation channel breadth	B (LLM 3-step chain + glossary)	❌	✅ ~25 channels	❌	Breadth vs depth: our reflect/adapt chain is deeper, their coverage wider
Translation caching + line-count validation	🟡 (fingerprints #281)	❌	✅ MD5 cache + timeline re-match	❌	Worth studying
Dictation
Global-hotkey dictation pill	B+ (#323 fixed)	✅ (v0.5.0, auto-paste macOS-only)	❌	❌	We're ahead on cross-platform (their gap violates our parity rule)
LLM transcript refinement (filler-word removal)	❌	✅ local Qwen3 0.6B–4B	❌	❌	Gap. Action 3
Captures library (replay / re-transcribe / refine)	🟡 (transcription history page)	✅ richer (v0.5.0)	❌	❌	Partial gap — we store, they iterate
Dictation while audio plays (echo cancel)	✅ opt-in (Settings → Capture): server-side NLMS AEC on /ws/transcribe?aec=1 + AudioWorklet PCM mic + player far-end tap — Action 8	❌	❌	✅ NLMS AEC	Ported Patter's NLMS canceller. Action 8
Engines & platform
TTS engine count	B (6)	✅ 7	✅ 33 channels (22 ASR, 25 translate)	✅ 7 (cloud)	pyvideotrans = breadth king (incl. cloud); we + voicebox are local-only by design
Engine plugin protocol	B+ (ABC + registry)	✅ Protocol + ModelConfig registry, agent skill for adding engines	✅ lazy dataclass plugins	✅ provider SDK	Everyone converged on the same pattern; their requires_cuda-gap lesson is free for us
MLX runtime on Apple Silicon	🟡 (MLX-Audio engine only)	✅ MLX for TTS+STT, 4–5× claimed	❌	❌	Gap — dual-runtime per engine. Action 6
CUDA binary auto-download (small installer)	❌ (venv on first run ships everything)	✅ in-app CUDA swap incl. sm_120	❌	❌	Different bootstrap philosophy; their #1 bug source too. Study only — failure-mode autopsy in the deep dive
Crash-isolated engine subprocesses	🟡 (Demucs/ffmpeg subprocesses)	❌	✅ (whisper.cpp etc. in child procs)	❌	Their JSON-log polling pattern is a cheap stability win. Action 7
ROCm support	A– (with edge cases)	🟡 (large breakage backlog)	🟡	n/a	We're ahead
Integration surface
OpenAI-compatible API	B+	✅ REST	❌	n/a	Parity
MCP server (agent speaks in your voice)	△ B– (FastMCP backend/mcp_server.py: 4 tools, stdio + SSE; not mounted on the main app, no per-agent voice binding)	✅ FastMCP at /mcp + stdio shim, per-agent voice binding	❌	✅ (client + server)	Gap is narrower than originally graded; what's missing is exactly the half voicebox shipped. Action 2
Web/Docker deployment	B– (headless image exists)	✅ (docker compose up)	❌ (desktop only)	✅	Parity-ish; our :latest/:stable retag (PR #338) helps
CLI / headless batch	🟡 (API only)	❌	✅ cli.py (stt/tts/sts/vtv)	✅	Partial gap for power users
Streaming TTS (websocket, low TTFA)	C+ (/ws/tts experimental)	❌	❌	✅ sentence-chunked streaming, first-flush	Patter's chunker + first-flush are portable. Action 8
Ops & quality discipline
Eval harness for output quality	△ 🟡 (probe judges tests/probe/judges/ + omnivoice/eval/ WER/MOS/speaker-sim — deterministic tier exists, no semantic/LLM-judge tier)	❌	❌	✅ LLM-judge evals + CLI	Patter's harness adds the missing semantic tier. Action 9
Docs-drift CI	△ 🟡 (scripts/validate-install-docs.py gates docs/install/*.md in ci.yml — no inventory-wide drift job)	❌	❌	✅ daily inventory-vs-docs diff job	Patter's rolling-issue automation is the missing half. Action 9
Model-evaluation decision log	🟡 (ROADMAP phases)	✅ PROJECT_STATUS.md accepted/abandoned log	❌	❌	Cheap practice to adopt
Telemetry design (consent-bounded)	n/a (opt-in GH Issues only)	❌	❌	✅ consent module, bucketed values	Reference design for the bug reporter — allowlist pattern in the deep dive

Where we are unique (defend these)

• Incremental re-dub with fingerprint tracking — nobody else has it.
• 646-language claim via VoiceStudio model — voicebox tops out at 23, pyvideotrans is engine-dependent.
• AudioSeal watermarking + detection — unique among all three.
• Cross-platform dictation as a default (their auto-paste is macOS-only).
• 3-step LLM translation chain (translate → reflect → adapt) + glossary — deeper than anyone's single-pass.

Second-pass reality check: market evidence per item — which of these users actually ask for — is in User sentiment & positioning. Short version: (a) and (d) are strong levers, (b) is a reach lever with a quality-risk tail, (c) and (e) are real but nobody searches for them by name.

---

Second-tier landscape

Surveyed 2026-06-11; stars / last-push / open-issue counts verified via the GitHub API. These either compete for the same user or carry portable ideas, but none (except KrillinAI) warrants a per-capability matrix column yet.

Project	What	Stack	Maturity (2026-06-11)	License	Overlap w/ us	Verdict
KrillinAI/KlicStudio	LLM video translate+dub for humans and AI agents	Go + web/desktop/CLI	10.3k★, v2.0.3 2026-06-09, 28 issues	GPL-3.0	Dubbing, cloning, API server	Direct competitor (promoted); reimplement only
VideoLingo	One-click "Netflix-grade" subtitle + dubbing pipeline	Python/Streamlit, cloud LLM	17.4k★, v3.0.1 2026-02, 208 issues	Apache-2.0	Dubbing, cloning (GPT-SoVITS), subtitles	Direct competitor (web, cloud-LLM-dependent); port OK
voice-pro	All-in-one local cloning/TTS/Whisper/Demucs/translate WebUI	Python/Gradio (Windows-leaning)	10.9k★, push 2025-12, 47 issues	GPL-3.0	Cloning, TTS, STT, separation, translation	Direct competitor (web form factor, 6 months quiet); reimplement only
ebook2audiobook	Ebook → audiobook w/ cloning, 1,158+ languages	Python/Gradio + CLI/Docker	19.2k★, push 2026-06-11, 10 issues	Apache-2.0	Cloning, multi-engine TTS	Adjacent vertical; port OK
GPT-SoVITS	Few-shot cloning + TTS WebUI w/ training tools	Python/Gradio + API	58.6k★, push 2026-04, 872 issues	MIT	Cloning, TTS (we already wrap it as GPTSoVITSBackend)	Component + partial competitor; port OK
SoniTranslate	Gradio video-dubbing pipeline w/ diarization + cloning	Python/Gradio	1.4k★, push 2026-04, no release since 2024-05, 122 issues	Apache-2.0	Dubbing, cloning, multi-TTS (we sidecar it)	Component + adjacent; port-with-attribution OK
Speaches	OpenAI-compatible local STT+TTS+Realtime server	Python/Docker	3.4k★, push 2026-06-10, 136 issues	MIT	API-server surface only	Component-not-competitor; port OK

Notable ideas / gaps per project

KrillinAI / KlicStudio — the promotion case: released two days before this survey, 28 open issues at 10.3k★ (most operationally disciplined direct competitor), and a skills/ Agent-Skills framework exposing stable CLI contracts + JSON artifact manifests so AI agents drive each pipeline stage independently — it validates and extends our MCP-server direction. Also: portrait/landscape re-rendering + cover generation for TikTok/Shorts, a social-publishing tail we ignore. GPL-3.0: ideas only.

VideoLingo — its "Translate-Reflect-Adaptation" chain is the same 3-step idea as our translator.py chain (convergent evolution; theirs predates the comparison — worth a diff of prompt strategies, Apache-2.0 so even portable). Netflix-standard subtitle segmentation with word-level alignment overlaps our subtitle_segmenter.py. Per-step pause/resume/stop on long pipelines is a UX gap we have. Counter-positioning: it requires a cloud LLM key — our local-first translation is the differentiator to message against it.

voice-pro — closest single-app feature overlap in the tier (cloning + TTS + Whisper + Demucs + YouTube ingest + translation). Proves 10k+★ demand for exactly our bundle in a clunkier (Gradio, Windows-leaning, GPL) package; quiet since 2025-12 — its users are capturable if it stalls. The recurring expectation across this whole tier (voice-pro, VideoLingo, KrillinAI, SoniTranslate): built-in YouTube-download → process loop. Our URL-ingest dub path covers part of this.

ebook2audiobook — the long-form vertical we don't serve: chapterized m4b output, inline SML tags (break/pause/voice-switch mid-text), OCR'd PDFs, per-file voice mapping in batch. A "narrate a whole book" mode is a credible extension (Apache-2.0, portable). Separately: 10 open issues at 19.2k★ is the best issue-hygiene benchmark in the landscape — study their template/triage setup for our bug-reporting milestone.

GPT-SoVITS — dual role: an engine we wrap and a competitor for the DIY cloning user. The big capability we lack vs. that crowd is its in-app fine-tuning chain (dataset slicing → ASR → labeling → train); MIT, so the training-tool code is portable. Version churn (v2/v2Pro/v3/v4) is a live threat to our GPTSoVITSBackend engine-compat constraint — pin and contract-test the API surface.

SoniTranslate — we already depend on it: backend/services/sonitranslate.py runs it as an isolated Gradio sidecar (port 7860) via gradio_client. Apache-2.0 means we may also vendor its code in-tree with attribution if sidecar reliability ever becomes a problem (process-level integration carries no derivative-work questions at all, so the current setup is the safest). Risk: its release cadence stalled (v0.5.0, May 2024) while we depend on it at runtime — consider pinning a fork.

Speaches — "Ollama for audio": dynamic model auto-load/unload per request is directly applicable to our multi-engine VRAM juggling (MIT, portable). Its OpenAI Realtime-API emulation is the compatibility surface third-party clients will eventually ask our API server for. From the same bucket: Kokoro-FastAPI (Apache-2.0, 5.0k★) ships weighted voice mixing (blend voices by ratio) — a cheap, differentiating voice-design feature. New Action 12.

Watch items (not competitors)

• TTS-WebUI (MIT, 3.2k★, active) — a 40+-model local audio hub whose repo description already advertises a VoiceStudio extension. Verify what that extension wraps and that AGPL terms are respected. Its per-extension uv venv isolation parallels our sidecar approach.
• F5-TTS (14.7k★) — engine candidate, not an app. Trap: code is MIT but pre-trained weights are CC-BY-NC — an F5-TTS engine would need prominent non-commercial-weights labeling like our existing engine-license gates.
• alltalk_tts (AGPL-3.0, 2.4k★, solo maintainer) — adjacent TTS server. Third-party AGPL = study-only for us (see license ground rule). Its narrator/character voice-switching markup and low-VRAM modes are reimplementable ideas.
• Linly-Dubbing (Apache-2.0, 3.2k★, ~15 months stale) — idea quarry only; its lip-sync integration (re-syncing mouth movements to the dubbed track) is the one feature nobody in the landscape, us included, ships.
• resemble-ai/chatterbox (~25.0k★, MIT) — engine candidate; full evaluation in the appendix.

---

Ranked actions

Effort: S < 1 day · M = 1–3 days · L = 1–2 weeks. "Port" = copy + adapt MIT/Apache code with attribution header. "Reimplement" = clean-room from the functional spec in this doc — do not open pyvideotrans source files while writing ours (the specs below were written by a dedicated spec pass precisely so implementers never have to). Each action now links to an implementation spec mapped onto our codebase.

#	Action	From	Mode	Effort	Why now
1	Dub-length fitting v2: absorb inter-segment silence slack → if speedup ≤ 1.2× stretch audio only (pitch-preserving) → else split burden ~50/50 with per-segment video slowdown → regenerate subtitle timeline from actual dub durations → freeze last frame for drift	pyvideotrans _rate.py design	Reimplement (spec)	L	Our #280 onset-snapping is a band-aid; this is the algorithm that makes dubs fit. Highest user-visible quality win available
2	MCP server v1: mount the existing FastMCP server on the main FastAPI app + stdio shim + per-agent voice binding	voicebox backend/mcp_server/, mcp_shim/	Port (spec)	M	We already have the FastMCP half (backend/mcp_server.py); voicebox shipped exactly the missing half. Agents-speak-in-your-voice is organic marketing
3	Dictation refinement: deterministic repetition-collapse pre-pass + optional local-LLM filler-word removal before paste	voicebox services/refinement.py	Port (adapt to our llm_backend.py) (spec)	M	Biggest dictation quality jump per line of code; WisprFlow's whole pitch. The deterministic pre-pass alone (fixes Whisper hallucination loops) is worth shipping first
4	Per-segment clone refs: cut the voice-clone reference for each dub segment from the separated vocal track at that segment's timestamps, instead of one 5–15 s ref per speaker	pyvideotrans per-line ref idea	Reimplement (spec)	S–M	Prosody of each line matches its source line; cheap because Demucs stems + segment times already exist
5	Second-pass ASR QC: after dub generation, re-run ASR on the synthetic audio to regenerate exactly-timed subtitles (and flag segments whose recognized text drifts from the target text)	pyvideotrans pipeline stage	Reimplement (spec)	M	Turns subtitle timing from "trusted math" into "measured truth"; doubles as an automatic dub-quality check
6	MLX runtime pass: route Whisper + at least one TTS engine through MLX on Apple Silicon via backend-aware model-repo mapping	voicebox mlx_backend.py pattern	Port pattern (spec)	L	M-series Macs are a huge slice of local-AI users; 4–5× claimed speedup
7	Crash-isolated ASR subprocess: run native-crashy engines (whisper.cpp class) in a single-use child process, so a segfault never kills the backend	pyvideotrans subprocess pattern	Reimplement (spec)	M	Directly serves "first-run that actually works"; engine crashes become per-job failures. The sentiment pass confirmed whisper GPU-teardown crashes are endemic to the category
8	Streaming polish kit: sentence-aware chunker (abbreviation + multilingual punctuation, aggressive first-flush for TTFA) for /ws/tts + NLMS AEC so dictation works during playback	Patter sentence_chunker.py, aec.py	Port (spec)	M	Moves /ws/tts from C+ experiment toward production; AEC unlocks dictate-over-playback
9	Quality rails: (a) extend our existing install-docs gate into a daily inventory-vs-docs drift job with rolling-issue automation; (b) LLM-judge eval harness as a non-gating semantic tier above our deterministic probe judges	Patter docs-feature-drift.yml, evals/	Port (spec)	S + M	(a) we already have validate-install-docs.py — this is the second half; (b) gives dub translation quality what the probe judges give DSP correctness
10	Practice adoptions (no code): PROJECT_STATUS.md-style engine decision log; platform-gating flag audit in our engine registry (pre-empt voicebox's top bug class); add Chatterbox Turbo to the engine roster for paralinguistic tags	voicebox	n/a	S each	Cheap, compounding
11	Fix the pyvideotrans bridge: their _omnivoice.py integration is verified broken (speaks Gradio /_clone_fn to our FastAPI :3900 — hard-stops on connect). Either ship a small Gradio-compatible shim or upstream a REST/OpenAI-style integration PR to pyvideotrans	pyvideotrans	Build/PR (spec)	S–M	17.9k★ project routing users to us — the bridge is down right now
12	Weighted voice mixing: blend preset/cloned voices by ratio as a voice-design primitive	Kokoro-FastAPI (Apache-2.0)	Port	S–M	Cheap, differentiating, and feeds the voice-design surface where we're already ahead
13	Dynamic engine load/unload: per-request model lifecycle (load on demand, unload on idle/VRAM pressure) — 🟡 idle-unload shipped for both engine classes: in-process default model via model_manager.idle_worker; subprocess-engine sidecars via subprocess_backend.reap_idle_sidecars (lock-guarded, OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S, default 300 s). Cross-engine VRAM-pressure preemption (unload B to make room for A) still TODO	Speaches (MIT)	Port pattern	M	Multi-engine VRAM juggling is our chronic background pain; "Ollama for audio" solved it
14	Positioning kit (no code): ElevenLabs-dubbing-cost comparison page + incremental-re-dub demo GIF; Show HN; name-collision disambiguation; "WisprFlow alternative" docs entry point	sentiment research	n/a	S each	Grounded in what users actually search; details in positioning moves

Explicitly not recommended

• Copying any pyvideotrans / KrillinAI / voice-pro code — GPL-3.0 files would stay GPL under their author's copyright inside our AGPL tree and break the commercial dual-license. Ideas are fair game; code is not. Same for third-party AGPL code (alltalk_tts).
• Cloud TTS/ASR/translate channel breadth (pyvideotrans's 33/22/25) — violates the local-first constraint. Our breadth play is local engines only.
• voicebox's CUDA-binary-swap bootstrap — their own top bug category; our uv-venv bootstrap is healthier. The deep dive below documents six concrete failure modes to avoid; the lessons transfer to our sidecar pattern, the mechanism does not.
• Patter's telephony stack — different product. Only the audio/streaming/ops pieces above are relevant.
• Patter's opt-out telemetry default — its allowlist/bucketing design informs our opt-in bug reporter (see deep dive), but the consent default and the third-party endpoint both violate our local-first constraint.
• F5-TTS as an unlabeled engine — MIT code, but CC-BY-NC weights; only with the same prominent license gate we use for OpenRAIL engines.

---

Deep dives

Source-level briefs from the second research pass. voicebox and Patter are MIT — briefs may quote and porting means copying with attribution. The pyvideotrans section is a clean-room functional spec: written by a dedicated spec pass that read the GPL source so implementers never have to; it contains no code, only behavior and constants. File paths are repo-relative, with approximate line ranges, for verification only.

voicebox (MIT — portable)

1. Chunked long-form TTS (backend/utils/chunked_tts.py, 299 LOC)

Engine-agnostic wrapper around any generate(). Text ≤ max_chunk_chars (default 800, per-request overridable) takes a zero-overhead single-shot path. Longer text splits greedily left-to-right with descending boundary priority: sentence end ([.!?] + CJK equivalents, skipping a 25-entry abbreviation set and decimals) → clause boundary (;:,—) → whitespace → safe hard cut. Paralinguistic tags ([laugh]) are atomic — a regex guards every split candidate. Per-chunk seed is seed + i (decorrelates RNG artifacts, keeps (text, seed) reproducible). Optional per-chunk trim_fn for engines that hallucinate trailing noise. Joining is a linear crossfade, default 50 ms (0 = hard cut), overlap clamped to the shorter side. Known latent bug: sample rate is taken from the first chunk; later mismatches are silently ignored. No per-chunk failure recovery — any chunk exception fails the whole generation. Deps: numpy only.

Porting: new backend/utils/ module invoked from our tts_backend.py synthesis orchestration — it only needs an awaitable generate() returning (ndarray, sr), which our TTSBackend ABC already provides. Make the abbreviation set and sentence-end regexes language-aware for our 646-language scope; wire max_chunk_chars/crossfade_ms into the request models. Not a fit for streaming /ws/tts as-is (batch concat, not incremental emit), but split_text_into_chunks alone is reusable there. Effort: S.

2. Dictation LLM refinement (backend/services/refinement.py, 295 LOC + capture plumbing)

Local Qwen3 0.6B/1.7B/4B (mlx-community 4-bit on Apple Silicon, upstream elsewhere), default 0.6B, temperature=0.2, thinking off. Prompts are assembled from three boolean toggles (smart_cleanup, self_correction, preserve_technical, all default on) — no raw prompt editor. The base instruction frames the model as "a text filter, not an assistant" with explicit anti-instruction-following rules. Seven few-shot examples are passed as structured chat turns, not inline prompt text — small models pattern-match and echo inline examples; ordering is recency-weighted with hardest rules last.

The underrated half is a deterministic pre-pass, collapse_repetitive_artifacts(): strips Whisper hallucination loops before the LLM — word-level (token repeated ≥6×) and character-level (any 2–60-char unit repeating ≥6×, catches CJK loops with no spaces). Rhetorical repeats below 6 survive. This alone fixes the classic Whisper "thanks for watching" loop and works with no LLM configured.

Flow: record → full-audio STT (raw transcript persisted) → if auto-refine, one LLM round-trip → refined-or-raw text is what auto-pastes. Refinement is re-runnable with different flags. A readiness endpoint checks on-disk model presence so the global hotkey never hangs on an un-downloaded model; refine errors leave the raw transcript standing.

Porting: maps almost 1:1 onto our dictation stack. Pre-pass + prompt builder → new backend/services/refinement.py; the LLM call goes through our existing llm_backend.py OpenAI-compat adapter (Ollama/LM Studio) instead of an in-process Qwen — simpler: the few-shot examples become standard messages pairs. Hook after the final transcript in capture_ws.py, never on partials. Readiness becomes "is an LLM endpoint configured/reachable". Needs Settings toggles + i18n keys. Effort: M (service S; UX, persistence, ws-protocol addition make it M).

3. MCP server + stdio shim (backend/mcp_server/ ~650 LOC, shim ~200 LOC)

FastMCP mounted on the main FastAPI app at /mcp (Streamable HTTP); FastMCP's session manager must run inside the ASGI lifespan — they stack their existing startup/shutdown with FastMCP's via AsyncExitStack. Tools (dotted names): voicebox.speak(text, profile, engine, personality, language) → async {generation_id, poll_url}; voicebox.transcribe(audio_base64 | audio_path) — audio_path restricted to loopback callers (so a 0.0.0.0-bound server isn't an arbitrary-file-read primitive), 200 MB cap; voicebox.list_captures, voicebox.list_profiles.

Per-agent voice binding — the headline: every MCP client sends an X-Voicebox-Client-Id header (from its MCP config, or forwarded by the shim from an env var). Middleware copies it into a ContextVar so tool handlers read it without plumbing, and stamps last_seen_at. A mcp_client_bindings table holds per-client {label, profile_id, default_engine, default_personality} — "Claude Code speaks in Morgan, Cursor in Scarlett." Resolution precedence: explicit tool arg → per-client binding → global default → helpful error. Bindings managed from Settings over plain REST. The stdio shim is a ~200-line stdio↔HTTP proxy (waits for /health, relays JSON-RPC, captures/replays the MCP session id, maps HTTP errors to JSON-RPC errors); only dep is httpx.

Porting: we already have the FastMCP half (backend/mcp_server.py, stdio+SSE, 4 tools) — this is precisely the missing half. Order: (1) mount + lifespan composition on our main app (wrap our startup hooks, don't replace); (2) client-id middleware + bindings table (alembic migration per the backward-compat constraint) + resolve chain over our voice profiles; (3) shim nearly verbatim. The loopback gate on file-path tools is a security pattern worth copying anywhere. Effort: M.

4. MLX runtime (backend/backends/mlx_backend.py, 367 LOC + factory routing)

Not per-engine plugins — a fork inside the lazy backend factory: only Qwen TTS, Whisper STT, and the Qwen3 LLM branch on get_backend_type(); everything else is torch-only everywhere. Detection: Darwin+arm64, then import mlx.core inside try/except catching ImportError, OSError, RuntimeError — in a PyInstaller bundle the native dylib/metallib can fail to load even when the package imports. The fork extends to model selection: the same model_size key resolves to mlx-community/... weights on MLX (~3× smaller downloads) vs upstream weights on torch. Backends are duck-typed Protocols; inference runs via asyncio.to_thread; clone failure degrades to generation without the voice prompt. Claimed 4–5× speedup is README-grade (no benchmark file in the repo).

Porting: we already have MLXAudioBackend in the registry, so the structure exists. Worth stealing: (a) backend-aware ModelConfig repo mapping — one engine key, platform-resolved artifact; (b) the robust import mlx.core probe (we will hit the same PyInstaller failure in bundled builds); (c) MLX Whisper for the dictation path — we already have MLXWhisperBackend in asr_backend.py, the gap is routing dictation through it. Cross-platform rule satisfied: MLX is implementation-level, default behavior identical. Effort: S–M.

5. Stories multi-track editor (shape only; backend/services/stories.py, 966 LOC)

Two tables: Story {id, name, description, timestamps} and StoryItem {id, story_id, generation_id FK, version_id FK nullable, start_time_ms, track, trim_start_ms, trim_end_ms, volume, created_at}. The key design move: a clip references a Generation, never copies audio — trims are non-destructive offsets, and version_id pins a clip to a specific regeneration take while sibling clips can use different takes. Split = two items sharing one generation_id with complementary trims (row-locked against double-click races). Export sums clips into a float32 buffer at sample offsets (overlaps mix additively), then peak-normalizes if needed.

Porting: the 9-column schema is the valuable part — a full multi-track NLE over our existing generation history (additive alembic migration, backward-compatible). The frontend timeline is the actual cost. **Effort: L full feature, S for data model

• export mixer alone.**

6. CUDA binary swap bootstrap — study-only failure autopsy (backend/services/cuda.py, 422 LOC + Rust launch logic)

Mechanism: CPU PyInstaller sidecar ships in the installer; a CUDA variant downloads as two independently versioned tarballs (server core, versioned with the app; CUDA libs ~4 GB, keyed on a toolkit string). At every launch the Rust shell runs the downloaded binary with --version and compares; mismatch or error → silent CPU fallback. Six failure modes documented from their tracker, all to avoid:

1. Staleness detection requires spawning the possibly-broken binary — corrupt onedir → --version fails → silently CPU, never repaired this launch. (They also paid a 30 s torch import on every version check until adding a fast path.)
2. One-launch GPU lag after every app update ("update disabled my GPU" reports).
3. Extract-over-old-dir without wiping — orphaned files from previous layouts shadow the new binary while --version still passes.
4. Auto-update vs manual download raced on the same temp file (fixed late with a lock that now silently skips user-initiated downloads).
5. Manifest-vs-disk drift: libs staleness reads a JSON manifest that survives failed or hand-deleted extractions; toolkit bumps force full 4 GB re-downloads.
6. Every failure path degrades to CPU with stdout-only logging — users discover it as "the app got slow", not as an error.

Lessons for our sidecars (subprocess_backend.py + engine bootstraps): write the version manifest atomically after successful extract — never interrogate a binary to learn its version; extract to temp dir + atomic rename; make GPU/CPU fallback loudly visible in the UI; the split-archive idea (app-versioned core vs toolkit-versioned libs) genuinely fixed their 4 GB re-download complaint and is the one piece worth keeping. Effort: n/a (declined) — this list is the deliverable.

pyvideotrans (GPL-3.0 — clean-room functional specs, no code)

1. Dub-length fitting (videotrans/task/_rate.py, ~lines 288–877)

The algorithm that makes dubs fit their slots. All constants verified against source:

Parameter	Value	Meaning
Audio-only threshold (both-mode)	1.2	Required speedup ≤ 1.2× → audio absorbs everything, video untouched
Burden split beyond 1.2×	50/50	Joint target = slot + (dub − slot)/2, applied to both audio stretch and video slowdown; caps deliberately ignored in this branch
Max audio speed (audio-only mode)	100 (default setting)	Effectively unlimited; past it, slot is overrun instead
Max video slowdown (video-only mode)	10×	Past it, video target clamps to slot × 10, dub truncated later
Video retime bias	+0.005	Slowdown factor padded to compensate frame-rounding undershoot
Retime no-op epsilon	0.001	Near-1.0 factors skip the filter
Min valid video clip	1024 bytes	Smaller output = failed; retried without retiming, then dropped
Audio stretch clamp	0.2–50.0 (rubberband); ≤2.0-step chained tempo filter as fallback	Pitch-preserving when rubberband present
Working audio format	48 kHz / 2ch / 16-bit PCM	All silence + dub segments normalized
Dub-clip silence trim	threshold = clip dBFS − 20; min silence 100 ms; keep 80 ms head / 200 ms tail	Applied to every synthesized clip, default on
First-segment snap	< 100 ms start → cut from 0	Avoids sub-frame clips when video slowdown is on
Video encode for cuts	x264 CRF 20, veryfast, GOP=1, yuv420p	Every frame an I-frame → clean concat boundaries
End-of-video freeze	last-frame clone pad = audio − video duration	Applied at final mux

Decision tree. Pre-pass (slack absorption): every segment's slot end is rewritten to the next segment's start — the silent gap after each line is donated to that line; the last line's end becomes total media duration. Failed/missing dubs get a silent placeholder exactly one slot long so the pipeline never stalls. Per-line: dub ≤ slot → untouched. Otherwise in the combined mode: ratio ≤ 1.2 → audio-only compress to exactly the slot; ratio > 1.2 → both sides meet at slot + overflow/2. Execution: audio stretches run on a CPU process pool; a stretch is skipped if the target exceeds current length (silence padding covers that). Each line becomes an independent cut of the original video (seek, window, retime, hard duration limit); a pre-roll clip covers 0 → first line; failed clips are re-cut without retiming, then dropped; clips concat-copied in order. Timeline regeneration: the measured duration of each generated clip (not the requested target) becomes the line's final slot; a running cursor rewrites every subtitle to start at the cursor and end at cursor + slot — subtitles track actual dub placement exactly; overruns are hard-truncated when video slowdown is active, appended whole otherwise; short dubs get tail silence. Overlapping segments never reach this stage (an ASR post-fix clamps each line's end to the next line's start upstream). A TTS-only variant forces fit to the slot and restores the original timeline on output.

2. Per-subtitle-line clone references (trans_create.py ~882–1009)

Reference source priority: separated vocal stem → fresh mono 44.1 kHz extraction from the original media. Cut timestamps are the line's original SRT start/end (before gap-absorption rewriting). One 16 kHz/16-bit wav per line, indexed by line number; the source-language subtitle text at the same index rides along as the ref transcript. Thread pool of min(8, line count, CPU count); all cuts complete before TTS starts. Deliberately no min/max duration and no neighbor-borrowing — a 400 ms subtitle yields a 400 ms reference and quality degradation is accepted; engines that can't handle it fail that line only (a dub job fails only when zero lines succeed). Dub-file cache key: MD5 of text + role + rate + volume + pitch + channel id — lines whose output already exists are skipped entirely.

3. Second-pass ASR QC (trans_create.py ~419–482)

Runs after alignment, before final mux, opt-in, only when source ≠ target language and the subtitle-embed mode doesn't require matched line counts. The assembled dub track is downsampled to 16 kHz mono and re-recognized with deliberately halved VAD windows (min speech halved with 500 ms floor; max speech halved; min silence halved, clamped 50–1000 ms) so recognized lines come out short; short-line merging is disabled. The resulting SRT wholesale replaces the target-language subtitle file — recognized text wins unconditionally; the design accepts ASR drift in exchange for frame-accurate display timing (dub audio is already rendered and untouched). Every failure is a silent skip leaving the align-stage subtitles in place, so this stage can never break a job. Engine falls back to faster-whisper large-v3-turbo when the user's ASR channel can't do the target language.

Our spec deviates here (see Spec 5): we want drift flagged, not silently accepted.

4. Translation cache + line-count validation (videotrans/translator/_base.py ~46–175)

Cache: one text file per batch (5 lines plain mode / 20 lines or whole-file SRT mode), keyed by MD5 of channel id + API URL + srt-mode flag + model + source lang + target lang + serialized batch content — any config or content change is the invalidation mechanism (no TTL). Empty results are never cached; an all-empty batch set raises. Line-count defense, plain mode: response split on newlines, extra lines discarded, short responses padded with empty strings, then mapped 1:1 by index. SRT mode: on count mismatch (LLM merged/split lines), recovery is exact timestamp-string matching — translated cues map back by their time-range string; unmatched source cues get empty text rather than shifting subsequent lines. Source timeline is always authoritative. A final cosmetic pass strips the leading/trailing ellipsis runs LLMs tend to add.

5. Crash-isolated engine subprocesses (videotrans/configure/base.py ~170–276)

Every native-crashy stage (ASR, separation, diarization, VAD, retiming, stretch) runs in a process pool with spawn start method and one task per child — every job gets a fresh interpreter, memory fully returned, a segfault can't poison a warm worker, and "restart" semantics are free (the pool replaces dead workers automatically). CPU pool size: manual cap → min(cap, 8, CPUs); else clamp(available-RAM-GB / 4, 2, 8). GPU pool defaults to 1 (strict serialization); multi-GPU opt-in scales to min(GPU count, 8, CPUs). Progress transport: the child overwrites a single-JSON log file; a parent daemon thread polls mtime every 1 s and forwards to the UI bus; monitor gives up after ~1 h of no change. Child contract: return (result, error); falsy result or non-empty error → typed task error. Native death → the pool's broken-pool exception, decorated with model name + GPU index before surfacing. GPU selection: first card with > 24 GB free VRAM, else most-free; CUDA re-verified at submit time, unavailable → kwargs rewritten to CPU.

6. Batch wave control (videotrans/task/mult_video.py, job.py)

Nine FIFO stage queues (prepare → recognize → diarize → translate → dub → align → second-pass ASR → assemble → done), each with dedicated workers. GPU-heavy stages get 1 thread by default (2 with 2–3 GPUs in multi-GPU mode, 4 with ≥ 4); network-bound or globally stateful stages (translate, dub, align, second-pass, done) are always 1. With waves off, all videos enter the conveyor at once — video A can be translating while B is in ASR. With batch_nums > 0, the file list is chunked; a dispatcher busy-waits (1 s ticks) until every task in a chunk finishes or is stopped before releasing the next — bounding peak temp-disk and VRAM at the cost of inter-wave overlap. Inside the dub stage, lines fan out on a thread pool (dubbing_thread, default 1); per-line errors are collected and a job fails only if zero lines succeed. Stage workers convert any exception into a stage-prefixed UI error and mark the task ended, so a wave can never hang on a failed member.

7. Fact-check: the pyvideotrans ↔ VoiceStudio integration

videotrans/tts/_omnivoice.py (~12–77) speaks Gradio, not REST: it builds a gradio_client.Client against a user-pasted URL and calls the named endpoint /_clone_fn with text, a natural-language language name (~35 ISO codes mapped), a per-line reference wav + transcript, and a knob set (steps/guidance/denoise/ duration/post-process flags) that matches a 12-input Gradio clone function from an older or forked VoiceStudio build. It expects a filesystem path to a wav back.

Verdict: broken against current VoiceStudio. Our backend is FastAPI on :3900 (backend/main.py) with REST routers; we expose no Gradio app and no /_clone_fn (our only Gradio surface is the optional SoniTranslate subprocess on :7860 — a different application). A gradio client pointed at :3900 fails fetching the Gradio config, which lands in pyvideotrans's fatal "Could not fetch config" branch — users get an immediate hard stop. Their tracker already has an open bug against this integration. Remedies in Spec 11.

Patter (MIT — portable)

1. Sentence chunker (libraries/python/getpatter/services/sentence_chunker.py, 565 LOC)

Streaming sentence segmentation for low-TTFA TTS. Accumulates tokens; push(token) returns zero or more complete sentences. Boundary detection is regex marker-replacement: protect non-terminal periods (honorifics for EN/IT/ES/DE/FR/PT, website TLDs, decimals, ellipses, initials, acronym chains, company suffixes), mark real terminators, split. Terminator tables cover Latin + CJK + 8 non-Latin scripts (Devanagari, Arabic, Armenian, Ethiopic, Khmer, Burmese, Tibetan). Three emission paths: standard (≥ min length and > 1 sentence → emit all but the buffered tail); short flush for instant single-sentence replies (guards: one terminator, ≥ 1 word, no preceding digit, no ALL-CAPS acronym tail, no honorific tail); aggressive first-clause flush (opt-in, first clause of each turn only) on soft punctuation , — – at ≥ 40 chars — claimed 200–500 ms TTFA savings — with seven guards (decimals, currency within 8 chars, unbalanced brackets/quotes, ellipsis, sub-token ambiguity) and a hard disable for Italian (comma = decimal separator). Constants: min_sentence_len=20, aggressive_first_min_len=40 (comment: below ~40 chars hurts prosody; ElevenLabs buffers ~120 internally). flush() emits the remainder; reset() discards (barge-in). Deps: stdlib re only. A TS mirror + golden parity scenarios ship alongside.

Porting: feeds /ws/tts (tts_stream.py) so text synthesizes sentence-by- sentence ahead of the PCM stream; aggressive first-flush directly serves the < 100 ms TTFA goal. Distinct from subtitle_segmenter.py (offline length-balancing) but the terminator/honorific tables could become shared constants. Port the parity scenarios as pytest fixtures. Effort: S.

2. NLMS acoustic echo canceller (libraries/python/getpatter/audio/aec.py, 333 LOC)

Time-domain sample-by-sample NLMS adaptive filter with leakage + frame-wise Geigel double-talk detector. Far-end (TTS) PCM feeds a ring buffer; near-end (mic) frames get e = near − w·x as output, with the weight update frozen during double-talk (max|near| > 0.6 · max|far|) and when the far reference is near-silent (≤ −60 dBFS — raised from a smaller epsilon after a weight-blowup bug). Two-phase step schedule: mu 0.5 for the first 0.5 s, then 0.1. Pass-through guards: far buffer not primed, and a 250 ms staleness window — no recent far-end push → pass through rather than convolve against a frozen reference (was producing an audible buzz during silence). Defaults: 512 taps (= 32 ms @ 16 kHz; 2048 tested → 8–12 s convergence, rejected), leakage 0.9999, far buffer 0.5 s. I/O: int16 mono PCM, 8 or 16 kHz only, not thread-safe (one instance per session). Self-declared limitations: no frequency-domain partitioning, no residual-echo suppressor, no delay estimation — docstring recommends libwebrtc AEC3 for production-grade. Deps: numpy.

Porting: sits ahead of /ws/transcribe in capture_ws.py — push_far_end() fed from every chunk /ws/tts ships, process_near_end() on mic frames. The real work: our TTS streams 24 kHz while the canceller accepts 8/16 kHz (resample the far-end to the capture rate, or relax the check and scale taps); the staleness clock must track playout time, not send time (desktop speaker latency ≠ carrier RTT); rho likely needs tuning for loud desktop-speaker bleed. Pure numpy → identical default behavior on all three platforms. Effort: M (the port is an hour; sample-rate plumbing, delay alignment, and tuning are the work).

3. LLM-judge eval harness (libraries/python/getpatter/evals/, 1,636 LOC)

YAML/JSON suites of EvalCase {name, turns, expected_behavior, rubric, tags}; per-case error containment (a mid-case exception keeps the partial transcript and still judges it; a judge failure records score 0 + reasoning instead of aborting the suite). Judge: chat-completions with JSON response format, temperature 0, pass threshold 0.7 — with two hardening details worth copying verbatim: tolerant JSON parsing (strips code fences; invalid JSON → fail with reasoning), and the verdict is recomputed locally (passed = score >= threshold) because trusting the model's self-reported passed once let a hallucinated pass through at score 0.2. A deterministic chainable assertions layer (expect(...).tool_called(...).judge(...)) raises plain AssertionErrors so pytest reports work. CLI exits non-zero unless all cases pass — CI-gateable. The judge backend is injectable (any object with judge(prompt)).

Porting: complements our deterministic stack — probe judges (tests/probe/judges/) score DSP correctness, omnivoice/eval/ scores WER/MOS/speaker-sim; Patter's harness adds the missing semantic tier (dub translation naturalness, dictation-correction quality). Keep our "no LLM on the verdict path" rule for CI gates: run LLM-judge suites as a separate non-gating job. Swap the judge backend for our llm_backend.py (local model, keeps local-first). Port case.py + runner.py + assertions.py nearly verbatim; drop session.py (telephony) in favor of our probe Actor. Effort: M.

4. Docs-drift CI (.github/workflows/docs-feature-drift.yml 112 LOC + checker 211 LOC)

Daily cron (03:00 UTC) + manual dispatch; contents: read, issues: write. Three-way cross-reference: canonical feature inventory (theirs is an xlsx in a private sibling repo — their weakest design point, soft-failing when the token is missing) × docs filename stems × regex-parsed public SDK exports. Three drift buckets; only inventory↔docs mismatches gate (export drift is report-only). The best part is the issue automation: on failure, look up the open docs-drift-labeled issue and update its body in place (single rolling issue, no spam); create it if absent; on success, comment "drift resolved" and auto-close. Idempotent and self-healing.

Porting: we already gate docs/install/*.md via scripts/validate-install-docs.py in ci.yml — this is the second half. Replace the private-repo xlsx with an in-repo canonical inventory (a checked-in features.yaml, or generated from the engine registry + FastAPI route table); adopt the rolling-issue pattern verbatim. Note this auto-filed issue is maintainer-facing CI, distinct from the user-facing opt-in bug reporter. Effort: S (workflow is copy-adapt; defining the inventory is the only design decision).

5. Consent-bounded telemetry (reference design only; getpatter/telemetry/, 1,235 LOC)

Not adopting telemetry — Patter's default is opt-OUT with a third-party endpoint, both of which violate our local-first constraint. What transfers to our opt-in prefilled-URL GitHub bug reporter (CLAUDE.md Capability 2):

1. Two-layer key+value allowlist before an event is built: unknown keys dropped, values checked against closed enums with off-list values coerced to "other" — making a leaked custom name "structurally impossible to emit, even from a buggy caller". Stronger than regex-scrubbing after the fact; directly implements our planned token/key/home-path exclusions for the issue body.
2. Model-name sanitization: anything with separators, whitespace, or > 40 chars (fine-tune IDs, self-hosted paths) collapses to "{vendor}-other"; date suffixes stripped; final shape re-checked by regex. Apply the same to user voice-profile and engine names in bug reports.
3. Coarse buckets: counts → {0, 1, 2_3, 4_6, 7_12, 13_plus}, versions → major.minor only, OS → family only, arch → x86_64/arm64/other — explicitly anti-fingerprinting.
4. Precedence-ordered consent resolver, inverted to opt-IN for us (default OFF; Settings toggle → marker file; DO_NOT_TRACK honored as an absolute OFF), with the invariant "checking consent never writes to the filesystem".

Effort: S for the allowlist/sanitizer port (~250 LOC of pure stdlib functions); the reporter UI around it is separate work.

---

Implementation specs

Each ranked action mapped onto this codebase: integration points, shapes, and a test plan. File references verified against main as of 2026-06-11. These are work-item-grade specs, not designs — the implementer still owns the details.

Spec 1 — Dub-length fitting v2

Goal: replace "trust the math + onset snap" with the measured-fit algorithm from the functional spec above (reimplemented; do not open pyvideotrans source).

• Files: new backend/services/length_fit.py (the decision tree); integrate in the dub orchestration in backend/services/dub_pipeline.py after TTS, before mux; backend/services/speech_rate.py keeps its role as the pre-generation estimator (LLM trim/expand to fit the slot) — length-fit is the post-generation enforcer; backend/services/onset_align.py stays (it solves start alignment, not duration); video cuts/retime/concat via backend/services/ffmpeg_utils.py (respect the ffmpeg semaphore and register_proc for abortability).
• Design deltas vs pyvideotrans: keep their 1.2× audio-only threshold and 50/50 burden split as defaults but expose both in dub settings; integrate with our incremental re-dub — slack absorption must be computed over the full segment list even when only stale segments regenerate (backend/services/incremental.py fingerprints must include the two new knobs in _GEN_INPUT_FIELDS, since they affect output); time-stretch with pitch preservation (rubberband when available, chained ffmpeg tempo fallback) under the existing GPU/CPU job queue.
• Subtitle regeneration: rewrite segment times from measured clip durations with a running cursor (as specced) before dub_export.py renders SRT/burn-in.
• Tests: probe spec dub_export.probe.yaml + tests/probe/judges/dubbing.py already gate segment duration ratio [0.5–1.6×] — add a judge check that final audio fits final video ±1 frame and that regenerated SRT cue times equal measured placements; unit tests for the decision tree at the 1.2 boundary, zero-length slots, last-segment, and overlap-clamped inputs.
• Effort: L. Land the audio-only path first (pure win, no video retime), video slowdown second.

Spec 2 — MCP server v1

Goal: ship the missing half of our MCP story: mounted endpoint + per-agent voice binding + stdio shim (ported from voicebox, MIT attribution).

• Files: backend/mcp_server.py (existing FastMCP: keep tools, add transcribe); mount on the main app in backend/main.py with lifespan composition (wrap existing startup hooks via AsyncExitStack — do not replace); new middleware + ContextVar for X-VoiceStudio-Client-Id; new alembic migration for mcp_client_bindings {client_id, label, profile_id, default_engine, last_seen_at} (additive — satisfies the backward-compat constraint); REST CRUD router backend/api/routers/mcp_bindings.py; new backend/mcp_shim/ (port nearly verbatim — httpx-only stdio↔HTTP proxy); Settings UI section in frontend/src/pages/Settings.jsx; update docs/mcp.json + docs/ MCP doc.
• Resolution chain: explicit tool arg → client binding → global default profile → helpful error. Copy voicebox's loopback-only gate for any file-path-accepting tool argument.
• Tests: pytest for the resolve chain + middleware; probe spec addition driving speak/transcribe over the mounted endpoint; shim smoke test against a live backend.
• Effort: M (S shim, M mount + bindings + Settings).

Spec 3 — Dictation refinement

Goal: refined-by-default-quality dictation: deterministic artifact collapse for everyone, LLM filler-word removal for users with a local LLM configured.

• Phase 1 (S, no LLM): port collapse_repetitive_artifacts() (word-level ≥6 repeats; char-level 2–60-char units ≥6, catches no-space scripts) into a new backend/services/refinement.py; apply to final transcripts in backend/api/routers/capture_ws.py after _transcribe_buffer, never to partials. Port voicebox's test corpus pattern.
• Phase 2 (M): prompt builder with the three toggles (smart_cleanup, self_correction, preserve_technical), few-shot examples as structured messages pairs, executed through backend/services/llm_backend.py (get_active_llm_backend().chat(...), timeout-bounded). Readiness = active LLM backend reachable; on any failure the raw transcript stands. Persist both raw and refined in the transcriptions table (backend/core/db.py, additive column via alembic) so history supports re-refine. WS protocol: {type:"final", text, refined_text?, ...} — frontend pastes refined_text ?? text.
• Settings: auto-refine toggle (default ON only when an LLM backend is active — cross-platform default behavior stays identical: no LLM → identical pass-through everywhere) + the three flag toggles; i18n keys for all labels.
• Tests: unit tests for the collapse pass (incl. CJK-free fixtures using Latin repetition patterns); contract test that a dead LLM endpoint yields the raw transcript within timeout.

Spec 4 — Per-segment clone refs

Goal: per-line prosody matching — cut each dub segment's clone reference from the separated vocal track at that segment's own timestamps.

• Files: backend/services/speaker_clone.py — add extract_segment_refs(vocals_path, segments, out_dir) alongside the existing per-speaker extract_speaker_clones(); dub pipeline passes the per-segment ref to the TTS call when the engine supports reference audio.
• Design deltas vs pyvideotrans: unlike their no-floor policy, keep a quality floor — segment shorter than MIN_REF_DURATION_S falls back to the existing per-speaker 5–15 s reference (we already have it; they don't). Use original (pre-slack-absorption) segment times. Cut with the thread pool pattern already used in the pipeline; ride the source-language text along as ref transcript.
• Mode: per-segment refs default ON with per-speaker fallback; expose a dub setting to force per-speaker (long-form consistency sometimes beats per-line prosody). Add the mode to incremental.py _GEN_INPUT_FIELDS.
• Tests: unit test slicing math + fallback threshold; probe voice_clone / dub_export flows with a two-speaker fixture asserting each segment got a ref file of its own span (or the fallback).
• Effort: S–M.

Spec 5 — Second-pass ASR QC

Goal: measured subtitle truth + automatic dub-quality flagging.

• Files: new stage in backend/services/dub_pipeline.py after assembly, before export; reuse backend/services/asr_backend.py (active backend; fall back to WhisperX defaults when the active backend can't do the target language); job events via backend/core/job_store.py append_event.
• Design deltas vs pyvideotrans: they let recognized text replace subtitles unconditionally; we keep generated text authoritative for content and use the second pass for timing + QC: re-recognize the dubbed track with halved VAD windows, re-time cues from recognized boundaries, and compute per-segment drift (normalized WER between recognized and target text — scorer exists in omnivoice/eval/wer/). Segments above a drift threshold get flagged in the job events and surfaced in DubTab.jsx as "verify this line" markers feeding the incremental re-dub loop. Stage is opt-out, never fatal: any failure leaves align-stage subtitles in place.
• Tests: pipeline test with an injected mispronounced segment asserting the flag fires; probe judge asserting second-pass SRT stays well-formed and within the existing dubbing duration-ratio gates.
• Effort: M.

Spec 6 — MLX runtime pass

Goal: Apple Silicon speedup via dual-runtime routing, no behavior divergence.

• Files: backend/services/tts_backend.py — adopt voicebox's backend-aware model mapping inside the registry: one engine key resolving to mlx-community/... vs upstream weights by platform probe; harden the probe to import mlx.core catching ImportError, OSError, RuntimeError (PyInstaller bundles); backend/services/asr_backend.py — route dictation + dub ASR through the existing MLXWhisperBackend when the probe passes (today it exists but isn't the default path on Apple Silicon).
• Constraint check: implementation-level only — output behavior, defaults, and UI identical on all platforms (explicitly allowed by the parity rule).
• Tests: probe engines spec on macOS runner asserting MLX route is selected and produces passing DSP judges; regression: CUDA/CPU platforms unaffected (registry resolution unit tests with mocked probes).
• Effort: L across engines; ship Whisper-first (M) since MLXWhisperBackend already exists.

Spec 7 — Crash-isolated ASR

Goal: a native ASR crash becomes a failed job, never a dead backend.

• Files: generalize backend/services/subprocess_backend.py (today TTS-oriented: length-prefixed JSON, GPU slots, op allowlists) with an ASR sidecar subclass — sidecar_script() wrapping the crashy engine; wire as an ASRBackend implementation in asr_backend.py.
• Design choice vs pyvideotrans: they use one-task-per-child process pools (fresh interpreter per job — max isolation, max model-reload cost). Our sidecar is long-lived with a handshake + health check. Hybrid: keep the long-lived sidecar for warm-model latency, add their automatic respawn-on-death semantics — parent detects EOF/broken pipe, marks the in-flight job failed with a decorated error (engine + device, like their broken-pool message), respawns lazily on next request. Their single-JSON progress-file pattern is unnecessary here — we already have a frame protocol with progress ops.
• Tests: kill the sidecar mid-transcription in a pytest (send SIGKILL) — assert job fails with the decorated error, backend stays healthy, next request respawns; smoke-test addition for the respawn path.
• Effort: M.

Spec 8 — Streaming polish kit

Goal: production-grade /ws/tts TTFA + dictation-during-playback.

• Chunker (S): port Patter's sentence_chunker.py (MIT) to backend/services/sentence_chunker.py; use in backend/api/routers/tts_stream.py to synthesize sentence-by-sentence and flush the first clause aggressively (≥ 40 chars on soft punctuation, their seven guards, Italian comma disable). Port their golden parity scenarios as pytest fixtures. Extend terminator/honorific tables toward our language list; share constants with subtitle_segmenter.py where they overlap.
• AEC (M): port Patter's aec.py (MIT) to backend/services/aec.py; in capture_ws.py, feed push_far_end() from the audio /ws/tts ships (resampled 24 kHz → capture rate) and run process_near_end() on mic frames before transcription. Key adaptations (from the deep dive): staleness clock must track playout time, not send time; Geigel rho needs desktop-speaker tuning; one AEC instance per ws session (not thread-safe). Behind a Settings toggle initially ("dictate during playback"), defaulting ON only once probe-verified on all three platforms — until then it's opt-in, per the platform-default rule.
• Tests: chunker parity fixtures; AEC unit tests (echo-only input converges to near-silence; double-talk freezes adaptation); probe dictation spec variant with synthetic far-end playback.
• Effort: M total.

Spec 9 — Quality rails

(a) Docs-drift CI (S). New scheduled workflow alongside .github/workflows/ci.yml (daily cron + dispatch, issues: write): a checked-in canonical inventory (docs/features.yaml — engines, capabilities, platform flags; seed it from the README feature grid + engine tables) diffed against docs stems and the engine registry (list_backends() output). Reuse the self-test pattern from tests/scripts/test_validate_install_docs.py for the new checker. Adopt Patter's rolling-issue automation verbatim: update one docs-drift-labeled issue in place, auto-close on green. Existing scripts/validate-install-docs.py stays as the PR-gating half.

(b) LLM-judge eval tier (M). Port Patter's case.py + runner.py + assertions.py into tests/evals/ with the judge backend swapped to llm_backend.py (local model — keeps local-first). Hard rule preserved: LLM judges never gate CI — they run as a separate non-blocking scheduled job whose report lands as an artifact; deterministic probe judges remain the only gates. First suites: dub translation naturalness (segments from the probe dub fixture) and dictation-refinement quality (Spec 3 outputs). Copy their two hardening details: recompute pass locally from score; tolerant JSON parsing.

Spec 11 — pyvideotrans bridge

Goal: restore the inbound bridge from a 17.9k★ upstream integrator.

• Option A — upstream a REST integration (preferred, S–M): PR to pyvideotrans replacing the Gradio call in their _omnivoice.py with our REST API (clone-TTS endpoint or the OpenAI-compatible surface in backend/api/routers/openai_compat.py). They already ship OpenAI-style TTS channels, so the precedent exists. Friendly-fork etiquette: file their open integration bug first, reference it.
• Option B — Gradio compat shim (fallback, M): a tiny optional Gradio app in our backend exposing a /_clone_fn-compatible signature that proxies to our REST pipeline. Only if upstream declines — it adds a gradio runtime dep for one integration and another surface to keep compatible.
• Either way: add a contract test pinning whatever surface they consume, so the bridge can't silently break again (the engine-compat constraint extended to an external consumer).

Specs 12–13 — voice mixing · dynamic engine lifecycle (sized, not yet designed)

12 (S–M): weighted voice mixing from Kokoro-FastAPI (Apache-2.0) — blend embeddings/style vectors by ratio where the engine exposes them; surface as a "blend" control in CloneDesignTab.jsx. Engine-dependent: start with the preset engines whose voice representations are vectors. 13 (M): Speaches-style (MIT) per-request model lifecycle — idle-unload timers and VRAM-pressure eviction layered on the existing gpu_queue + unload() contract in tts_backend.py/asr_backend.py. Design doc first: interaction with the GPU slot accounting in subprocess_backend.py is the tricky part.

Actions 10 and 14 are practice/positioning items — no code spec needed; 14's content is in positioning moves.

---

Roadmap directions (community discussion #346)

Researched 2026-06-12 (third pass, four research agents + five verification sub-agents). The maintainer's discussion #346 announced a feature roadmap toward full ElevenLabs feature-parity. This section grounds each direction in the landscape: what exists, what's license-clean, what the honest constraints are, and a scope ladder per direction.

Mapping the announcement to this doc — several items are already covered:

Discussion item	Status
Unlimited-length generation	Covered — voicebox chunked TTS (deep dive, port, S)
WisprFlow-like dictation for agentic/code editors	Covered — Spec 3 + positioning move 4
Polished dubbing experience	Covered — Specs 1, 4, 5
Better MLX / Nvidia / AMD / CPU	Spec 6 + new compat-matrix research (§R4)
Polished OpenAPI specs with Scalar	Mostly shipped — Scalar mounted at /docs since #307; remaining work is spec hygiene (§R2)
Agentic voice workflow	New — §R1
Remote GPU · Tailscale · remote API in UI	New — §R2
Ebook/audiobook/stories creator · persona gallery	New — §R3
Better model & env management	New — §R4

R1 — Agentic voice workflow

Runtime landscape (licenses verified against LICENSE files, 2026-06):

Runtime	License	Fit
pipecat (12.8k★, v1.0)	BSD-2	Best fit. A Python library that runs inside our existing FastAPI process (FastAPIWebsocketTransport) — no extra server. Local VAD (Silero) + smart turn detection + barge-in. Its OpenAITTSService takes a base_url and defaults to 24 kHz — our /v1/audio/speech plugs in with configuration, not code
LiveKit Agents (10.9k★)	Apache-2.0	Good, heavier: needs a LiveKit media server alongside. The right choice only if self-hosted SIP at scale becomes the priority (their SIP server is Apache-2.0). Their openai plugin TTS/STT classes accept base_url (verified in source) — OVS works as a provider today
Patter (MIT)	MIT	Parts donor (already deep-dived). If embedded, its opt-out telemetry must be hard-disabled to honor our local-first guarantee
vocode-core	MIT	Avoid as runtime — no commits since Nov 2024
TEN Framework	Apache-2.0 + conditions	Disqualified: LICENSE bans hosting on "End User devices" (fatal for a desktop app) + an Agora non-compete

Telephony honesty. There is no fully-local path to the PSTN — reaching a real phone number requires a carrier (Telnyx ~$0.005–0.007/min, Twilio ~$0.014/min; even self-hosted Asterisk/FreeSWITCH needs a SIP trunk as the gateway). So under our constraints, outbound calling must be an explicit opt-in integration where the user supplies carrier credentials — never a default. Two prerequisite spikes before promising calls: (a) TTFA benchmark of our engines in a streaming pipeline against the ~600 ms p95 voice-to-voice budget; (b) AudioSeal detection survival through the 8 kHz G.711 phone leg (untested anywhere — phone-band downsampling may strip the watermark, and it certainly reduces cloned-voice fidelity).

Persona/community bots. Prior art exists but is assembled hobby-grade (closest: Discord-Local-LLM-VoiceChat-Bot — local Whisper + LM Studio + cloning). Text-persona bots are trivial on our stack (LLM adapter + /v1/audio/speech voice replies). Live voice-channel bots are harder: discord.py has never shipped voice receive (years-open RFC); the working options are Pycord's recording sinks or discord-ext-voice-recv (both MIT, both single-maintainer risk). The conversation loop (VAD/turn-taking) should come from pipecat, not from this prior art.

Safety/regulatory (binding, not optional). FCC ruling FCC 24-17 (Feb 2024): AI/cloned voices are "artificial" under the TCPA — consumer calls require prior express consent ($500–1,500/call private right of action). Texas SB 140 requires AI disclosure within the first 30 seconds of a call. Tennessee's ELVIS Act extends liability to tool providers. EU AI Act Article 50 applies from 2026-08-02: people must be told they're talking to an AI, and generative-audio output must be marked machine-readably — the open-source exemption does not cover Article 50, and our AudioSeal default maps directly onto the marking obligation (a structural advantage no competitor ships). The "my own cloned voice, my own errand" single-call case is a genuine legal gray zone — docs should say so rather than imply it's safe.

Guardrails to build in (concrete): (1) non-removable disclosure preamble on every outbound call — satisfies Texas + FCC direction + EU Art 50(1) in one stroke; (2) consent-locked voice profiles — agentic features require a profile flagged verified-own-voice (recorded consent phrase), exactly the lock voicebox was criticized for lacking; (3) AudioSeal always-on for agentic output, no toggle; (4) destination allowlist + daily call cap, and no bulk-dial API surface ever — architecturally incapable of being robocall infrastructure; (5) local immutable call log (with two-party-consent warning before audio recording); (6) honest jurisdiction notice in docs.

Scope ladder: v1 (S–M, mostly docs): OVS as TTS/STT provider for pipecat/LiveKit — both verified to point at localhost:3900/v1 via base_url today; ship a docs/agentic-voice.md recipe + a pipecat smoke test, fix whatever param mismatches it exposes. Users wire their own agent; we stay a model server. v2 (M–L): built-in Discord persona bot (opt-in by construction — user supplies their own bot token; identical on all platforms): text replies via the LLM adapter + voice replies via /v1/audio/speech, persona attached to a consent-locked profile; live voice-channel mode as a stretch. Mount the MCP server in the same milestone (Spec 2) so external agents can drive OVS voices. v3 (L, only after guardrails 1–5 exist): telephony via opt-in carrier credentials, pipecat embedded with Telnyx/Twilio serializers — disclosure preamble, watermark, allowlist, and call log land in the same PR, not a follow-up.

R2 — Remote GPU, Tailscale, remote API, Scalar

The pattern is settled across Ollama / LM Studio / Open WebUI / Jellyfin: server binds a port, client has a base URL setting, optional bearer key. Nobody comparable ships custom tunneling — LM Studio's remote story (LM Link, June 2026) took a Tailscale partnership on tsnet to do more, which is exactly the bar we should not chase. The existing Tauri app is the thin client; it needs a Backend URL setting + /health handshake, with the local backend supervisor disabled when remote.

Security is the non-negotiable half. The cautionary tale: ~175,000 publicly exposed no-auth Ollama servers found in early-2026 scans, with documented LLMjacking. Our voice-cloning endpoints are more sensitive than chat. The consensus mechanism (LM Studio, vLLM, Speaches): optional bearer key — OMNIVOICE_API_KEY; when set, all non-loopback HTTP+WS requires Authorization: Bearer. Our existing NetworkAccessMiddleware PIN gate has the right ASGI shape and needs a bearer variant. Loopback-only stays the desktop default. Note: Tailscale Serve terminates on-node and forwards from 127.0.0.1 — Serve traffic looks loopback to the PIN gate, so the token must still apply in server mode. Docs say plainly: bearer-over-plain-HTTP is sniffable; use Tailscale (WireGuard) or Serve (TLS) beyond a trusted LAN; never Funnel without the key.

Tailscale depth: ship rung (a) — documentation ("install Tailscale both ends, paste the MagicDNS URL"), plus a Serve recipe — which is all Home Assistant, Open WebUI, and Jellyfin actually ship. Embedding is not viable from Python: tsnet is Go-only; libtailscale's Python binding and tailscale-rs are explicitly experimental/unaudited and not on PyPI. Mention headscale for users wanting a fully open control plane. Tailscale's client core is BSD-3; documenting it imposes nothing on us.

Remote LLM endpoint UI: vLLM's OpenAI-compat server is verified drop-in for our llm_backend.py (today env-only via TRANSLATE_BASE_URL) — the work is Settings fields for base URL + model + optional API key, which Ollama ignores and vLLM/LM Studio require. Watch item: vLLM-Omni now serves TTS first-class with an OpenAI-compatible /v1/audio/speech — including CosyVoice3, an engine we wrap — so "OVS on the GPU box" will eventually compete with "vLLM-Omni on the GPU box"; a future option is consuming a remote vLLM-Omni endpoint as an engine.

Scalar: already shipped (#307 — mounted at /docs, scalar-fastapi is MIT, actively maintained). The remaining "polished spec" work is OpenAPI hygiene Scalar renders but can't create: stable operation_ids, router tags + descriptions, response_model + examples on every endpoint — /v1 and core TTS routes first, since those are what remote users hit. ~1 day for tags/IDs; the response-model long tail is incremental.

Ladder: (1) Backend URL setting + health handshake — S; (2) bearer token incl. WS paths + tests — S–M; (3) "Remote GPU over Tailscale" docs page — S; (4) remote LLM endpoint UI — S–M; (5) OpenAPI hygiene — M incremental. Don't build: custom tunneling/relay, tsnet embedding, Funnel as a promoted path, mTLS/OAuth (overkill vs bearer + WireGuard), a second thin-client binary.

R3 — Audiobook/stories creator + persona gallery

Status (2026-06-13): backend shipped — backend/services/audiobook.py parses a chapter-delimited script (Markdown # H1 chapters + inline [voice:NAME]; [pause …] delegated to the shared parse_pause_markers) into a chapter/span plan, renders each chapter through the active TTS engine (synthesize_chapter + chunked_tts), and muxes a chapterized m4b (FFMETADATA1 chapters). POST /audiobook/plan previews the plan; POST /audiobook runs the synth job streaming SSE progress (ffmpeg-gated). A dedicated Audiobook tab (script editor → plan preview → streamed synth → m4b player/download) ships in the frontend. Deferred: epub/pdf/docx ingest, ACX loudnorm mastering, and crash-resume.

The production bar (verified against ebook2audiobook, audiblez, epub2tts, abogen, Pandrator): broad ingest (epub/mobi/pdf/docx + OCR for image PDFs), chapter detection (TOC-driven for epub — even the 19.2k★ leader's algorithm is thinly documented), chapterized m4b output (FFMETADATA1 chapters + cover via attached_pic — ffmpeg writes both Nero and QuickTime chapter forms), inline tags ([pause:N], [voice:...] — ebook2audiobook's square-bracket dialect is Apache-2.0 and portable), batch with per-file voice mapping, and crash resume (their --session pattern). The mastering bar is the ACX technical spec: RMS −23 to −18 dB, peaks < −3 dB, noise floor < −60 dB RMS, ≥192 kbps CBR MP3, ≤120 min/file, room tone 1–5 s both ends — implementable as two-pass ffmpeg loudnorm + an astats verifier. Framing caveat: ACX itself prohibits AI narration unless authorized — market this as "masters to ACX technical spec," never "Audible-ready."

License traps in the parser stack (verified — this is the §R3 landmine): ebooklib is AGPL-3.0 with no commercial-license option — it poisons our commercial build; parse EPUB with zipfile + lxml instead (EPUB is zip+XHTML). PyMuPDF is AGPL (Artifex sells exceptions) — use pypdf (BSD-3). The PyPI mobi package is GPL — shell out to Calibre's ebook-convert instead (process boundary is safe). OCR via pytesseract/tesseract is Apache-2.0, clean.

What we already have: voicebox's Stories schema (deep-dived above) is the assembly/timeline half. The missing halves are symmetric: ingest (parsing, chapterization, long-form batch with per-chapter resume) and export (m4b + ACX mastering). Ladder: A1 EPUB ingest + TOC chapters + resumable per-chapter TTS queue (M) → A2 chapterized m4b export (S) → A3 ACX mastering pass (S–M) → A4 PDF/txt/docx (M) → A5 inline tags + per-chapter voices (M) → A6 OCR + Calibre shell-out (M) → A7 book→Stories timeline round-trip (L) — the differentiator no surveyed tool has.

Status (2026-06-13): the browse-preview-install surface already exists — VoiceGallery.jsx's Community zone over the omnivoice-gallery git manifest (backend/api/routers/community.py), plus .omnivoice bundle export/import (marketplace.py). The missing piece for the synthetic-only gate was data integrity: imported community presets and bundle round-trips silently dropped kind/vd_states, demoting designed personas to clones. Fixed — community "preset" imports as kind='design' (a "voice" as clone), and bundles now carry kind+vd_states (old bundles import as clone). This makes the "accept only designed/synthetic" gate enforceable. Still to do: the consent attestation + AudioSeal-on-preview gate and the curation workflow below.

Persona gallery — the territory is genuinely unoccupied. The field splits into consent-heavy commercial (ElevenLabs Voice Library: live-read Voice Captcha verification, human review, sharing limited to professional clones), a consent-free gray market (voice-models.com, ~28k RVC models), and read-only single-project registries (piper-voices' single voices.json is the proven local-first pattern). No OSS, consent-aware, browse-preview-install voice gallery exists. The build recipe: piper-voices-style JSON index in a public git repo (checksums + preview URLs, payloads on HF) + Obsidian-style PR curation + our existing VoiceGallery.jsx as the browser (it already has community hooks: useCommunityX, communitySubmitUrl).

Consent gates (legal floor, not nice-to-have): Tennessee ELVIS reaches tools whose "primary purpose" is unauthorized voice likeness — a gallery distributing named-person clones is much closer to that line than a TTS engine; Illinois HB 4875 reaches distribution facilitators; EU Art 50 marking applies from 2026-08-02. Gates: accept only designed/synthetic voices (#317 mapper personas) and self-recorded voices with a recorded consent statement (spoken attestation, not a checkbox — the Consumer Reports critique); AudioSeal watermark mandatory on preview audio (it's already a direct dependency and now fully MIT incl. weights, with a 16-bit payload — enough to carry a persona ID); PR-based human curation; takedown via issue template propagating on index refresh. Honest note: none of this stops a determined fork — the gates protect the project and set norms; the index is the one chokepoint we actually control.

Portable persona format: no standard exists (the one attempt, vox-format, has zero adoption; the de-facto reality is five incompatible engine-native formats). Recommend a minimal .ovsvoice zip: manifest (schema version, engine + design params for deterministic-mapper reproducibility, tags), optional reference audio + transcript, consent.json (creation method + attestation + timestamp), SPDX-style license tag, watermarked preview. Demand signal: voicebox's #138 (export profiles for Piper/Home Assistant) — design the format so a Piper-ONNX export target can be added later. Ladder: G1 .ovsvoice export/import (S–M, standalone value) → G2 community index + Gallery "Community" tab (M) → G3 in-app submission via prefilled GitHub PR/issue — mirrors our bug-reporter pattern, no accounts (M) → G4 similarity-search/ratings/ONNX export (L, later).

R4 — Model & env management, GPU compat matrix

Environment management — what the field converged on (licenses verified): ComfyUI's one-shared-env model is the cautionary tale (conflict UIs, downgrade blacklists, and pip-state-repair files as product features; its 2026 fix is uv + whole-env lockfiles, not isolation). Pinokio (MIT) and StabilityMatrix (AGPL — patterns only, no code) both landed on one venv per app — exactly our sidecar architecture — then clawed disk back at the filesystem layer. LM Studio/Ollama sidestep Python entirely with decoupled, hot-swappable native runtime packs — the strongest pattern, but ours only if we ever ship prebuilt engine binaries. Code-portable references: Pinokio, Ollama, llama.cpp, lms CLI (all MIT), huggingface_hub/hf-xet (Apache-2.0), uv (MIT/Apache). Not portable: StabilityMatrix (AGPL), ComfyUI + Manager + comfy-cli (all GPL-3.0).

The torch-duplication math (measured 2026-06): uv's global cache dedupes via link mode — default clone (CoW) on macOS and Linux, hardlink on Windows; same wheel across N venvs ≈ one copy on disk, iff cache and venvs share a filesystem. But dedup is per-identical-wheel: our IndexTTS2 sidecar (torch 2.6.x) vs parent (torch 2.8.0) shares nothing — the Windows cu128 torch wheel alone is ~3.2 GiB (measured), Linux ~0.83 GiB + multi-GB nvidia-* deps. Partial consolation on Linux: nvidia-* packages dedupe independently wherever pinned versions coincide across torch versions. Levers: keep UV_CACHE_DIR + sidecar venvs on one filesystem; consider pinning UV_LINK_MODE=hardlink on Linux (reflink degrades on ext4); and treat "align the sidecar's torch pin with the parent whenever the engine permits" as the single biggest disk decision. Watch item: PyTorch wheel variants (shipped experimental in 2.8, NVIDIA+Astral collaboration) will eventually make uv install torch auto-select the right CUDA build; uv already ships --torch-backend=auto.

The compat matrix is two-dimensional — (torch version, CUDA wheel variant) → supported sm_XX set, published in pytorch RELEASE.md: Blackwell sm_120 needs 2.7.0 + cu128 or later; from 2.8 the cu128+ wheels dropped Maxwell/Pascal (Turing sm_75 is the floor; Pascal users must pin cu126 variants). Driver minimums: CUDA 12.x wheels ≥ 525, 13.x ≥ 580. Both failure directions ("GPU too new" sm_120-on-cu126 and "GPU too old" sm_61-on-cu128) throw the same lazy no kernel image error after cuda.is_available() returns True — which is why preflight must check capability, not availability. The documented antipattern is Ollama's silent CPU fallback (their own #14258); voicebox's silent-fallback bootstrap is our other autopsy. Build: detect (capability via torch, driver via NVML) → gate engine installs with a specific message ("this engine's cu128 build needs Turing+; you have Pascal — installing the cu126 build instead") → loud persistent CPU-fallback banner, never silent. We already have probes to build on (engine_env.py compute-capability check — today it only gates torch.compile; hardware_probe.py).

Model management — the HF cache is the blessed single source of truth. The hub cache layout is now a language-agnostic spec adopted by llama.cpp among others; blobs are content-addressed (LFS SHA-256 = filename, so integrity is re-checkable offline), and the v1.x CLI ships exactly the manager primitives a UI needs: hf cache ls --filter "accessed>30d", hf cache rm, hf cache prune, hf cache verify. Gotchas verified: Windows without Developer Mode falls back to copy-per-snapshot (degraded dedup); concurrent downloads are lock-protected (.locks/) but deletion is not — a delete UI over a shared cache must handle delete-vs-reader races (fine on Linux fd semantics, breaks on Windows); env vars are read at import time, so a Settings-controlled cache path needs a restart. Offline/restricted: HF_HUB_OFFLINE, HF_HUB_ETAG_TIMEOUT (falls back to cache on timeout), HF_ENDPOINT for mirrors — hf-mirror.com is community-run, not HF-official, and its compatibility with the new Xet/CAS download path is untested (escape hatch: HF_HUB_DISABLE_XET=1). Also: hf_transfer is now fully deprecated (Xet is the default transfer path) — consistent with our existing stack guidance.

Ladder: (a) uv link-mode + shared-cache audit, document the dedupe behavior + sidecar pin-alignment policy — S; (b) in-app preflight compat gate (capability + driver → engine × wheel-variant table) with specific errors + loud CPU banner — M; (c) model manager UI over scan_cache_dir() (per-model disk usage, evict, re-verify, mirror setting) — M; (d) LM-Studio-style decoupled runtime packs — L, not recommended now: our sidecar architecture already decouples engines; revisit only if we ship prebuilt binaries.

Consolidated new actions

#	Action	Mode	Effort	First rung
15	Agentic v1: provider recipe + pipecat smoke test against :3900/v1	Docs + test	S–M	§R1 v1
16	Remote backend: URL setting + bearer token + Tailscale docs page	Build	M total	§R2 rungs 1–3
17	Audiobook v1: EPUB ingest → chapterized m4b → ACX mastering	Build (+ port Apache-2.0 pieces)	M+S+S–M	§R3 A1–A3
18	.ovsvoice portable persona export/import	Build	S–M	§R3 G1
19	Engine preflight compat gate + loud CPU-fallback banner	Build	M	§R4 (b)
20	Model manager UI over the HF cache primitives	Build	M	§R4 (c)
21	OpenAPI hygiene pass under the shipped Scalar UI	Build	M incremental	§R2 rung 5
22	Consent-locked voice profiles (prerequisite for §R1 v2/v3 and §R3 G2+)	Build	M	§R1 guardrail 2

---

User sentiment & market positioning

Researched 2026-06-11. Issue volumes are hand-clustered from title analysis (both trackers use almost no labels): the 250 most recent open voicebox issues, and all 32 open + 80 recent closed pyvideotrans issues. Reddit data is partly secondhand (reddit.com blocks direct fetching); flagged where so.

voicebox: what its users hit (~377 open / ~156 closed — maintainer drowning)

Complaint theme	Volume	Representative	For us
CUDA/GPU bring-up failures (sm_120 "no kernel image", Pascal unsupported, AMD/Intel ignored, 2.4 GB CUDA re-downloads)	~45–50 of 250; 4 of their top-10 most-commented ever	#417, #594, #728, #676	Warning + opportunity. Endemic local-AI tax we share — but their Windows installers shipped with CUDA silently broken for months. GPU auto-detect with explicit per-arch errors is a real differentiator if it holds on the edges
Model download / offline failures (cached models phoning home, infinite offline retry, no mirror option)	~25 of 250	#557, #434, #546	Opportunity. Maps exactly to our uv-mirror + HF-token capabilities. A "local-first" app that breaks offline is a betrayed promise users notice loudly
Startup crashes / white screens / no Linux binary at all	~30 of 250	#513, #617, #606 (regression of an earlier fix), #682	Opportunity. Regressions recur because fixes ship fast without cross-platform gates — our parity rule + 3-platform smoke matrix is precisely this gap
Generation quality bugs (30 s transcription cutoff, reference audio leaking into output, refinement silently translating to English)	~25 of 250	#604, #609, #603	Warning. Engine-level artifacts we inherit too; the fixable subset (silent language handling, truncation) are pipeline bugs — testable
Dictation/capture friction (Windows keyboard hooks, double auto-paste)	~10 of 250	#687, #697	Warning. We shipped the same class of fix (#287/#299). OS-hook dictation is a permanent treadmill on all three platforms
Docs rage — their single most-reacted issue ever is a failed first run blamed on missing docs	top-reacted (17 reactions)	#108; #185 (32 comments, top open: fine-tune instructions)	Opportunity. This is literally our core value. The community reply in #108 — "this is open-source software, not free support" — is the failure mode we exist to avoid

Praise (consistent across coverage): cloning quality ("near-perfect" from 3–5 s; one reviewer scored it above ElevenLabs Multilingual v2 on cloning accuracy); privacy + zero cost as the hook of every viral post; the Stories timeline editor and MCP agent-voice as "genuinely innovative"; out-of-box Metal acceleration; maintainer responsiveness — even as the backlog grows. Notably its 29.7k★ came from X/Threads/LinkedIn/Reddit virality; it barely registered on HN (1 point).

Abandonment: mostly switch-backs to ElevenLabs on Windows — the substack reviewer's verdict: "Windows users should wait weeks for GPU fixes. Low-volume creators should stick with ElevenLabs' $5/month simplicity." Plus churn-risk from unstable main (#648) and a reputational drag: TechTimes covered voicebox having "no consent lock" amid voice-fraud concerns — an angle where our AudioSeal default is the counter-story.

Most-engaged requests: fine-tune instructions (top open, 32 comments); export voice profiles to ONNX for Piper/Home Assistant (#138 — a self-hosting crowd signal); SenseVoice/FunASR STT requested five separate times in one week; AMD DirectML; Linux support.

pyvideotrans: what its users hit (32 open / ~887 closed — aggressive solo triage)

Complaint theme	Volume	Representative	For us
Pipeline hangs + faster-whisper GPU teardown crashes (long-standing per users)	recurring across versions	#1129, #1118	Warning. Upstream faster-whisper lifecycle bugs — we run the same stack. User-found mitigations (pre-segment audio, CPU fallback for long audio) are worth implementing as automatic fallbacks; Spec 7 contains the blast radius
Subtitle/audio sync drift, silence-removal eating final words, merged batch translations	~10 open + the most-reacted closed bugs	#923 (22 comments), #1012	Warning (endemic to dubbing). Speech-rate mismatch is the hardest unsolved problem in the category — anyone evaluating our dub pipeline judges us on exactly this. Specs 1 + 5 are the answer
External TTS engine integration breakage (GPT-SoVITS, F5-TTS, index-tts break at the API seam)	~10 open	#636 (26 comments), #954	Opportunity. They delegate TTS to a zoo of self-hosted side-servers; every seam is a support ticket. Our bundled-engine model removes this entire class — a concrete pitch. (Their broken _omnivoice.py is this same theme pointed at us — Spec 11)
Install failures, especially macOS (source-only; Windows gets a praised .exe)	~6 recent cluster; all-time #2 most-commented issue is literally "Installation tutorial" (#193, 67 comments)	#950, #952	Opportunity. macOS/Linux users are second-class there; a signed mac installer with a working first run is a direct wedge
CUDA errors (GPU fails, silently falls back to CPU)	steady trickle; #287 22 comments	#177, #980	Warning — though notably smaller than voicebox's, because they treat CPU as the default path and GPU as opt-in
LLM translation plumbing (thinking-tags leaking into subtitles, stripped punctuation, merged lines)	~8	#921, #979	Opportunity (partial). LLM-output sanitization is cheap, testable hygiene that visibly differentiates output quality
Their VoiceStudio integration is reported broken by users	open	#1124	Confirms the Spec 11 finding from their side of the bridge

Praise: the packaged Windows .exe needing zero Python setup; completely free with no login/registration/gates; breadth of integrations; responsive maintainer. The Aug 2024 HN thread (182 points) praised the democratization angle — dubbing for material nobody would pay a human to dub. Abandonment: toward subtitles-over-dubbing entirely, Yandex Browser / YouTube auto-dub for casual use, paid dubbing (ElevenLabs, DeepDub) when emotional fidelity matters, and VideoLingo within the OSS niche for subtitle quality. Most-engaged requests: model-chasing (SenseVoice — same ask hit voicebox 5× the same month; index-TTS v2; Fish Audio; Qwen3-TTS), automatic per-speaker role assignment, emotion transfer into dubs, a manual subtitle-proofread checkpoint before merge, per-line audio export.

Comparative read

• voicebox↔pyvideotrans comparisons are essentially absent — they own different frames (English-social "ElevenLabs alternative" vs Chinese-ecosystem "video translation pipeline"). We are unusual in straddling both, which is a positioning asset nobody currently contests.
• vs ElevenLabs, what tips the decision: toward local — cost at volume and privacy, every time; back toward cloud — (1) first-run failure (especially Windows GPU), (2) raw quality ceiling / emotional fidelity, (3) "just works" simplicity for low-volume users. Notably, quality is no longer the automatic cloud win (voicebox cloning reviews beat ElevenLabs Multilingual v2) — the deciding factor has shifted to reliability of install and GPU bring-up, i.e. exactly our stated core value.
• The category wishlist (duplicated across both trackers): SenseVoice/FunASR ASR, index-TTS v2, Fish Audio, per-speaker dubbing, emotion control, fine-tuning instructions.

ElevenLabs pricing pressure (verified on elevenlabs.io/pricing, 2026-06-11)

Free $0 / 10k credits (no commercial license, no cloning) · Starter $6/mo / instant cloning · Creator $22/mo / professional cloning · Pro $99 · Scale $299 (3 pro clones) · Business $990 (10 pro clones). The complaints that push users local:

• Dubbing multiplies cost per target language — a 10-min video into 3 languages bills as 30 minutes; Creator includes ~50 dubbing minutes with $0.60/min overage (their help article). This is exactly the multi-language batch workload where local-and-free is most compelling — and our 50-video batch users' workload.
• Editing a dub costs credits — regenerating a clip bills each time; the credit rebate covers roughly one full re-dub (dubbing studio docs). They built per-segment regeneration and had to bolt a rebate scheme onto it — direct evidence the incremental-re-dub pain is real and monetized against.
• Cloning is paywalled at every tier boundary; commercial use paywalled on Free.
• Credits-as-abstraction generates its own churn-intent search ecosystem (third-party "what do credits actually cost" explainers).

Honest verdicts on our unique five

Differentiator	Verdict	Evidence
Incremental re-dub	Strong lever, unmarketed	ElevenLabs monetizes against this exact pain (rebate scheme). Nobody searches the term — show it (demo GIF), don't name it
646 languages	Strong reach/press lever; niche for retention	It's the hook in all existing coverage of us; Chatterbox's English-only limit draws recurring complaints. Caveat: most users need 1–3 languages, and no third party has verified long-tail quality — overselling invites "language #412 sounds terrible" backlash
AudioSeal watermarking	Nobody-asked (users) / press + compliance asset	Zero end-user search demand (hobbyists prefer unwatermarked); but reviewers spontaneously praise it, roundups list missing watermarking as an open-source limitation, 2026 disclosure regulation makes it a compliance story, and voicebox's "no consent lock" press is the counter-example. Frame as the commercial-use objection-killer; don't lead with it
Cross-platform dictation default	Strong lever, especially Linux	The WisprFlow-alternative market is crowded on macOS, thin on Windows, and served only by single-purpose tools on Linux (active 2026 development = live demand). Nobody offers dictation+cloning+dubbing in one cross-platform app. Caveat: dictation searchers want a small focused app — message it as "already on your machine", not as a lightweight utility it isn't
3-step translation chain	Niche; users value the outcome, never the mechanism	Real evidence that LLM translation quality matters (pyvideotrans documents it beats Google/DeepL; context-aware modes sell subtitle tools). Nobody searches "3-step chain" — market as "translations that don't sound like Google Translate" with a before/after; bury the architecture in docs. Note VideoLingo ships the same idea — it's a parity feature inside the niche, a differentiator outside it

Our own footprint (2026-06-11)

• Repo: 6,808★ / 1,045 forks, created 2026-04-09 — strong two-month trajectory.
• Press: MarkTechPost (accurate, positive) and an uncritical blog endorsement; a viral X post framing us as killing "$700/year in ElevenLabs and HeyGen subscriptions" — note it positions dubbing as the hero feature. No independent quality review of the 646-language claim exists yet — our most exposed flank: expectations are being set high with no third-party validation behind them.
• Hacker News: effectively absent. No submission with traction; every comparable tool got its bump there. An unclaimed opportunity.
• Name collision (the big misconception risk): three entities share "VoiceStudio" — (1) the k2-fsa VoiceStudio model (our default engine; their community-projects page lists us, underselling us as "desktop application for voice generation"); (2) omnivoice.app, an unrelated commercial cloud product; (3) us. Most "VoiceStudio" YouTube traffic and the pyvideotrans VoiceStudio docs page are about the model — search demand is being split three ways and both competitor trackers contain "support VoiceStudio" requests that mean the model, not us.

Positioning moves

1. "vs ElevenLabs dubbing cost" comparison page anchored on the multiplier math (10 min × 3 languages = 30 billed minutes; $0.60/min overage; pay-to-edit), with one table: "20-min video → 3 languages → fix 5 lines → re-export" priced on Creator vs $0 local. Natural home for the incremental-re-dub demo (30 s GIF: edit one line → only that segment regenerates). Targets the highest-intent query cluster ("elevenlabs pricing/dubbing cost") that third parties currently monetize.
2. Show HN, leading with the install story, not the model. HN's documented objections to local TTS are install friction and English-only — our installer + GPU auto-detect + 646 languages answer both. Title shape: "Show HN: Local ElevenLabs alternative — dub, clone, dictate on your own GPU, one installer."
3. Claim the name before the collision hardens: README/FAQ disambiguation ("VoiceStudio Studio, the desktop app built on the k2-fsa VoiceStudio engine — not omnivoice.app"); ask k2-fsa to upgrade our one-line community listing to mention dubbing/dictation; get the pyvideotrans VoiceStudio docs page pointing at Studio as the GUI path (pairs with Spec 11 — arrive with the fixed integration).
4. A discoverable dictation entry point: a docs/landing section "open-source WisprFlow alternative for Mac, Windows, and Linux (built into VoiceStudio Studio)" + PRs to the alternative-list aggregators. Proven, high-conversion query pool; no incumbent covers Linux well.

---

Appendix: engine evaluation — ResembleAI Chatterbox (2026-06-11)

Verdict: integrate later — not now. Full facts verified against the HF cards, GitHub pyproject, and PyPI (0.1.7, 2026-03-26).

• License: clean. MIT on code and all three weight variants (original 0.5B EN, Multilingual 23-lang, Turbo 350M) — compatible with our AGPL + commercial dual-license. The "Resemble uses special weight terms" worry did not materialize.
• What it would add: Turbo's inline paralinguistic tags ([laugh], [cough], [chuckle]) and the single-knob exaggeration expressiveness control — genuinely unique in our roster. Fast English cloning (cloning + speed is a gap; KittenTTS is fast but can't clone). The 23-lang multilingual cloning is not differentiating for us (VoiceStudio 646, VoxCPM2 30 @ 48 kHz).
• Why not now:
  1. chatterbox-tts hard-pins torch==2.6.0 + transformers==5.2.0; we constrain torch==2.8.0 and require transformers>=5.3.0 — unresolvable in the parent venv, forcing a dedicated-venv sidecar (IndexTTS2 pattern, ~800–1000 LOC) that downloads a second multi-GB torch. The disk/download cost is the price, not the code.
  2. resemble-perth (its built-in PerTh watermarker) is a git-URL dependency — unmirrorable on restricted networks, against our bootstrap story. Also untested interaction: PerTh + our AudioSeal = double watermarking.
  3. MPS is buggy upstream (float64 conversion crash on Turbo; placeholder-storage errors); honest Apple-Silicon support means carrying community patches. Mac-ARM users already get Chatterbox today via our MLX-Audio curated list (mlx-community/Chatterbox-TTS-4bit).
• Cheapest path / re-eval triggers: ResembleAI publishes official chatterbox-turbo-ONNX exports. If a 1-day spike proves it runs on plain onnxruntime, Turbo slots into the lightweight supertonic3-style sidecar (~700 LOC, no second torch) and we get the paralinguistic tags cheaply. Also re-evaluate if upstream relaxes the torch/transformers pins or publishes resemble-perth to PyPI.