Local AI Setup Guide (Wizard-Only)

July 6, 2026 · View on GitHub

Last validated: 2026-05-05

This guide configures GuideAnts for fully local AI using the Setup Wizard only. If you only need Python sandbox/script execution and plan to use cloud/provider AI for model calls, use the explicit --backend slim stack instead of this local model setup.

For GPU acceleration without CUDA 13 or ROCm, start the stack with --backend vulkan (see docker/guideants-ai-vulkan.md). Vulkan GPU-accelerates llama and image generation; ASR, TTS, and embeddings still run on CPU inside the image.

Prerequisites

GuideAnts is running at http://localhost:5107.
Local runtime containers are running.
A Hugging Face token is already stored.

Target Local Configuration

After completion, these local providers and values should be active:

Speech Transcription: SpeechTranscription.LocalAsr.Http, TimeoutSeconds=300
Image Generation: ImageGeneration.LocalSd.Http, TimeoutSeconds=900, LocalOutputFormat=png
Speech Synthesis: SpeechSynthesis.LocalTts.Http, TimeoutSeconds=300
Document Intelligence: DocumentIntelligence.LocalDocling.Http, TimeoutSeconds=600, MaxConcurrentConversions=1, AsyncStatusPollIntervalMs=2000
Embeddings: Embeddings.LocalEmb.Http, TimeoutSeconds=300, LocalMinIntervalMs=5000

Deterministic model choices used in this flow:

ASR: Qwen/Qwen3-ASR-0.6B
TTS: catalog model chatterbox with reference voice en_us_cv_001
Embeddings: catalog model qwen3_embedding_0_6b (Qwen3-Embedding-0.6B GGUF)
Image bundle:
- Diffusion: unsloth/FLUX.2-klein-4B-GGUF + flux-2-klein-4b-Q4_K_S.gguf
- VAE: black-forest-labs/FLUX.2-small-decoder + full_encoder_small_decoder.safetensors
- Text encoder: unsloth/Qwen3-4B-GGUF + Qwen3-4B-Q4_K_M.gguf

Step-by-Step Wizard Flow

1. Open Setup Wizard from Home

Wizard start on home page

Open Setup Wizard:

Wizard opened

2. Select Local AI Provider

Pick Local AI and continue.

Local AI selected

3. Connection Details (Prerequisites)

Confirm the HF token is already stored and infrastructure statuses are configured, then continue.

Prerequisites passed

4. Models (Chat)

Ensure at least one local llama-cpp chat model is available, then continue.

Chat model ready

5. Speech Transcription

Confirm provider is fixed to Local ASR HTTP, keep TimeoutSeconds=300, and ensure readiness is Ready.

ASR ready

6. Image Generation

Set TimeoutSeconds=900, keep output format png, install/activate the exact bundle, load engine, and wait for Ready.

Image generation ready

7. Speech Synthesis

Confirm provider is Local TTS HTTP, TimeoutSeconds=300, catalog model chatterbox, reference voice en_us_cv_001, and readiness Ready. Local TTS infers language from the selected voice-pack voice.

TTS ready