whisper

Run a local ASR server with automatic backend selection based on your platform:

• macOS Apple Silicon → mlx-whisper (Metal acceleration)
• Linux/CUDA → faster-whisper (CTranslate2)
• HuggingFace → transformers (supports safetensors models, Qwen3-ASR, and known remote-code ASR models such as Cohere Transcribe)
• NVIDIA Parakeet → NeMo (e.g., parakeet-tdt-0.6b-v3)

pip install "agent-cli[faster-whisper]"
agent-cli server whisper

pip install "agent-cli[mlx-whisper]"
agent-cli server whisper --backend mlx

pip install "agent-cli[whisper-transformers]" librosa soundfile sentencepiece protobuf
agent-cli server whisper \
  --backend transformers \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --default-language en

pip install "agent-cli[whisper-transformers]"
agent-cli server whisper \
  --backend transformers \
  --model Qwen/Qwen3-ASR-1.7B-hf

agent-cli install-extras nemo-whisper wyoming
agent-cli server whisper --backend nemo

Features

• OpenAI-compatible API at /v1/audio/transcriptions - drop-in replacement for OpenAI's Whisper API
• Wyoming protocol for Home Assistant voice integration (Wyoming is the standard protocol for local voice services)
• TTL-based memory management - models unload after idle period, freeing RAM/VRAM
• CUDA cache cleanup - the transformers backend releases unused GPU cache after each transcription while keeping model weights loaded
• Multiple models - run different model sizes with independent TTLs
• Background preloading - downloads start at startup without blocking; use --preload to wait
• Multi-platform support - automatically uses the optimal backend for your hardware (auto switches to nemo for Parakeet models)

Usage

agent-cli server whisper [OPTIONS]

Examples

# Run with default large-v3 model (5-minute TTL)
agent-cli server whisper

# Use smaller model with 10-minute TTL
agent-cli server whisper --model small --ttl 600

# Run multiple models (requests can specify which to use)
agent-cli server whisper --model large-v3 --model small --default-model large-v3

# Force CPU mode
agent-cli server whisper --device cpu

# Download model without starting server (requires faster-whisper)
agent-cli server whisper --model large-v3 --download-only

# Run NVIDIA Parakeet via NeMo
agent-cli server whisper --backend nemo

# Keep Parakeet loaded after first request (avoids repeated load cost)
agent-cli server whisper --backend nemo --ttl 0

# Run a specific Parakeet model instead of the NeMo default
agent-cli server whisper --backend nemo --model parakeet-tdt-0.6b-v3

# Preload model at startup and wait until ready
agent-cli server whisper --preload

# Run Cohere Transcribe through the transformers backend
agent-cli server whisper \
  --backend transformers \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --default-language en

# Run Qwen3-ASR through its native transformers integration
agent-cli server whisper \
  --backend transformers \
  --model Qwen/Qwen3-ASR-1.7B-hf

Options

Options

General Options

API Endpoints

Once running, the server exposes:

Using the API

curl Example

# Transcribe an audio file
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1"

# With language hint and verbose output
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1" \
  -F "language=en" \
  -F "response_format=verbose_json"

# Get SRT subtitles
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1" \
  -F "response_format=srt"

Python Example (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:10301/v1", api_key="not-needed")

# Transcribe audio
with open("recording.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
    )
print(transcript.text)

WebSocket Streaming Protocol

The /v1/audio/transcriptions/stream endpoint provides real-time streaming transcription.

Protocol

1. Connect to ws://localhost:10301/v1/audio/transcriptions/stream?model=whisper-1
2. Send binary audio chunks (16kHz, 16-bit, mono PCM)
3. Send EOS (3 bytes: 0x45 0x4F 0x53) to signal end of audio
4. Receive JSON response with transcription

Message Format

Server response:

{
  "type": "final",
  "text": "transcribed text here",
  "is_final": true,
  "language": "en",
  "duration": 3.5,
  "segments": [...]
}

Error response:

{"type": "error", "message": "error description"}

Model Selection Guide

Troubleshooting

Using with agent-cli Commands

The Whisper server is designed to work seamlessly with other agent-cli commands. See Configuration: Local Whisper Server for setup instructions.

Installation

Requires server deps and a backend:

# faster-whisper backend (default on CUDA/CPU)
pip install "agent-cli[faster-whisper]"
# or
uv sync --extra faster-whisper

macOS Apple Silicon

For optimal performance on M1/M2/M3/M4 Macs, install mlx-whisper:

pip install "agent-cli[mlx-whisper]"

The server will automatically detect and use the MLX backend when available.

For NeMo/Parakeet models on Apple Silicon, --device auto uses PyTorch MPS when available. You can also request it explicitly:

agent-cli server whisper --backend nemo --model parakeet-unified-en-0.6b --device mps

HuggingFace Transformers

For loading models in safetensors format (instead of CTranslate2 .bin files):

pip install "agent-cli[whisper-transformers]"
agent-cli server whisper --backend transformers

This uses HuggingFace's transformers library, which supports loading .safetensors models directly from the Hub. Qwen3-ASR requires transformers>=5.13.0 and supports automatic language detection and transcription prompts.

Qwen3-ASR

pip install "agent-cli[whisper-transformers]"
agent-cli server whisper \
  --backend transformers \
  --model Qwen/Qwen3-ASR-1.7B-hf

Qwen3-ASR supports transcription, automatic language detection, and context or hotwords through the OpenAI-compatible prompt field. Translation and timestamped subtitle output are not currently supported.

The default output budget is 4096 tokens, which accommodates long recordings. For unusually long or dense speech, increase it with --max-new-tokens. If Qwen3-ASR exhausts the configured budget, the request fails with a clear error instead of returning a silently truncated transcript.

To install it as the standard whisper background daemon:

agent-cli daemon install whisper -- \
  --backend transformers \
  --model Qwen/Qwen3-ASR-1.7B-hf

For Cohere Transcribe specifically:

pip install "agent-cli[whisper-transformers]" librosa soundfile sentencepiece protobuf
agent-cli server whisper \
  --backend transformers \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --default-language en

Notes for Cohere Transcribe:

• The HuggingFace repo is gated; you must accept the model terms on Hugging Face first.
• Set --default-language unless every client request will pass language.
• translate, srt, and vtt are not available for this model because it does not return timestamped segments or translation output.
• Use --trust-remote-code only for additional custom Hugging Face models that require repository Python code and are not recognized automatically.

NVIDIA NeMo (Parakeet)

The NeMo backend runs NVIDIA Parakeet models behind the same OpenAI-compatible HTTP API and Wyoming protocol as the other Whisper backends. With --backend nemo, the default model is parakeet-unified-en-0.6b.

Install the runtime dependencies:

agent-cli install-extras nemo-whisper wyoming

Start the server:

agent-cli server whisper --backend nemo

Use it from agent-cli through either protocol:

# OpenAI-compatible HTTP API
ag transcribe \
  --asr-provider openai \
  --asr-openai-base-url http://localhost:10301/v1

# Wyoming protocol
ag transcribe \
  --asr-provider wyoming \
  --asr-wyoming-ip localhost \
  --asr-wyoming-port 10300

Run it as a daemon with persistent arguments:

agent-cli daemon install whisper -y -- \
  --backend nemo \
  --model parakeet-unified-en-0.6b \
  --ttl 0

Notes for NeMo/Parakeet:

• --backend nemo defaults to parakeet-unified-en-0.6b.
• --backend auto switches to NeMo when any requested model name is a Parakeet model.
• NeMo uses CUDA when available and CPU otherwise; it does not use MLX/MPS on Apple Silicon.
• First request can be slow because the model loads lazily. Use --preload to load at startup, or --ttl 0 to keep it loaded after the first request.
• REST uploads can be WAV, MP3, M4A, FLAC, OGG, AAC, or WebM when ffmpeg is available.
• NeMo models are currently transcription-only in this server. Requests to /v1/audio/translations return 400.
• Segment formats (verbose_json, srt, vtt) require timestamp output from the selected NeMo model.

Python 3.14 note:

On Python 3.14, the plain published extra path does not apply agent-cli's NeMo override. The agent-cli install-extras nemo-whisper path is uv-aware and can apply agent-cli's temporary pinned NeMo Git install plus override file. Prefer it over plain pip install "agent-cli[nemo-whisper]" on Python 3.14.

Docker

Pre-built images are available from GitHub Container Registry:

# Run with GPU support
docker run -p 10300:10300 -p 10301:10301 --gpus all ghcr.io/basnijholt/agent-cli-whisper:latest-cuda

# Run CPU-only
docker run -p 10300:10300 -p 10301:10301 ghcr.io/basnijholt/agent-cli-whisper:latest-cpu

Or build from source using the whisper.Dockerfile:

# Build and run with GPU support
docker build -f docker/whisper.Dockerfile --target cuda -t agent-cli-whisper:cuda .
docker run -p 10300:10300 -p 10301:10301 --gpus all agent-cli-whisper:cuda

# Build and run CPU-only
docker build -f docker/whisper.Dockerfile --target cpu -t agent-cli-whisper:cpu .
docker run -p 10300:10300 -p 10301:10301 agent-cli-whisper:cpu

Or use Docker Compose:

# With GPU
docker compose -f docker/docker-compose.whisper.yml --profile cuda up

# CPU only
docker compose -f docker/docker-compose.whisper.yml --profile cpu up

Configure via environment variables: