AI Orchestrator

February 21, 2026 · View on GitHub

Intelligent Multi-Model AI Router with Secure Credential Management

Python 3.10+ License: MIT Security: Hardened

AI Orchestrator automatically routes your queries to the best AI model based on task type. It supports 10 providers including OpenAI, Anthropic Claude, Google (Gemini & Vertex AI), Mistral, Groq, xAI (Grok), Perplexity, DeepSeek, Moonshot (Kimi K2), and MLX (Apple Silicon optimized) - all with secure credential management and production-ready features.

Features

  • Native Mac App: Beautiful ChatGPT-like desktop application
  • Menu Bar App: Quick access from your Mac's menu bar
  • Terminal UI: Gorgeous terminal-based interface
  • Intelligent Routing: Automatically selects the best model based on task classification
  • Multi-Model Chaining: Sequential model execution for complex tasks (e.g., web search → deep analysis)
  • Extended Thinking: Kimi K2 Thinking integration with visible reasoning traces
  • Secure Credentials: API keys stored in OS keychain or encrypted file - NEVER in code
  • High Performance: Async operations, rate limiting, and retry logic with exponential backoff
  • Vertex AI Resilience: Structured retryable 429/resource-exhausted handling with clearer error messages
  • Security Hardened: Input validation, audit logging, no credential leakage
  • VS Code Extension: Full-featured extension with keyboard shortcuts
  • Local Models: Privacy-first option with MLX on Apple Silicon
  • Cost Optimization: Route to cheaper models when appropriate
  • Web Search: Perplexity integration for real-time information
  • Local Agent Mode: Codex-like local agent loop with tools, memory file, and optional browser automation
  • MCP & Skills Interop: Imports MCP servers and skill/context docs from Codex, Claude, and Gemini installations
  • Image Generation: DALL-E integration for creating images
  • Music Generation: Create MIDI and audio files with AI (separate drums/bass/chords tracks, 90s tech-house patterns)
  • Animated Status: Real-time processing indicators across all UIs (validating, routing, generating, etc.)
  • Conversation Export: Export to markdown or JSON from all interfaces (CLI: -o, TUI: Ctrl+E, GUI: Ctrl/Cmd+E)
  • Model Attribution: Every response includes which model generated it for transparency
  • Incognito Mode: Disable history saving while preserving existing context
  • Conversation Compaction: Summarize long conversations to save context tokens (TUI: Ctrl+K, GUI: Ctrl+K, Menu Bar: Compact History)
  • Audit Timestamps: All messages and responses include ISO timestamps for security analysis
  • Security Analyzer: Post-incident detection of prompt injection, jailbreaks, and response leakage with severity-based logging

Quick Start

Installation

# Clone the repository
git clone https://github.com/jasonvassallo/ai-orchestrator.git
cd ai-orchestrator

# Sync environment from lockfile (recommended)
# Uses .python-version automatically
uv sync --extra all --extra dev

# Or install specific components with uv extras
uv sync                                 # Minimal (CLI only)
uv sync --extra openai --extra anthropic  # Specific providers
uv sync --extra gui                     # Native Mac app
uv sync --extra tui                     # Terminal UI
uv sync --extra mlx                     # MLX local models (Apple Silicon)
uv sync --extra ui                      # All UI options

# Run commands in the project environment
uv run python -m src.orchestrator "Hello from uv"

Configure Credentials (REQUIRED)

NEVER store API keys in code or config files!

# Interactive configuration (recommended)
uv run python -m src.credentials

# Or set environment variables
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
export MISTRAL_API_KEY="..."
export GROQ_API_KEY="..."
export XAI_API_KEY="..."
export PERPLEXITY_API_KEY="..."
export DEEPSEEK_API_KEY="..."
export MOONSHOT_API_KEY="..."

Vertex AI models use Application Default Credentials (ADC) instead of an API key:

gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT="your-gcp-project"
export GOOGLE_CLOUD_LOCATION="global"  # or a region like us-central1

Basic Usage

# CLI usage
uv run python -m src.orchestrator "Explain quantum computing"

# Codex-like local agent mode (tools + memory + MCP/skills import)
uv run python -m src.orchestrator "Refactor this module and run tests" --agent --local
# or
ai-agent "Search latest release notes and summarize with sources"

# Bootstrap local SearxNG for agent web tools (includes health check)
ai-searx-bootstrap
# Optional standalone health check
ai-searx-health

# With model override
uv run python -m src.orchestrator "Debug this Python code" --model claude-sonnet-4.5

# Prefer local models (MLX)
uv run python -m src.orchestrator "Summarize this text" --local

# Agent mode with browser automation (explicitly dangerous)
uv run python -m src.orchestrator "Open site and capture screenshots" --agent --browser-automation

# Cost optimization mode
uv run python -m src.orchestrator "Write a haiku" --cheap

# Export response to file (markdown or JSON)
uv run python -m src.orchestrator "Summarize AI news" -o response.md
uv run python -m src.orchestrator "Explain quantum computing" -o response.json

# Verbose mode for debugging
uv run python -m src.orchestrator "Complex analysis task" --verbose

Python API

import asyncio
from src.orchestrator import AIOrchestrator

async def main():
    orchestrator = AIOrchestrator(
        prefer_local=False,
        cost_optimize=False,
        verbose=True
    )

    # Simple query
    response = await orchestrator.query(
        "Write a Python function to sort a list"
    )
    print(response.content)

    # With system prompt
    response = await orchestrator.query(
        "Review this code for bugs",
        system_prompt="You are an expert code reviewer focusing on security.",
        model_override="claude-opus-4.6"
    )
    print(response.content)

    # Multi-model comparison
    responses = await orchestrator.multi_model_query(
        "Explain machine learning",
        models=["gpt-4o", "claude-sonnet-4.5", "gemini-2.0-flash"]
    )
    for model, resp in responses.items():
        print(f"\n{model}:\n{resp.content[:200]}...")

asyncio.run(main())

Utilities

Helper scripts for Gemini CLI troubleshooting live in scripts/:

  • scripts/gemini-retry.sh: Wraps gemini to retry 429/resource-exhausted responses with backoff.
  • scripts/gemini-429-diagnose.sh: Parses Gemini debug logs to classify rate-limit vs quota exhaustion signals.

Model Management

Use the model manager to check/download recommended local models and clean cache:

python -m src.manage_models
python -m src.manage_models --yes
python -m src.manage_models --yes --no-clean

It includes MLX Qwen3 Coder 30B (4-bit), MLX Qwen3 VL 30B (4-bit), MLX Llama 3.2 11B Vision (4-bit), MLX Ministral 14B Reasoning (6-bit), plus all supported MusicGen variants (small, medium, large, stereo, melody, stereo-melody, and style). If hf-transfer is installed, downloads will use it automatically. For targeted removals, use hf cache rm <repo_id>.

Mac Applications

AI Orchestrator includes three beautiful interface options:

Native Mac App (ChatGPT-like)

A full-featured desktop application with:

  • Conversation history sidebar
  • Model selector dropdown
  • Feature toggles (Think, Web, Research, Image, Music)
  • Streaming responses
  • Markdown rendering with syntax highlighting
# Run the GUI app
ai-app
# or
uv run python -m src.gui.app

Quick access from your Mac's menu bar:

  • Always accessible
  • Quick query popup
  • Model switching
  • Launch other apps
# Run the menu bar app
ai-menubar
# or
uv run python -m src.menubar.app

Terminal UI

Beautiful terminal-based interface:

  • Modern, stylish design
  • Keyboard-driven
  • Works over SSH
  • Runtime settings panel (Ctrl+R) to inspect effective agent/session/tool limits live
# Run the terminal UI
ai-chat
# or
uv run python -m src.tui.app

Building the .app Bundle

Create a standalone macOS application:

# Install build dependencies
uv sync --extra dev

# Build the .app bundle
uv run python setup_app.py py2app

# Output: dist/AI Orchestrator.app

Supported Models (25+)

Anthropic (Claude)

ModelBest ForContextStrengths
claude-opus-4.6Complex tasks, coding200KMost intelligent, nuanced writing
claude-sonnet-4.5Everyday coding200KBalanced, fast, great for code
claude-haiku-4.5Simple tasks200KVery fast, cost-effective

OpenAI (GPT)

ModelBest ForContextStrengths
gpt-4oGeneral purpose128KMultimodal, fast
gpt-4o-miniSimple tasks128KCost-effective
o1Math, reasoning200KDeep reasoning
o1-miniCoding, reasoning128KBalanced reasoning

Google (Gemini)

ModelBest ForContextStrengths
gemini-3-proComplex tasks, coding2MNext-gen intelligence, massive context
gemini-3-flashSpeed, multimodal1MNext-gen speed, cost-effective
gemini-2.5-proReasoning, coding2MAdvanced reasoning, stable
gemini-2.5-flashSpeed, production1MFast, stable, efficient
gemini-2.0-flashSpeed, long docs1MMassive context, fast
gemini-1.5-proVery long docs2MLargest context window

Vertex AI (Google Cloud)

ModelBest ForContextStrengths
vertex-gemini-3-proEnterprise, complex tasks1MEnterprise-grade, latest Gemini
vertex-gemini-3-flashEnterprise, speed1MFast enterprise inference
vertex-gemini-2.5-flashEnterprise, speed1MFast enterprise inference

Note:

  • Vertex AI uses Google Cloud ADC instead of API keys. See Configure Credentials.
  • The orchestrator now prefers Vertex for the Gemini 3 Preview models by default:
    • gemini-3-pro → vertex-gemini-3-pro
    • gemini-3-flash → vertex-gemini-3-flash This provides higher limits using your Vertex subscription. You can still explicitly select the Google Gemini endpoints if desired.

Mistral

ModelBest ForContextStrengths
mistral-largeCoding, reasoning128KMultilingual, function calling
codestralCode generation32KSpecialized coding model
mistral-smallSimple tasks32KCost-effective, fast

Groq (Ultra-Fast Inference)

ModelBest ForContextStrengths
groq-llama-3.3-70bGeneral, coding128KUltra-fast, versatile
groq-mixtral-8x7bGeneral use32KVery fast, multilingual

xAI (Grok)

ModelBest ForContextStrengths
grok-2General, creative131KReal-time knowledge
grok-2-visionVision tasks32KMultimodal
ModelBest ForContextStrengths
perplexity-sonar-proResearch, citations200KWeb search, real-time info
perplexity-sonarQuick lookups128KFast web search

DeepSeek (Cost-Effective)

ModelBest ForContextStrengths
deepseek-chatGeneral, coding64KVery cost-effective
deepseek-reasonerMath, reasoning64KDeep reasoning, problem-solving

Moonshot (Kimi K2 - Extended Thinking)

ModelBest ForContextStrengths
kimi-k2-thinkingComplex reasoning, math256KExtended thinking with visible reasoning traces
kimi-k2General, coding256KFast, cost-effective

Local Models (MLX)

ModelProviderBest ForStrengths
mlx-llama-vision-11bMLXApple Silicon, VisionVision, documents, charts, writing, 128K context (4-bit)
mlx-qwen3-coder-30bMLXApple Silicon, Large/Complex CodingAgentic coding, long context, local (4-bit default)
mlx-qwen3-vl-30bMLXApple Silicon, Vision + ChatVision understanding, multimodal reasoning, local (4-bit default)
mlx-ministral-14b-reasoningMLXApple Silicon, Deep ThinkingReasoning, math, STEM, local (6-bit)

Smart Routing Configuration

Enable LLM-based routing for every prompt (uses your routing model subscription):

{
  "defaults": {
    "enableLLMRouting": true,
    "routerAllTasks": true,
    "routingModel": "gemini-3-flash-preview"
  }
}

Routing preferences (subscription + web search):

{
  "defaults": {
    "preferSubscriptionProviders": ["vertex-ai"],
    "preferSubscriptionModels": ["vertex-gemini-3-pro"]
  },
  "models": {
    "gemini-2.0-flash": { "enabled": false }
  },
  "taskRouting": {
    "websearch": ["perplexity-sonar-pro", "perplexity-sonar-reasoning-pro"]
  }
}
  • Auto mode prefers local MLX for simple prompts, Vertex Gemini 3 Flash for standard tasks, Kimi K2 Thinking for advanced math/logic/coding, and Vertex Gemini 3 Pro for advanced long-context/general reasoning.
  • Web search defaults to Perplexity Sonar Pro; when reasoning is required it uses Sonar Reasoning Pro.

API Keys Required

Based on which providers you want to use:

ProviderGet API KeyRequired?
OpenAIplatform.openai.com/api-keysRecommended
Anthropicconsole.anthropic.comRecommended
Googleaistudio.google.comOptional
Vertex AIADC via gcloud auth (see above)Optional
Mistralconsole.mistral.aiOptional
Groqconsole.groq.comOptional
xAIconsole.x.aiOptional
Perplexityperplexity.ai/settings/apiOptional
DeepSeekplatform.deepseek.comOptional
Moonshotplatform.moonshot.aiOptional
MLXN/A (local, requires mlx-lm or mlx-vlm)Optional

You already have: Claude (Anthropic), OpenAI, and Perplexity - these are ready to use!

Security Features

Secure Credential Storage

The credential manager uses a priority-based fallback chain:

  1. System Keychain (Most Secure)

    • macOS: Keychain
    • Windows: Credential Manager
    • Linux: Secret Service (GNOME Keyring, KWallet)

  2. Encrypted File (Secure, Portable)

    • Machine-specific key derivation (PBKDF2)
    • Fernet encryption (AES-128)
    • File permissions restricted to owner

  3. Environment Variables (Fallback)

    • Least secure, but always available
    • Useful for CI/CD and containers

Input Validation

  • Maximum prompt length enforcement (500K chars)
  • Suspicious pattern detection (logged, not blocked)
  • Auto-routing logs store prompt length + hash only (no prompt/response content)
  • Rate limiting to prevent abuse

Audit Trail

All API calls are logged with:

  • Timestamp
  • Model/provider used
  • Token usage
  • Latency
  • Success/failure status
  • No prompt/response content (only hashes for routing diagnostics)

Music Generation

AI Orchestrator includes a powerful music generation module optimized for electronic music production:

Features

  • Separate MIDI Tracks: Generates individual files for drums, bass, and chords
  • Combined MIDI: Full arrangement with all tracks in one file
  • 90s Tech-House Patterns: Authentic patterns with:
    • Syncopated kicks (4-on-the-floor with groove)
    • Offbeat open hi-hats
    • Minor chord progressions with 7ths
    • Default 124-128 BPM range
  • AI Audio Generation: MusicGen integration with selectable models for audio file creation

Usage

In the GUI app, click the Music toggle and configure:

  • Key signature (default: G Minor)
  • BPM (default: 124-128 for tech house)
  • Genre/style
  • Energy level
  • Duration
  • AI Model (all supported MusicGen variants, including stereo/melody/style)

You can also use the built-in CLI:

uv run python -m src.orchestrator \
  "90s tech house groove, 126 BPM, deep bassline" \
  --music \
  --music-model musicgen-small \
  --music-duration 5

MusicGen model keys currently available:

  • musicgen-small
  • musicgen-medium
  • musicgen-large
  • musicgen-stereo-small
  • musicgen-stereo-medium
  • musicgen-stereo-large
  • musicgen-melody
  • musicgen-melody-large
  • musicgen-stereo-melody
  • musicgen-stereo-melody-large
  • musicgen-style

When a selected MusicGen model is missing locally, AI Orchestrator now downloads it on demand and shows download progress before generation starts.

Generated files are saved to ~/Music/AI Orchestrator/ and can be opened in:

  • Logic Pro
  • Ableton Live
  • GarageBand
  • Any DAW that supports MIDI

Dependencies

# For MIDI generation
uv pip install midiutil

# For AI audio generation (optional)
# Recommended in a separate venv; see MUSICGEN.md
uv pip install torch "transformers>=5.1,<6.0" "huggingface-hub>=1.4,<2.0" scipy accelerate

For dedicated MusicGen environment setup and troubleshooting, see MUSICGEN.md.

MLX Smart Cache Tips

  • To keep a single consistent cache location, set:
export HF_HOME="$HOME/Library/Caches/huggingface"
  • The orchestrator automatically scans both ~/.cache/huggingface/hub and ~/Library/Caches/huggingface/hub (and respects HF_HOME) for MLX snapshots that include *.safetensors shards. If found, it enables offline mode to prevent network usage.

Using the Project Environment

Always prefer the project uv-managed environment for running and testing:

uv run python -m src.orchestrator "Hello"

This ensures consistent dependencies (e.g., MLX, Transformers) and avoids mixing with user-level Python.

Configuration

Copy config/config.sample.json to ~/.ai_orchestrator/config.json:

{
  "version": "2.0.0",
  "defaults": {
    "preferLocal": false,
    "localProvider": "mlx",
    "costOptimize": true,
    "enableLLMRouting": true,
    "routerAllTasks": true,
    "routingModel": "gemini-3-flash-preview",
    "preferSubscriptionProviders": ["vertex-ai"],
    "preferSubscriptionModels": ["vertex-gemini-3-pro"]
  },
  "models": {
    "gemini-2.0-flash": { "enabled": false }
  },
  "taskRouting": {
    "general": ["mlx-ministral-14b-reasoning", "vertex-gemini-3-flash", "vertex-gemini-3-pro"],
    "code": ["mlx-qwen3-coder-30b", "kimi-k2-thinking", "vertex-gemini-3-pro"],
    "reasoning": ["kimi-k2-thinking", "vertex-gemini-3-pro"],
    "websearch": ["perplexity-sonar-pro", "perplexity-sonar-reasoning-pro"],
    "long-context": ["vertex-gemini-3-pro"]
  }
}

To force MLX for local routing, set defaults.preferLocal to true, keep defaults.localProvider as mlx, and point taskRouting.local at mlx-llama-vision-11b.

To override the Vertex AI location (defaults to global), add:

{
  "providers": {
    "vertex-ai": {
      "location": "global"
    }
  }
}

VS Code Extension

Installation

cd vscode-extension
npm install
npm run package
# Install the .vsix file in VS Code

For local MLX models in the extension:

  • set ai-orchestrator.pythonProjectPath to the repository root
  • optionally set ai-orchestrator.pythonExecutable (leave empty to auto-detect .venv)

Commands

CommandShortcutDescription
Ask AICtrl+Shift+AOpen query input
Explain CodeCtrl+Shift+EExplain selected code
Improve Code-Suggest improvements
Configure Credentials-Set up API keys
Select Model-Override model selection
Clear History-Clear conversation

Model Catalog Sync

# Regenerate VS Code model catalog from src/orchestrator.py
python scripts/sync_vscode_model_catalog.py

A scheduled GitHub Action (refresh-vscode-model-catalog.yml) runs weekly and opens a PR when the catalog changes.

CLI/TUI/Menu Bar/GUI already read directly from ModelRegistry in src/orchestrator.py, so model updates there are immediate once registry entries change.

Context Menu

Right-click selected code for:

  • "AI Orchestrator: Explain Selected Code"
  • "AI Orchestrator: Improve Selected Code"

Testing

# Install dev dependencies
uv sync --extra dev

# Run tests (project mytest wrapper)
mytest() { uv run pytest -q; }
mytest

# Run with coverage
uv run pytest --cov=src --cov-report=html

# Type checking
uv run mypy src

# Linting + formatting
uv run ruff check src
uv run ruff format --check .

# Security scans
uv run bandit --ini .bandit -r -q
uv run pip-audit

Project Structure

ai-orchestrator/
├── src/
│   ├── __init__.py
│   ├── orchestrator.py    # Main orchestrator logic (50+ models, 11 providers)
│   ├── credentials.py     # Secure credential management
│   ├── storage.py         # Conversation storage (SQLite)
│   ├── music.py           # Music generation (MIDI + MusicGen audio)
│   ├── gui/               # Native Mac GUI app (PySide6)
│   │   ├── app.py         # Main entry point
│   │   ├── main_window.py # Primary window
│   │   ├── chat_widget.py # Chat display
│   │   ├── input_widget.py# Input area with toggles
│   │   ├── sidebar.py     # Conversation history
│   │   └── styles.py      # macOS styling
│   ├── tui/               # Terminal UI (Textual)
│   │   └── app.py         # Terminal interface
│   └── menubar/           # Menu bar app (rumps)
│       └── app.py         # Menu bar interface
├── config/
│   ├── config-schema.json # JSON schema for config
│   └── config.sample.json # Sample configuration
├── tests/
│   ├── test_orchestrator.py
│   └── __init__.py
├── vscode-extension/
│   ├── src/
│   │   └── extension.js   # VS Code extension
│   └── package.json
├── setup_app.py           # py2app build script
├── pyproject.toml         # Python project config
├── LICENSE                # MIT License
└── README.md

Troubleshooting

"API key not configured"

# Check if credentials are set
python -c "from src.credentials import get_api_key; print(get_api_key('openai'))"

# Configure interactively
python -m src.credentials

"Rate limited"

The orchestrator handles rate limiting automatically with exponential backoff. If you consistently hit limits, consider:

  • Using a cheaper model for simple tasks
  • Spreading requests over time
  • Upgrading your API tier

Contributing

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/amazing-feature
  3. Commit changes: git commit -m 'Add amazing feature'
  4. Push to branch: git push origin feature/amazing-feature
  5. Open a Pull Request

License

MIT License - see LICENSE for details.

Acknowledgments

  • Anthropic for Claude
  • OpenAI for GPT models
  • Google for Gemini and Vertex AI
  • Mistral AI for Codestral
  • Groq for fast inference
  • xAI for Grok
  • Perplexity for web search
  • DeepSeek for cost-effective models
  • Moonshot AI for Kimi K2 extended thinking
  • Apple for MLX framework