Supported Models

November 11, 2025 · View on GitHub

UFO supports a wide variety of LLM models and APIs. You can configure different models for HOST_AGENT, APP_AGENT, BACKUP_AGENT, and EVALUATION_AGENT in the config/ufo/agents.yaml file to optimize for performance, cost, or specific capabilities.

Available Model Integrations

ProviderDocumentationVisual SupportAuthentication
OpenAIOpenAI API✅API Key
Azure OpenAI (AOAI)Azure OpenAI API✅API Key / Azure AD
Google GeminiGemini API✅API Key
Anthropic ClaudeClaude API✅API Key
Qwen (Alibaba)Qwen API✅API Key
DeepSeekDeepSeek API❌API Key
OllamaOllama API⚠️ LimitedLocal
OpenAI OperatorOperator (CUA)✅Azure AD
Custom ModelsCustom APIDependsVaries

Model Selection Guide

By Use Case

For Production Deployments:

  • Primary: OpenAI GPT-4o or Azure OpenAI (enterprise features)
  • Cost-optimized: GPT-4o-mini for APP_AGENT, GPT-4o for HOST_AGENT
  • Privacy-sensitive: Ollama (local models)

For Development & Testing:

  • Fast iteration: Gemini 2.0 Flash (high speed, low cost)
  • Local testing: Ollama with llama2 or similar
  • Budget-friendly: DeepSeek or Qwen models

For Specialized Tasks:

  • Computer control: OpenAI Operator (CUA model)
  • Code generation: DeepSeek-Coder or Claude
  • Long context: Gemini 1.5 Pro (large context window)

By Capability

Vision Support (Screenshot Understanding):

  • ✅ OpenAI GPT-4o, GPT-4-turbo
  • ✅ Azure OpenAI (vision-enabled deployments)
  • ✅ Google Gemini (all 1.5+ models)
  • ✅ Claude 3+ (all variants)
  • ✅ Qwen-VL models
  • ⚠️ Ollama (llava models only)
  • ❌ DeepSeek (text-only)

JSON Schema Support:

  • ✅ OpenAI / Azure OpenAI
  • ✅ Google Gemini
  • ⚠️ Limited: Claude, Qwen, Ollama

Configuration Architecture

Each model is implemented as a separate class in the ufo/llm directory, inheriting from the BaseService class in ufo/llm/base.py. All models implement the chat_completion method to maintain a consistent interface.

Key Configuration Files:

  • config/ufo/agents.yaml: Primary agent configuration (HOST, APP, BACKUP, EVALUATION, OPERATOR)
  • config/ufo/system.yaml: System-wide LLM parameters (MAX_TOKENS, TEMPERATURE, etc.)
  • config/ufo/prices.yaml: Cost tracking for different models

Multi-Provider Setup

You can mix and match providers for different agents to optimize cost and performance:

# Use OpenAI for planning
HOST_AGENT:
  API_TYPE: "openai"
  API_MODEL: "gpt-4o"

# Use Azure OpenAI for execution (cost control)
APP_AGENT:
  API_TYPE: "aoai"
  API_MODEL: "gpt-4o-mini"

# Use Claude for evaluation
EVALUATION_AGENT:
  API_TYPE: "claude"
  API_MODEL: "claude-3-5-sonnet-20241022"

Getting Started

  1. Choose your LLM provider from the table above
  2. Follow the provider-specific documentation to obtain API keys
  3. Configure config/ufo/agents.yaml with your credentials
  4. Refer to the Quick Start Guide to begin

For detailed configuration options: