Models

Otari routes requests to LLM providers through any-llm-sdk. This page covers the model format, supported providers, and capabilities.

Model format

Models are specified as provider:model_name:

openai:gpt-4o
anthropic:claude-sonnet-4-6
mistral:mistral-large-latest
vertexai:gemini-2.0-flash

The provider prefix tells Otari which backend to route to. The model_name is passed directly to that provider's API.

Supported providers

Otari depends on any-llm-sdk[all]. Provider support can change as the SDK evolves.

Use this list as a quick reference for common providers supported by the current Otari build.

Provider	Config key	Example model	Notes
Anthropic	anthropic	anthropic:claude-sonnet-4-6
AWS Bedrock	bedrock	bedrock:anthropic.claude-v2	AWS credentials required
Azure OpenAI	azureopenai	azureopenai:gpt-4o	Requires api_base
Azure Anthropic	azureanthropic	azureanthropic:claude-sonnet-4-6	Requires api_base
Cerebras	cerebras	cerebras:llama3.1-8b
Cohere	cohere	cohere:command-r-plus	Also supports rerank
DashScope	dashscope	dashscope:qwen-turbo	Alibaba Cloud
Databricks	databricks	databricks:dbrx-instruct	Requires api_base
DeepInfra	deepinfra	deepinfra:meta-llama/Llama-3-70b
DeepSeek	deepseek	deepseek:deepseek-chat
Fireworks	fireworks	fireworks:llama-v3-70b
Gemini	gemini	gemini:gemini-2.0-flash
Groq	groq	groq:llama3-70b-8192
HuggingFace	huggingface	huggingface:meta-llama/Llama-3-70b	Pin a backend with :<backend> (see Pinning a HuggingFace inference backend)
Inception	inception	inception:mercury-coder-small
Llama.cpp	llamacpp	llamacpp:default	Local server
Llamafile	llamafile	llamafile:default	Local server
LM Studio	lmstudio	lmstudio:local-model	Local server
MiniMax	minimax	minimax:abab5.5-chat
Mistral	mistral	mistral:mistral-large-latest
Moonshot	moonshot	moonshot:moonshot-v1-8k
Nebius	nebius	nebius:llama-3-70b
Ollama	ollama	ollama:llama3	Local server
OpenAI	openai	openai:gpt-4o
OpenRouter	openrouter	openrouter:openai/gpt-4o
Perplexity	perplexity	perplexity:llama-3-sonar-large
SageMaker	sagemaker	sagemaker:my-endpoint	AWS credentials required
SambaNova	sambanova	sambanova:llama3-70b
Together	together	together:meta-llama/Llama-3-70b
Vertex AI	vertexai	vertexai:gemini-2.0-flash	Requires service account
Vertex AI Anthropic	vertexaianthropic	vertexaianthropic:claude-sonnet-4-6	Requires service account
vLLM	vllm	vllm:my-model	Self-hosted
Voyage	voyage	voyage:voyage-large-2	Embeddings only
WatsonX	watsonx	watsonx:ibm/granite-13b
xAI	xai	xai:grok-2

Capabilities

Not all providers support all endpoints. Here's what each endpoint type requires:

Endpoint	Capability	Example providers
/v1/chat/completions	Chat completion	Most providers
/v1/messages	Anthropic Messages API	Anthropic, Vertex AI Anthropic
/v1/responses	OpenAI Responses API	OpenAI
/v1/embeddings	Text embeddings	OpenAI, Cohere, Voyage, Vertex AI
/v1/moderations	Content moderation	OpenAI
/v1/rerank	Document reranking	Cohere
/v1/images/generations	Image generation	OpenAI, Vertex AI
/v1/audio/transcriptions	Audio transcription	OpenAI
/v1/audio/speech	Text-to-speech	OpenAI
/v1/batches	Batch processing	OpenAI, Anthropic

In deployments connected to otari.ai, the final model/provider choices are resolved by otari.ai routing policy, not by local providers configuration.

Configuring a provider

In config.yml:

providers:
  openai:
    api_key: "sk-..."
    api_base: "https://api.openai.com/v1"  # optional for hosted OpenAI

Or via environment variable:

export OPENAI_API_KEY="sk-..."

Both approaches work. Config file values take precedence over environment variables.

In standalone mode, provider config only tells Otari how to reach the backend. Otari also requires pricing for the model you call by default, unless default_pricing covers it or require_pricing: false is set.

For the full configuration reference, see Configuration.

Named provider instances

The providers map is keyed by instance name. Most of the time that name is also the provider, such as openai or anthropic.

If you want to use multiple backends that share one provider implementation, give one of them a custom name and set provider_type. This is common for self-hosted OpenAI-compatible servers such as vLLM, llama.cpp, or LM Studio:

providers:
  openai:                       # key is a real provider, no provider_type needed
    api_key: ${OPENAI_API_KEY}

  home_lab:                     # custom instance name
    provider_type: openai       # underlying any-llm implementation
    api_base: "https://nathans-mac-studio.example.ts.net/v1"
    api_key: ${HOME_LAB_TOKEN}

Route to a named instance with instance_name:model. For example, home_lab:deepseek-v4-flash uses the home_lab config, but Otari sends the request through the OpenAI provider implementation with that instance's api_base and api_key. openai:gpt-4o still uses the regular openai config.

Pricing and usage are keyed on the instance name, so configure pricing under home_lab:deepseek-v4-flash if you want that model to be priceable. If you do not set provider_type, the key works as before and names the provider directly.

provider_type: openai-compatible and provider_type: openai_compatible are both accepted as aliases for openai.

Named instances are a standalone-mode feature. In hybrid mode, provider credentials come from otari.ai per request, so local named instances are not used.

Declaring models for backends without /v1/models

/v1/models lists an instance's models by calling the backend's model-listing endpoint. When a backend does not expose /v1/models, declare the served model ids so they still appear in the listing:

providers:
  edge_box:
    provider_type: openai
    api_base: "https://edge.example.ts.net/v1"
    api_key: ${EDGE_TOKEN}
    models:
      - llama-3.3-70b
      - qwen3-32b

The declared models are listed as edge_box:<model>. Direct requests work with or without this list; it only affects discovery.

Listing available models

Query Otari to see which models are available:

curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer <your-api-key>"

Provider-specific notes

Pinning a HuggingFace inference backend

HuggingFace Inference Providers is a router: the same model id (for example zai-org/GLM-4.6) can be served by several backends (Together, Novita, and others), and in the default "auto" mode the backend, and therefore the price, is chosen at request time. To route (and price) deterministically, pin a backend with a :<backend> suffix on the model id, which the HuggingFace router honors server side:

huggingface:zai-org/GLM-4.6:together
huggingface:zai-org/GLM-4.6:novita

The pinned-selector grammar is huggingface:<model>:<backend>. Otari splits the provider off the first :, so everything after it (<model>:<backend>) is forwarded as the model id and the :<backend> suffix reaches the router unchanged. The router also accepts policy suffixes such as :cheapest, :fastest, :preferred, and :auto.

This grammar is the contract consumers build against. The otari.ai platform's pricing UI, for instance, offers each priceable backend as a pinned huggingface:<model>:<backend> selector, because a pinned selector resolves to a single backend, which is what makes a HuggingFace model priceable (auto mode cannot be priced from the model id alone).