Configuration

Otari is configured through a YAML file and environment variables.

Config file

The config file is passed at startup with --config:

otari serve --config config.yml

Full example

# Database
database_url: "postgresql://otari:otari@postgres:5432/otari"

# Server
host: "0.0.0.0"
port: 8000

# Auth
master_key: "your-secret-master-key"

# Rate limiting (requests per minute per user, omit to disable)
# rate_limit_rpm: 60

# Prometheus metrics at /metrics
# enable_metrics: true

# Providers
providers:
  openai:
    api_key: "sk-..."
  anthropic:
    api_key: "sk-ant-..."
  mistral:
    api_key: "..."
  vertexai:
    credentials: "/app/service_account.json"
    project: "my-gcp-project"
    location: "us-central1"

# Pricing (USD per million tokens)
pricing:
  openai:gpt-4o:
    input_price_per_million: 2.50
    output_price_per_million: 10.00

Config reference

Field	Type	Default	Description
database_url	string	sqlite:///./otari.db	Database connection URL (SQLite or PostgreSQL)
auto_migrate	bool	true	Run Alembic migrations automatically on startup
db_pool_size	int	10	Persistent DB connections per worker
db_max_overflow	int	20	Extra burst connections above pool size
db_pool_timeout	float	30.0	Seconds to wait for a connection
db_pool_recycle	int	-1	Recycle connections older than N seconds (-1 = disabled)
host	string	0.0.0.0	Server bind host
port	int	8000	Server bind port
master_key	string	none	Master key for management endpoints
rate_limit_rpm	int	none	Max requests per minute per user (none = disabled)
cors_allow_origins	list	[]	Allowed CORS origins (empty = disabled)
providers	dict	{}	Provider credentials (see below)
aliases	dict	{}	Model name aliases (display name to target selector instance:model or provider:model). The alias is what users see in GET /v1/models and in response model fields; pricing, budgets, and usage key on the resolved target. Standalone mode only.
pricing	dict	{}	Model pricing entries
enable_metrics	bool	false	Enable Prometheus /metrics endpoint
enable_docs	bool	true	Enable /docs, /redoc, /openapi.json
bootstrap_api_key	bool	true	Create a first-use API key on startup when none exist
log_writer_strategy	string	"single"	Usage log writing: "single" (inline) or "batch" (background). Prefer "batch" for streaming clients: with "single", a client that disconnects at the SSE [DONE] marker can leave the usage row uncommitted and the budget reservation unreconciled. "batch" queues in memory and flushes on a 1s interval or 100-row batch, so it removes that race but is not crash-durable.
budget_strategy	string	"for_update"	Budget validation: "for_update", "cas", or "disabled"
require_pricing	bool	true	Reject requests for models with no configured pricing (HTTP 402, fail-closed). When false, unpriced models are served and logged without cost. Audio and moderation endpoints are always exempt.
default_pricing	bool	false	When a model has no pricing in the database, fall back to community-maintained defaults from the bundled genai-prices dataset. Off by default (opt-in). Database pricing always wins. See Default pricing.
reject_user_mismatch	bool	true	When true, a non-master key whose request names a user other than its own is rejected (HTTP 403). When false, the client user is still forwarded to the provider but spend is always bound to the key's own user. The master key may always bill an arbitrary user.
stream_missing_usage_policy	string	"estimate"	How to bill a streamed response that completes with no provider usage data: "estimate" (charge the up-front estimate), "fail" (charge estimate and mark errored), or "allow_free" (don't bill).
budget_estimate_default_output_tokens	int	1024	Output-token count assumed when reserving budget for a request with no declared max output; reconciled to actual usage on completion.
model_discovery	bool	true	Auto-discover models for GET /v1/models, from configured providers and from any provider made callable by its native credential environment variable alone (so the catalog matches what routing can reach).
model_cache_ttl_seconds	int	300	TTL for the in-memory model-discovery cache (0 disables caching).
model_discovery_timeout_seconds	float	10.0	Per-provider timeout for a live model-discovery (list_models) call. Bounds how long an unreachable or slow provider can stall discovery before it is treated as failed.
model_discovery_negative_ttl_seconds	float	30.0	How long a failed model-discovery result is remembered before the provider is dialed again, so an unreachable provider is not re-tried on every request (0 disables negative caching).
models_dev_metadata	bool	true	Enrich the dashboard's model detail with metadata (modalities, capabilities, knowledge cutoff) fetched from the public models.dev catalog. Set false to disable the outbound call; the gateway then falls back to the bundled genai-prices data.
models_dev_cache_ttl_seconds	int	86400	TTL in seconds for the cached models.dev catalog (0 disables caching).
files_enabled	bool	true	Enable the /v1/files upload/storage endpoints (standalone mode).
files_backend	string	"local"	Blob backend for uploaded file bytes ("local" filesystem for now).
files_local_dir	string	"./otari-files"	Directory the local files backend writes uploaded bytes to.
files_max_bytes	int	536870912	Maximum size in bytes for a single uploaded file (512 MiB default).
files_retention_hours	int	none	Stop serving files older than N hours (they return 404); none keeps files indefinitely. Stored bytes are not auto-reclaimed.
file_understanding_enabled	bool	true	Normalize file/image content blocks before the provider call (pass through for capable models, extract to text otherwise). When false, blocks are forwarded unchanged.
vision_strategy	string	"describe"	How image blocks are handled for text-only models: "describe" (vision side-call), "ocr" (extract text only), or "off" (drop with a log line).
vision_describe_model	string	none	provider/model used to caption images for text-only targets when vision_strategy="describe". When unset, describe falls back to a logged drop.
vision_describe_max_tokens	int	1024	Cap on the describe model's output tokens per image; bounds cost and latency of the vision side-call.
model_capabilities	dict	{}	Per-model multimodal capability overrides (provider/model to {supports_image, supports_pdf}). Authoritative over any-llm's provider-level flags; needed for text-only local models behind OpenAI-compatible servers.
sandbox_url	string	none	Base URL of the code-execution sandbox backend for otari_code_execution tools. When unset, such requests are rejected with HTTP 400. Also settable via OTARI_SANDBOX_URL.
guardrails_url	string	none	Default input-guardrails service URL, used when a request does not pass its own guardrail url. Also settable via OTARI_GUARDRAILS_URL.
web_search_url	string	none	Base URL of the web-search backend (SearXNG instance or a search adapter) for otari_web_search tools. When unset, such requests are rejected with HTTP 400. docker-compose sets this to the bundled SearXNG container. Also settable via OTARI_WEB_SEARCH_URL.
tools_header	string	none	Override for the purpose-hint preamble header injected ahead of gateway-managed tool hints. When unset, a built-in default header is used. Also settable via OTARI_TOOLS_HEADER.
sandbox_purpose_hint	string	none	Default purpose hint forwarded to the sandbox backend when a tool entry supplies none. Also settable via OTARI_SANDBOX_PURPOSE_HINT.
web_search_purpose_hint	string	none	Default purpose hint for the web-search backend when a tool entry supplies none. Also settable via OTARI_WEB_SEARCH_PURPOSE_HINT.
web_search_engines	string	none	Comma-separated SearXNG engine list for the web-search backend. Also settable via OTARI_WEB_SEARCH_ENGINES.
web_search_max_results	int	none	Default cap on web-search hits (a per-tool max_results still overrides it). Also settable via OTARI_WEB_SEARCH_MAX_RESULTS.
web_search_extract	bool	none	Whether the web-search backend extracts page content in-process (true) or returns snippet-only results (false). When unset, extraction is on. Also settable via OTARI_WEB_SEARCH_EXTRACT.
web_search_allow_private_hosts	bool	false	SSRF gate: allow the web-search backend to fetch private/loopback/reserved hosts. Also settable via OTARI_WEB_SEARCH_ALLOW_PRIVATE_HOSTS.
mcp_allow_loopback	bool	true	SSRF gate: allow MCP server URLs that resolve to loopback (same-host sidecars). Also settable via OTARI_MCP_ALLOW_LOOPBACK.
mcp_allow_private_hosts	bool	false	SSRF gate: allow MCP server URLs that resolve to private/reserved hosts (and accept hostnames that fail to resolve at validation time). Also settable via OTARI_MCP_ALLOW_PRIVATE_HOSTS.
provider_allow_private_hosts	bool	true	SSRF gate: allow a provider api_base that resolves to private/loopback/reserved hosts. On by default (unlike the other gates), because api_base is master-key gated and the home-lab use case depends on private endpoints. Set to false to make provider connection tests and model discovery refuse an internal api_base. This gate covers only those report paths: chat dispatch (which dials the endpoint) and the credential write path (which persists it) are not gated, so it is not a general egress control. Also settable via OTARI_PROVIDER_ALLOW_PRIVATE_HOSTS.
mode	string	none	Operating mode ("standalone" or "hybrid"; the legacy value "platform" means hybrid): when unset, the mode is derived from the presence of OTARI_AI_TOKEN; when set explicitly it is enforced at startup, so "hybrid" without a token and "standalone" with a token both fail as conflicting configuration.
platform	dict	{}	otari.ai integration settings (base_url, timeouts, retries)

Environment variables

The following OTARI_ variables override config file values for their matching fields. For example, OTARI_PORT=9000 overrides port: 8000 in the YAML.

OTARI_ overrides apply to scalar fields (strings, numbers, booleans). List and dict fields (cors_allow_origins, providers, pricing, and the platform block) are not read from individual OTARI_ variables; set them in the YAML file, or supply the whole config through the environment with OTARI_CONFIG_YAML / OTARI_CONFIG_B64 (see Full config via environment). The platform block also has dedicated PLATFORM_* variables (see the otari.ai variables below).

The config file also supports ${ENV_VAR} interpolation:

master_key: "${MY_SECRET_KEY}"

Full config via environment

On PaaS platforms (Railway, Render, Fly.io, Kubernetes) where mounting a config.yml is awkward, you can supply the entire config, including the non-scalar providers and pricing fields, through the environment. This reaches the full schema with no file mount and no custom image.

Variable	Description
OTARI_CONFIG_YAML	The full config as raw YAML, parsed exactly like a config.yml.
OTARI_CONFIG_B64	The same YAML, base64-encoded, for env-var UIs that mangle multiline values.

The YAML supports the same ${ENV_VAR} interpolation as a config file, so you can keep secrets in separate variables:

# value of OTARI_CONFIG_YAML
providers:
  openai:
    api_key: ${OPENAI_API_KEY}
    api_base: https://my-proxy.example/v1
pricing:
  openai:gpt-4o:
    input_price_per_million: 2.5
    output_price_per_million: 10

Precedence, lowest to highest: the config file, then the env-structured config (OTARI_CONFIG_YAML or OTARI_CONFIG_B64, with raw YAML winning if both are set), then scalar OTARI_<FIELD> overrides. Env-structured keys replace the matching top-level keys from the file. Invalid base64, invalid YAML, or a non-mapping top level fails fast at startup with a clear error.

Note the require_pricing interaction: it defaults to true (fail-closed), so the gateway rejects any model without configured pricing. An env-only deploy therefore needs either pricing entries (as above) or OTARI_REQUIRE_PRICING=false to serve unpriced models.

Common variables

Variable	Description
OTARI_MASTER_KEY	Master key for management endpoints. When unset in standalone mode, one is generated on first run and printed to the logs (see Runtime provider management).
OTARI_SECRET_KEY	Fernet key that encrypts provider credentials added through the dashboard. Required to store a provider key in the UI. Generate one with otari gen-secret-key.
OTARI_DATABASE_URL	Database connection URL
OTARI_HOST	Server bind host
OTARI_PORT	Server bind port
OTARI_AUTO_MIGRATE	Auto-run migrations on startup
OTARI_BOOTSTRAP_API_KEY	Create first-use API key

Built-in tools and guardrails variables

These operator-facing settings configure the gateway-managed tools (otari_code_execution, otari_web_search), the MCP tool loop, and input guardrails. Each is validated at startup and, where it maps to a scalar GatewayConfig field, can also be set in the config file (see the config reference above).

Variable	Default	Description
OTARI_SANDBOX_URL	none	Base URL of the code-execution sandbox backend for otari_code_execution tools. Required to use that tool.
OTARI_WEB_SEARCH_URL	none	Base URL of the web-search backend for otari_web_search tools. Required to use that tool.
OTARI_GUARDRAILS_URL	none	Default input-guardrails service URL, used when a request does not pass its own guardrail url.
OTARI_TOOLS_HEADER	built-in	Override for the purpose-hint preamble header injected ahead of gateway-managed tool hints.
OTARI_SANDBOX_PURPOSE_HINT	none	Default purpose hint forwarded to the sandbox backend when a tool entry supplies none.
OTARI_WEB_SEARCH_PURPOSE_HINT	none	Default purpose hint for the web-search backend when a tool entry supplies none.
OTARI_WEB_SEARCH_ENGINES	backend default	Comma-separated SearXNG engine list (e.g. google,bing).
OTARI_WEB_SEARCH_MAX_RESULTS	backend default	Default cap on returned hits (a per-tool max_results still overrides it). Must be >= 1.
OTARI_WEB_SEARCH_EXTRACT	true	0/false disables in-process content extraction (snippet-only mode).
OTARI_WEB_SEARCH_ALLOW_PRIVATE_HOSTS	false	SSRF gate: allow the web-search backend to fetch private/loopback/reserved hosts.
OTARI_MCP_ALLOW_LOOPBACK	true	SSRF gate: allow MCP server URLs that resolve to loopback (same-host sidecars).
OTARI_MCP_ALLOW_PRIVATE_HOSTS	false	SSRF gate: allow MCP server URLs that resolve to private/reserved hosts, and accept hostnames that fail to resolve at validation time.
OTARI_PROVIDER_ALLOW_PRIVATE_HOSTS	true	SSRF gate: allow a provider api_base that resolves to private/loopback/reserved hosts. On by default (unlike the other gates). Set to false to make provider connection tests and model discovery refuse an internal api_base. Does not gate chat dispatch or the credential write path, so it is not a general egress control.

Provider credentials

Provider API keys can be set as environment variables instead of in the config file. These are picked up directly by the underlying SDK.

These credentials are used for standalone deployments. When connected to otari.ai, local provider credentials are not used.

Variable	Provider
OPENAI_API_KEY	OpenAI
ANTHROPIC_API_KEY	Anthropic
MISTRAL_API_KEY	Mistral
GEMINI_API_KEY	Google Gemini

Runtime provider management

Instead of setting providers at launch, you can add them from the admin dashboard after startup (standalone mode only). This lets you launch Otari with almost nothing set and finish configuration in the browser.

• First-run master key. If OTARI_MASTER_KEY is unset, Otari generates a master key on first startup, stores only its hash, and prints the plaintext once to the logs (look for Your master key:). Use it to sign in. An operator-set OTARI_MASTER_KEY always takes precedence and is never generated over.
• Encryption at rest. Provider keys added in the dashboard are stored encrypted with OTARI_SECRET_KEY (a Fernet key). Set it before adding a key; generate one with otari gen-secret-key. Keep it safe and separate from the database: losing it makes every stored provider key undecryptable, and a database dump alone cannot decrypt them. Rotate by prepending a new key (comma- or whitespace-separated); both old and new are tried on decrypt.
• Precedence. Dashboard-stored providers merge over config.yml providers. A stored provider with the same instance name as a config one takes precedence (the shadowing is logged at startup). In the dashboard, config providers are marked config and are read-only; stored providers are marked stored and can be edited, tested, and deleted.
• Scope. Providers that authenticate with an API key (OpenAI, Anthropic, Mistral, Gemini, and OpenAI-compatible backends) are supported. Providers that use ADC/IAM (Vertex AI, Bedrock) remain config-file only.

otari.ai variables

These are only relevant when running connected to otari.ai. See Modes for details.

Variable	Default	Description
OTARI_AI_TOKEN	none	Otari token from otari.ai (enables platform connection)
PLATFORM_RESOLVE_TIMEOUT_MS	5000	Timeout for provider resolution calls
PLATFORM_USAGE_TIMEOUT_MS	5000	Timeout for usage reporting calls
PLATFORM_USAGE_MAX_RETRIES	3	Max retries for transient usage reporting failures
STREAMING_FALLBACK_FIRST_CHUNK_TIMEOUT_MS	2000	Per-attempt timeout waiting for first streamed chunk
STREAMING_FALLBACK_FINAL_ATTEMPT_EXTRA_FIRST_CHUNK_TIMEOUT_MS	0	Extra first-chunk grace for the sole/final attempt, added on top of the per-attempt budget. 0 = no grace (unchanged)

Provider configuration

Each provider entry in the config file supports:

Field	Description
api_key	Provider API key
api_base	Custom API base URL (optional)
client_args	Extra client options: custom_headers, timeout (optional)

Vertex AI

Vertex AI requires additional fields instead of a simple API key:

providers:
  vertexai:
    credentials: "/app/service_account.json"
    project: "my-gcp-project"
    location: "us-central1"

The credentials field points to a Google Cloud service account JSON file.

Pricing

Model pricing is configured per model key (provider:model_name) in USD per million tokens:

pricing:
  openai:gpt-4o:
    input_price_per_million: 2.50
    output_price_per_million: 10.00
  anthropic:claude-sonnet-4-6:
    input_price_per_million: 3.00
    output_price_per_million: 15.00

Config pricing sets initial values. Pricing set via the /v1/pricing API takes precedence.

A pricing entry whose provider is not listed in the providers section is skipped at startup with a warning, not treated as a fatal error: the provider may still be reachable through environment credentials (any-llm reads keys like OPENAI_API_KEY), so a pricing/provider mismatch should not abort the gateway. To have such an entry take effect, add the provider to the providers section.

Cache token pricing

Providers that support prompt caching (OpenAI, Gemini, Anthropic) report cached-token counts that are captured in cache_read_tokens and cache_write_tokens on the usage log. Without a configured cache rate these tokens are still billed, at input_price_per_million (they are ordinary input tokens), just not at a discounted cache rate. To bill them at the provider's cache price instead, add the optional cache_read_price_per_million and cache_write_price_per_million rates (USD per million tokens):

pricing:
  openai:gpt-4o:
    input_price_per_million: 2.50
    output_price_per_million: 10.00
    cache_read_price_per_million: 1.25  # discounted rate for cached input
  anthropic:claude-sonnet-4-6:
    input_price_per_million: 3.00
    output_price_per_million: 15.00
    cache_read_price_per_million: 0.30  # cache-read rate
    cache_write_price_per_million: 3.75  # cache-creation (write) rate
    cache_write_1h_price_per_million: 6.00  # Anthropic 1-hour cache creation

Every physical input token is charged once. The input/prompt count is treated as the grand total that includes cache reads and writes; the uncached remainder is billed at input_price_per_million, and cache tokens are re-priced at their own rate when one is configured. Providers report cache counts two ways, and the gateway normalizes both onto that single model:

• OpenAI / Gemini: cache_read_tokens is already a subset of prompt_tokens, so setting cache_read_price_per_million discounts the cached portion (re-priced at the cache rate instead of the full input_price_per_million) rather than double-counting it. cache_write_tokens is always 0 for these providers.
• Anthropic: cache_read_tokens and cache_write_tokens are reported separately from prompt_tokens, so they are added on top and billed at cache_read_price_per_million / cache_write_price_per_million. cache_write_1h_tokens records the 1-hour subset of cache creation and uses cache_write_1h_price_per_million when set, otherwise the standard cache-write rate. This holds on every turn, including warm-cache reads that create no new cache (cache_write_tokens is 0).

When a cache rate is left unset (null), those cache tokens are not dropped or billed at $0: they remain part of the input total and are billed at input_price_per_million, since they are still real input tokens. Set the cache rate to bill them at the provider's discounted cache price. The same fields are available on the /v1/pricing API (SetPricingRequest and PricingResponse).

Long-context pricing tiers

Some models charge a different rate once a request reaches a context threshold. Set pricing_tiers when that rate applies to the whole request, as it does for the tiered catalog entries Otari imports from genai-prices:

pricing:
  openai:gpt-5:
    input_price_per_million: 1.25
    output_price_per_million: 10.00
    pricing_tiers:
      - min_input_tokens: 272000
        input_price_per_million: 2.50
        output_price_per_million: 15.00

At min_input_tokens and above, an entry replaces only the rates it lists for the entire request. Omitted rates inherit the base price. The dashboard's model detail editor exposes the same controls, and each usage row records the resulting billable meters and rate breakdown for auditability.

Default pricing

Default pricing is off by default. When you enable it (default_pricing: true in config.yml, or OTARI_DEFAULT_PRICING=true) and a model has no price in the database (neither config nor /v1/pricing), Otari falls back to community-maintained default pricing from the genai-prices dataset, which bundles per-million rates for hundreds of models across the major providers. With it on, common models (for example openai:gpt-4o, anthropic:claude-sonnet-4-6) are priced without any configuration, so require_pricing does not reject them. The defaults are bundled with the installed package; no network access is used by default. A master-key operator can use Check for price updates in dashboard Settings to fetch the latest upstream snapshot, review the added, changed, and removed rates, and explicitly accept or reject it. Pending reviews and accepted snapshots are stored in the database, so the decision survives a standalone restart. The accepted snapshot uses source genai-prices. Stored custom prices always take precedence and are not changed by a refresh.

It is opt-in because a billing gateway should generally charge on rates you control: community estimates can lag or differ from real provider rates, and turning this on changes what require_pricing: true guarantees (unpriced-but-known models are auto-priced rather than rejected).

Resolution order is always database first, defaults last, so any price you set in config or via /v1/pricing overrides the community default. Defaults are used only as a lookup fallback; they are never written to the database.

Limitations when enabled:

• Tiered pricing is retained from the dataset and applied when a request crosses a configured context threshold.
• A provider-agnostic match is attempted when the exact provider is not in the dataset; an ambiguous model name could resolve to a different provider's rate. Prefer configuring such models explicitly.
• HuggingFace is modeled per inference backend, so a model is priced only when you pin a backend with the huggingface:<model>:<backend> selector (see the model reference in models.md). Auto routing and the policy suffixes (:cheapest, :fastest, ...) cannot be priced from the id alone and fall through to require_pricing.

Fail-closed by default. With require_pricing: true (the default), a request for a model that has no pricing entry is rejected with HTTP 402 rather than served free and unmetered; an unpriced model would otherwise bypass the budget cap. To run genuinely free or self-hosted models, add an explicit $0 pricing entry, or set require_pricing: false. Audio and moderation endpoints are exempt. A startup warning is logged if require_pricing is on with no pricing configured.