litellm-rs

A high-performance self-hosted LLM gateway with a stable OpenAI-compatible HTTP contract. The litellm-rs crate is the gateway's reusable Rust kernel, with narrower support policies for runtime-backed APIs and legacy compatibility adapters.

Features

• 60+ runtime-wired providers - OpenAI, Anthropic, AWS Bedrock, Mistral, Cloudflare, plus 50+ OpenAI-compatible providers via the Tier 1 catalog. See Provider Support for the full matrix.
• Stable OpenAI-Compatible API - Versioned inference contract served at GET /openapi.json
• Admin control-plane OpenAPI - Auth, keys, teams, budgets, routing, ledger, and provider APIs at GET /admin/openapi.json
• Measured Performance - Reproducible gateway-overhead benchmark methodology
• Intelligent Routing - Load balancing, failover, cost optimization
• Gateway Controls - Default-on prompt-injection guardrails, configured IP access, auth, rate limiting, deterministic caching, metrics, and health endpoints

Quick Start: Self-Hosted Gateway

Run the primary supported product from source:

git clone https://github.com/majiayu000/litellm-rs.git
cd litellm-rs
cp config/gateway.dev.yaml.example config/gateway.yaml
cargo run --bin gateway

Or install the gateway binary:

cargo install litellm-rs --bin gateway
mkdir -p config
curl -L https://raw.githubusercontent.com/majiayu000/litellm-rs/main/config/gateway.dev.yaml.example -o config/gateway.yaml
gateway

The development config starts without provider credentials or auth secrets and uses the local vllm catalog provider. Use config/gateway.yaml.example for production-style deployments with real provider keys and auth enabled. Default features include SQLite storage, which satisfies the gateway binary's storage requirement.

The gateway serves its stable inference contract at GET /openapi.json; the versioned source is docs/openapi/inference.json. The admin control-plane contract is served at GET /admin/openapi.json (admin-authenticated) from docs/openapi/admin.json.

Supported Product Surfaces

Surface	Support policy
Self-hosted HTTP gateway	Primary product. Stable inference routes follow the published OpenAPI contract; configured deployments expose their canonical ProviderCapability.
Runtime-backed Rust APIs	Reusable gateway kernel. LLMClient::from_runtime, DefaultRouter::from_runtime, and runtime-configured completion() derive support from the selected deployment and fail unsupported capabilities with typed errors.
Legacy SDK and selector adapters	Compatibility surfaces with narrower coverage. The legacy adapter matrix is authoritative for these paths and is not a canonical runtime capability matrix.

Rust Crate

[dependencies]
litellm-rs = { version = "0.6", default-features = false, features = ["lite"] }

No make step is required for crate consumers.

Library Example

use litellm_rs::{completion, user_message, system_message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = completion(
        "gpt-4",
        vec![
            system_message("You are a helpful assistant."),
            user_message("Hello!"),
        ],
        None,
    ).await?;

    println!("{}", response.choices[0].message.content.as_ref().unwrap());
    Ok(())
}

Gateway Configuration

The gateway router config maps these fields into the runtime router:

• router.strategy selects the deployment routing strategy.
• router.circuit_breaker.failure_threshold controls consecutive failures before cooldown.
• router.circuit_breaker.recovery_timeout controls cooldown duration in seconds.
• router.circuit_breaker.min_requests sets the sample size required before cooldown.
• router.circuit_breaker.success_threshold sets the successes required to recover from cooldown.
• router.load_balancer.health_check_enabled enables pre-call deployment health checks.

Active provider probes are opt-in: a provider's health_check must differ from the defaults. Native probes are limited to provider types whose configured deployment is unambiguously a chat model (currently Anthropic and GitHub Copilot); they send a one-token request and may incur provider charges. OpenAI, Bedrock, OpenAI-compatible, Vertex AI, FalAI, and other multi-capability providers require a custom health_check.endpoint. Without an active probe policy, readiness remains fail-closed (Unknown).

router.load_balancer.sticky_sessions and router.load_balancer.session_timeout are reserved for future session affinity. Non-default values fail config validation until runtime affinity is implemented.

Gateway YAML can publish stable model names and deterministic primary/fallback tiers:

providers:
  - name: openai-primary
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 0
  - name: openai-fallback
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 10

model_aliases:
  production-chat: gpt-4o
  stable-chat: production-chat

router:
  strategy: priority_based

Alias chains are validated and flattened at startup; empty values, cycles, canonical-name collisions, and targets without an enabled deployment fail startup. Alias names appear in /v1/models alongside canonical models. Lower numeric priority wins under priority_based; omitted provider priorities default to 0. When rolling back to a binary that predates these fields, remove model_aliases and priority from YAML before rolling back the binary, because unknown fields are rejected.

Core Subsystem Runtime Status

Runtime wiring decisions are tracked in src/core/subsystem_registry.rs, and tests assert that every module exported from src/core/mod.rs is either referenced by the gateway runtime or explicitly classified. The current issue-838 subsystem decisions are:

Subsystem	Decision	Runtime status
core/guardrails	wire	Default-on prompt-injection checks run before provider execution and on non-streaming output; guardrails.enabled: false is the explicit opt-out.
core/ip_access	wire	Configured allow/block rules run as an outer Actix middleware and short-circuit before downstream side effects; empty/default rules allow all.
core/mcp	experimental-gate	Deprecated in 0.6 and excluded from default builds behind mcp; enabling it exposes library types but mounts no HTTP route. Removal is scheduled for 0.7. Responses API MCP descriptors still pass through independently.
core/a2a	experimental-gate	Deprecated in 0.6 and excluded from default builds behind a2a; enabling it exposes library types but mounts no HTTP route. Removal is scheduled for 0.7.
core/realtime	experimental-gate	Deprecated in 0.6 and default-off behind websockets; no gateway route is mounted. Removal is scheduled for 0.7.
core/observability and core/integrations	wire	Configured Langfuse, OpenTelemetry, and Datadog backends are initialized at startup and receive real chat, completion, response, and embedding lifecycle events.
core/audit	wire	enterprise.audit_logging: true registers request audit middleware; events use structured JSON on stderr unless a file or custom output is configured. Default is off.
core/batch	library-only	/v1/batches remains a wired provider proxy. The unreachable BatchProcessor is deprecated in 0.6 and scheduled for removal in 0.7.
core/webhooks	experimental-gate	Deprecated in 0.6 and excluded from default builds behind webhooks; it is not a gateway runtime capability and is scheduled for 0.7 removal.
core/semantic_cache	remove	Deprecated but retained with storage during the 0.6 compatibility window; cache.semantic_cache=true remains rejected before the planned 0.7 removal.
core/analytics	remove	Deprecated and default-off behind analytics, with removal planned for 0.7.
core/virtual_keys	wire	Runtime virtual keys use the canonical core::keys::KeyManager; the duplicate legacy VirtualKeyManager is deprecated for 0.7 removal.
core/user_management	internal/gated	Compatibility records back current auth/storage paths; the deprecated UserManager implementation is default-off behind user-management and scheduled for 0.7 removal.

Installation

# Full gateway with SQLite + Redis (default)
[dependencies]
litellm-rs = "0.6"

# API-only - lightweight, no actix-web/argon2/aes-gcm/clap
[dependencies]
litellm-rs = { version = "0.6", default-features = false }

# API-only with metrics
[dependencies]
litellm-rs = { version = "0.6", default-features = false, features = ["lite"] }

# Gateway modules in library context (not standalone gateway binary runtime)
[dependencies]
litellm-rs = { version = "0.6", default-features = false, features = ["gateway"] }

Provider Support

Providers are organised into two tiers (see CLAUDE.md → Provider Tiers for the engineering definition).

• Tier 1 — catalog-only: OpenAI-compatible endpoints declared as data in src/core/providers/registry/catalog.rs. Routed through OpenAILikeProvider. Always available (no cargo feature required). The current crate runtime exposes chat completions and chat streaming for these providers; embeddings, images, audio, and other non-chat endpoints are not forwarded yet.
• Tier 2 — code-based: providers with custom request/response handling, auth signing, or streaming. Wired into the Provider enum and the factory. Some Tier 2 builders are feature-gated.

Router deployments use the closed Provider enum. Implementing LLMProvider alone does not make a third-party provider routeable; use the generic OpenAI-compatible path for compatible endpoints, or wire a code-based provider into the enum, dispatch, registry metadata, and factory.

The provider and legacy adapter matrices below are validated against the provider registry and Tier 1 catalog. The source of truth for Tier 1 entries is catalog.rs; Tier 2 identity and dispatch metadata lives in src/core/providers/registry/types.rs, with construction branches in src/core/providers/factory/registry.rs. Legacy adapter availability lives in src/core/providers/registry/support_matrix.rs. passthrough means a retained adapter forwards the call to the upstream OpenAI-compatible endpoint without per-provider transformation.

Legacy adapter matrix

This matrix records selector-based compatibility adapters. It is not the capability source for LLMClient::from_runtime, DefaultRouter::from_runtime, or the free completion() functions. Canonical runtime support is derived from the selected deployment's ProviderCapability; unsupported capabilities fail closed with a typed provider error.

Legacy selector class	HTTP chat / stream adapter	HTTP embeddings / image adapter	SDK chat / stream / embeddings adapter	completion() chat / stream adapter	Notes
openai	✅ / ✅	✅ / ✅	✅ / ✅ / ✅	✅ / ✅	Reference provider across all current surfaces.
anthropic	✅ / ✅	– / –	✅ / ✅ / –	✅ / ✅	Native chat and streaming only.
azure	passthrough / passthrough	providers-extra / providers-extra	– / – / ✅	passthrough / passthrough	SDK exposes Azure embeddings; SDK chat is not implemented.
azure_ai	passthrough / passthrough	providers-extra / providers-extra	– / – / –	providers-extra / providers-extra	completion() supports azure_ai/ and azure-ai/ routes when the native feature is enabled.
bedrock	✅ / ✅	✅ / –	– / – / –	– / –	No legacy SDK or completion() adapter. Configured canonical runtimes use the deployment's chat, stream, and embedding capabilities.
databricks, snowflake	passthrough / passthrough	– / –	– / – / –	– / –	Governed OpenAI-compatible chat/SSE runtimes with platform-specific identity and auth.
oci	mode-dependent	mode-dependent / –	– / – / –	– / –	Compatible mode provides chat/SSE; IAM native mode provides embeddings and rerank.
watsonx	✅ / –	✅ / –	– / – / –	– / –	Native chat, embeddings, and rerank.
sagemaker	✅ / –	– / –	– / – / –	– / –	InvokeEndpoint requires an explicit supported payload transformer.
mistral, cloudflare, cohere, voyage, vertex_ai, gemini, fal_ai, replicate, stability, black_forest_labs, ollama	provider-specific	provider-specific	– / – / –	– / –	See support_matrix.rs for feature-gated HTTP support. Ollama retains its existing SDK stream-only path.
google / SDK Google	– / –	– / –	– / – / –	– / –	Google/Gemini SDK chat is intentionally unsupported until a real adapter exists.
Default catalog dynamic routes: openrouter, deepseek, moonshot, minimax, zhipu, zai, together_ai, fireworks_ai, aiml, groq, xiaomi_mimo, xai	passthrough / passthrough	– / –	– / – / –	✅ / ✅	OpenAI-compatible routes wired into default completion() routing.
Other Tier 1 catalog providers	passthrough / passthrough	– / –	– / – / –	– / –	HTTP gateway chat/stream only unless routed through explicit OpenAI-compatible config.
SDK Custom	– / –	– / –	– / – / ✅	– / –	SDK custom providers support embeddings when base_url is configured.
SDK Ollama	– / –	– / –	– / ✅ / –	– / –	SDK streaming uses the OpenAI-compatible stream parser; SDK chat is not implemented.

The runway-media feature exposes a library-only Runway task contract for text/image-to-video submit, query, cancel, and result polling. It does not mount a gateway video route.

Tier 2 — code-based providers

Provider	Cargo feature	Chat	Stream	Embed	Image	Audio	Notes
OpenAI (openai)	always	✅	✅	✅	✅	✅	Reference implementation.
Anthropic (anthropic)	always	✅	✅	–	–	–	Native Anthropic messages API.
Mistral (mistral)	always	✅	✅	passthrough	–	–	Native client.
Cloudflare Workers AI (cloudflare)	always	✅	–	–	–	–	Native client with account-id auth; streaming and embeddings currently return NotSupported.
Deepgram (deepgram)	always	–	–	–	–	✅	Native speech-to-text and text-to-speech REST transport.
ElevenLabs (elevenlabs)	always	–	–	–	–	✅	Native speech-to-text and text-to-speech REST transport.
Cohere (cohere)	native factory (providers-extended)	✅	✅	✅	–	–	Uses native Cohere /v2/chat and /v2/embed; the concrete provider also exposes a /v1/rerank helper. Explicitly unsupported without providers-extended.
Voyage (voyage)	always	–	–	✅	–	–	Uses native Voyage /v1/embeddings and the shared HTTP /v1/rerank route with exact model capability and pricing identity.
Azure OpenAI (azure)	native factory (providers-extra); OpenAILike fallback	✅	✅	✅	✅	–	Native Azure supports chat, streaming, embeddings, and image generation with providers-extra; otherwise the factory path uses OpenAILike chat/stream only.
Azure AI Inference (azure_ai)	native factory (providers-extra); OpenAILike fallback	✅	✅	✅	✅	–	Native Azure AI supports chat, streaming, embeddings, and image generation with providers-extra; otherwise the factory path uses OpenAILike chat/stream only.
AWS Bedrock (bedrock)	always	✅	✅	✅	helper API	–	Native AWS Bedrock runtime path with SigV4 signing. Use openai_compatible for Bedrock Access Gateway or other OpenAI-compatible proxies.
Databricks Model Serving (databricks)	always	✅	✅	–	–	–	OpenAI-compatible chat/SSE with typed workspace and token configuration.
Snowflake Cortex (snowflake)	always	✅	✅	–	–	–	OpenAI-compatible chat/SSE with typed account and token-type configuration.
OCI Generative AI (oci)	always	✅	✅	✅	–	–	Compatible mode provides chat/SSE; IAM native mode provides embeddings and rerank.
IBM watsonx.ai (watsonx)	always	✅	–	✅	–	–	Native chat, embeddings, and rerank with explicit project or space identity.
Amazon SageMaker (sagemaker)	always	✅	–	–	–	–	SigV4 InvokeEndpoint; payload transformer is required and unknown schemas fail closed.
Google Vertex AI (vertex_ai)	native factory (providers-extra)	✅	✅	✅	✅	–	Uses native Vertex auth and Google-specific URLs when providers-extra is enabled; otherwise explicitly unsupported.
Google Gemini (gemini)	native factory (providers-extended)	✅	✅	–	–	–	Uses native Google AI Studio Gemini auth; use vertex_ai for Vertex AI project/location credentials.
Meta Llama API (meta_llama)	catalog-only (OpenAILike)	✅	✅	–	–	–	Native module retained behind providers-extra, but runtime construction is catalog metadata.
Vercel v0 (v0)	catalog-only (OpenAILike)	✅	✅	–	–	–	Native module retained behind providers-extra, but runtime construction is catalog metadata.
Amazon Nova (amazon_nova)	catalog-only (OpenAILike)	✅	✅	–	–	–	Native module retained behind providers-extended, but runtime construction is catalog metadata.
fal.ai (fal_ai)	native factory (providers-extended)	–	–	–	✅	–	Uses native Fal AI image-generation endpoints; chat and streaming are explicitly unsupported.
Stability AI (stability)	native factory (providers-extended)	–	–	–	✅	–	Uses native v2beta multipart image generation and editing endpoints.
Black Forest Labs (black_forest_labs)	native factory (providers-extended)	–	–	–	✅	–	Uses native asynchronous submit/poll image generation and Kontext editing.
Replicate (replicate)	native factory (providers-extended)	✅	✅	–	✅	–	Uses native Replicate prediction lifecycle handling for chat, streaming, and image generation; explicitly unsupported without providers-extended.
Ollama (ollama)	native factory (providers-extended)	✅	✅	✅	–	–	Uses native /api/chat NDJSON streaming, /api/embed, and model tags/show endpoints. Localhost defaults to private-network endpoint policy; explicit endpoints keep their configured policy.
GitHub Models (github)	catalog-only (OpenAILike)	✅	✅	–	–	–	Native module retained behind providers-extended, but runtime construction is catalog metadata.
GitHub Copilot (github_copilot)	native factory (providers-extended)	✅	✅	–	–	–	Uses native GitHub Copilot auth and model access when providers-extended is enabled; otherwise explicitly unsupported.
Generic OpenAI-compatible (openai_compatible)	always	✅	✅	passthrough	passthrough	passthrough	For self-hosted / unlisted OpenAI-compatible chat, embeddings, image, and audio endpoints.

Tier 1 — catalog providers (OpenAI-compatible, always available)

All entries below route through OpenAILikeProvider. Chat and streaming work for any endpoint that follows OpenAI's /chat/completions SSE protocol. Embeddings, images, audio, and other non-chat endpoints are not exposed through this path today, even when the upstream provider offers them.

Cloud (Bearer auth via env var):

groq, ai21, huggingface, baseten, together, together_ai, fireworks, fireworks_ai, perplexity, cerebras, openrouter, deepinfra, deepseek, novita, nvidia_nim, nebius, nscale, hyperbolic, featherless, galadriel, sambanova, heroku, friendliai, xai, moonshot, dashscope, qwen, baichuan, minimax, volcengine, xiaomi_mimo, zhipu, zai, lemonade, linkup, poe, wandb, nanogpt, aiml_api, aiml, aleph_alpha, anyscale, bytez, comet_api, compactifai, maritalk, siliconflow, yi, lambda_ai, ovhcloud

Local (no API key):

vllm, hosted_vllm, lm_studio, llamafile, docker_model_runner, xinference, infinity, oobabooga

Experimental / module-only

The following modules exist under src/core/providers/ (gated on providers-extra or providers-extended) but are not wired into the unified Provider enum or the factory today. They compile but cannot be selected through create_provider/from_config_async. Treat them as experimental scaffolding subject to change:

custom_api

For self-hosted or unlisted OpenAI-compatible endpoints, prefer the generic openai_compatible provider type instead.

Environment Variables

# Provider API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
AZURE_OPENAI_API_KEY=...
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
GROQ_API_KEY=...
AI21_API_KEY=...
HF_TOKEN=...
BASETEN_API_KEY=...
DEEPSEEK_API_KEY=...
MOONSHOT_API_KEY=...
ZHIPU_API_KEY=...
MINIMAX_API_KEY=...

# Optional
LITELLM_VERBOSE=true  # Enable verbose logging

Examples

Multi-Provider Routing

use litellm_rs::{completion, user_message};

// Automatically routes to the right provider based on model name
let openai = completion("gpt-5.5", vec![user_message("Hi")], None).await?;
let anthropic = completion("anthropic/claude-opus-4-8", vec![user_message("Hi")], None).await?;
let groq = completion("groq/llama-3.1-8b-instant", vec![user_message("Hi")], None).await?;
let bedrock = completion(
    "bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    vec![user_message("Hi")],
    None,
)
.await?;

bedrock/ uses the native AWS Bedrock provider. It signs requests with AWS SigV4 and preserves AWS execution model IDs such as us.*, global.*, region-prefixed IDs, and Bedrock ARNs. Use openai_compatible for Bedrock Access Gateway or other OpenAI-compatible proxies instead.

Embeddings

use litellm_rs::{embedding, embed_text};

// Single text
let embedding = embed_text("text-embedding-3-small", "Hello world").await?;

// Batch
let embeddings = embedding(
    "text-embedding-3-small",
    vec!["Hello", "World"],
    None,
).await?;

Streaming

use litellm_rs::{completion_stream, user_message};
use futures::StreamExt;

let mut stream = completion_stream(
    "gpt-4",
    vec![user_message("Tell me a story")],
    None,
).await?;

while let Some(chunk) = stream.next().await {
    if let Ok(chunk) = chunk {
        print!("{}", chunk.choices[0].delta.content.unwrap_or_default());
    }
}

Performance

Gateway throughput and latency claims require a dated raw artifact tied to an exact Git revision. See the reproducible gateway-overhead benchmark methodology for the fixed workload, commands, environment metadata, percentile reporting, and artifact format.

The Criterion benchmarks under benches/ measure in-process components; they must not be presented as end-to-end HTTP gateway throughput.

Troubleshooting

Build/test uses too much CPU or memory

• Use API-only defaults first: cargo test --lib --tests --no-default-features --features "lite"
• Limit local parallelism when needed: CARGO_BUILD_JOBS=4 cargo test --lib --tests --no-default-features --features "lite" -- --test-threads=4
• Avoid --all-features unless you are doing release/nightly validation

I only need provider API aggregation, not gateway

• Prefer default-features = false with features = ["lite"]
• Use gateway runtime commands only when you need HTTP server/auth/storage middleware

Documentation

• API Documentation
• Documentation Index
• Development Gateway Config
• Production Gateway Config
• Examples

Contributing

See CONTRIBUTING.md for development setup and guidelines.

Security

See SECURITY.md for security policy and vulnerability reporting.

The Agent Infra Stack

This project is one layer of an open-source stack for running coding agents (Claude Code, Codex) as serious infrastructure. Every piece works standalone; together they close the loop:

litellm-rs is the Route layer — the gateway underneath everything else, speaking OpenAI format to 100+ providers.

Layer	Project	What it does
Extend	claude-skill-registry	Discover and search community Claude Code skills
Extend	spellbook	Cross-runtime skills for Claude Code, Codex, and multi-agent workflows
Trust	argus	Static install-time scanner for supply-chain attacks (npm / PyPI / crates.io)
Trust	vibeguard	Rules, hooks, and guards against hallucinated or unverified agent changes
Remember	remem	Local-first persistent memory for Claude Code and Codex sessions
Orchestrate	harness	Rust agent orchestration platform — rules, skills, GC, observability
Route	litellm-rs ◀ you are here	High-performance Rust AI gateway — 100+ LLM APIs via OpenAI format
Keep	keepline	Session command center — monitor, recover, never lose agent work

---

License

MIT License - see LICENSE for details.

Acknowledgments

Inspired by LiteLLM (Python).