OrcaRouter Lite

Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. model="auto".

Languages

• English
• 日本語
• 中文
• 한국어
• Deutsch
• Français
• Español
• Italiano
• Русский
• Português
• Tiếng Việt
• हिन्दी

OrcaRouter Lite is the open-source single-workspace edition of OrcaRouter. Run it on your laptop, ship it in your product, or use hosted api.orcarouter.ai directly for the long tail of models you don't want to manage keys for.

Why us? LiteLLM is a library; OpenRouter is closed-source hosted; Ollama is local-only. We're the self-hosted server with a managed fallback — a sentence none of those can say.

60-second quickstart

Two ways to use OrcaRouter:

Path A — Self-hosted (BYOK)

Run Lite on your own machine; bring your own provider keys.

git clone https://github.com/Continuum-AI-Corp/OrcaRouter-Lite.git
cd OrcaRouter-Lite
cp .env.example .env
# add at least one: OPENAI_API_KEY=sk-...  (or ORCAROUTER_API_KEY=...)

docker compose up
# logs: ✓ orcarouter-lite ready. API key: sk-orca-abc123...

Base URL: http://localhost:8000/v1. Use the sk-orca-* key printed at startup.

Path B — Hosted (account required)

No clone, no docker. Register, get a key, point any OpenAI SDK at hosted.

# 1. Register at https://www.orcarouter.ai and copy your sk-orca-* key
# 2. Use https://api.orcarouter.ai/v1 as the base URL

Account required. Hosted handles routing, billing, and the long tail of providers — billed per-token on your OrcaRouter account. See docs.orcarouter.ai/introduction.

Then call it from any OpenAI SDK

Examples below use Path A's localhost base URL — swap for https://api.orcarouter.ai/v1 if you're on Path B.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-orca-abc123...",
)
r = client.chat.completions.create(
    model="auto",  # or "gpt-4o-mini", "claude-3-5-sonnet-latest", ...
    messages=[{"role": "user", "content": "Hello!"}],
)
print(r.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "sk-orca-abc123...",
});

const r = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(r.choices[0].message.content);

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-orca-abc123..." \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'

Open http://localhost:8000/ for the dashboard — providers, routing, analytics, keys (Path A only).

Why?

model="auto" — the headline feature

Send model="auto" and OrcaRouter picks the cheapest model in your configured providers that meets the request's capability requirements (tools, vision, JSON mode). No manual routing rules; no rate-limit gymnastics; no if x: ... cost optimization in your code.

client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:..."}},
    ]}],
)
# → routes to the cheapest VISION-capable model your keys cover

The resolved model is exposed back to callers via the x-orca-resolved-model response header so you can log/display what was actually used.

Hosted as upstream (Lite + hosted)

Already running Lite? Set ORCAROUTER_API_KEY to your sk-orca-* from www.orcarouter.ai, and hosted becomes one more provider in the routing chain — covering models your local keys don't:

# .env
ORCAROUTER_API_KEY=sk-orca-hosted-abc...

Use cases:

• Try-before-you-buy — no local provider keys needed
• Local logging — hosted handles routing, Lite stores RequestLog rows for the dashboard
• Failover — local providers fail, hosted is the safety net

Streaming

OpenAI-compatible SSE format with the standard data: ... \n\n framing and a terminal [DONE] sentinel — drop-in for any SDK that already streams from OpenAI.

for chunk in client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Model catalog

100+ chat models are loaded at startup from LiteLLM's community-maintained pricing database — no model list to maintain manually. Each entry exposes:

• id (e.g. gpt-4o, claude-3-5-sonnet-latest)
• provider (mapped to your configured keys)
• Capability flags: supports_tools, supports_vision, supports_json_mode
• Per-token input/output cost (drives the savings widget + model="auto")

GET /v1/models returns the OpenAI-format catalogue.

Deploy somewhere else

Platform	One-click
Railway
Fly.io	fly launch --dockerfile Dockerfile
Render	Connect repo, root dir = .
Bare Docker	docker run -p 8000:8000 -e OPENAI_API_KEY=... ghcr.io/... (image coming soon)

What's in the box

• POST /v1/chat/completions — proxy + streaming + model="auto" + cross-provider prompt cache
• GET /v1/models — discoverable model catalog (100+ models from litellm.model_cost)
• GET/PUT/DELETE /v1/providers/{provider} — set / list / revoke encrypted provider keys
• GET/PUT /v1/routing — change strategy (balanced / cheapest / fastest / quality)
• GET /v1/analytics/{recent,spend,latency,savings,unreachable} — local analytics, no telemetry leaves the box
• GET /v1/hosted — hosted-fallback status (drives the dashboard's "Get $5 free credit" card)
• GET/POST/DELETE /v1/keys/... — list / rotate / revoke API keys
• Single-page dashboard at /
• SQLite by default; Postgres opt-in via DATABASE_URL; Redis optional

Cross-provider prompt cache

Deterministic requests (temperature=0 or pinned seed) are served from cache on repeat — works across every provider, not just Anthropic. Backend is Redis when REDIS_URL is set, in-process LRU otherwise. Cache hits return instantly with x-orca-cache: HIT and cost $0.

$ curl ... -d '{"model":"auto","messages":[...], "temperature": 0}' -i
HTTP/1.1 200 OK
x-orca-cache: MISS
x-orca-resolved-model: gpt-4o-mini

$ curl ...  # same payload again
HTTP/1.1 200 OK
x-orca-cache: HIT          ← served from cache, no upstream call

Savings widget

GET /v1/analytics/savings?baseline=gpt-4o&days=7 reports what your traffic would have cost on always-GPT-4 vs what it actually cost. The dashboard shows it as a tile.

Integrations

Drop-in configs for Continue.dev, Aider, Cursor, LangChain, LlamaIndex, Vercel AI SDK, and any tool that speaks the OpenAI Chat Completions protocol. See integrations/.

What's deliberately not

This is the single-workspace edition. By design, no:

• multi-tenancy, RBAC, SSO
• billing, wallets, points, partner program
• admin console, audit logs, trust & safety
• multi-pod deployment / Kubernetes
• email / Slack / webhooks for alerts

For those, see the hosted product or the (forthcoming) Teams edition.

Testing

Built test-first. Every behaviour shipped here had a failing test first.

pip install -e ".[dev]"
PYTHONPATH=. pytest -v
# 127 passed

Architecture

app/
├── main.py             FastAPI factory + lifespan + SPA mount
├── config.py           Settings (~15 fields)
├── deps.py             DI helpers
├── seed.py             First-run bootstrap
├── auto_routing.py     model="auto" capability + cost scoring
├── router_cache.py     Single-workspace router
├── prompt_cache.py     Cross-provider exact-match cache (Redis or in-memory LRU)
├── schemas.py          OpenAI-compatible request schema
├── middleware/auth.py  sk-orca-* validation
└── routes/
    ├── chat.py         /v1/chat/completions  (blocking + streaming)
    ├── models.py       /v1/models
    ├── providers.py    BYOK CRUD
    ├── routing.py      strategy config
    ├── analytics.py    recent / spend / latency / savings / unreachable
    ├── keys.py         list / rotate / revoke API keys
    ├── hosted.py       /v1/hosted — hosted-fallback status for the dashboard
    └── health.py

packages/
├── litellm_adapter/    Router wrapper + 100+ model catalog
├── auth/               hashing + AES-256-GCM
└── db/                 models + engine + session

Roadmap

• OpenAI-compatible chat completions
• Streaming (SSE)
• model="auto" cheapest-capable routing
• Hosted-as-upstream
• Encrypted BYOK at rest
• Local analytics dashboard
• CI (GitHub Actions)
• Cross-provider prompt caching
• Continue.dev / Aider / LangChain / Cursor / Vercel AI SDK integrations
• Public benchmark + savings claim
• Embeddings + image-gen proxy

See DEMO.md for the failover demo.

License

MIT. See LICENSE.