jev-router

September 20, 2026 · View on GitHub

LiteLLM proxy that routes each coding-agent turn to a cloud or local lane. jev decides; LiteLLM forwards.

Status: early. v1 routes by task difficulty only; the judge, policy, and extractor are swappable Protocols.

Run

uv sync
export TYPESAFE_API_KEY=...          # jev
export CLOUD_API_BASE=...            # e.g. https://api.anthropic.com
export CLOUD_API_KEY=...
export LOCAL_API_BASE=http://localhost:1234/v1   # LM Studio / mlx_lm.server / Ollama
export LITELLM_MASTER_KEY=sk-anything
uv run litellm --config litellm.config.yaml --port 4000

With judge.kind: jev (the default) the proxy refuses to start unless TYPESAFE_API_KEY is set; use judge.kind: fixed to run without it.

Edit model values in litellm.config.yaml to match your endpoints. ROUTER_CONFIG sets the path to the router config (default router.yaml).

Point Claude Code at it

export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY
claude

How a request is routed

  1. Hook extracts the latest user text and a conversation id.
  2. Requests whose model name contains haiku (configurable: passthrough.local_if_requested_model_contains) go straight to local without judging; Claude Code uses these for titles and summaries.
  3. New user turn → ask jev needs_frontier (probability). Tool-result continuation → reuse the lane from the sticky store.
  4. >= 0.6 cloud, <= 0.4 local, otherwise default_lane. jev error → default_lane.
  5. model is rewritten to the lane; LiteLLM forwards. local failures fall back to cloud.

Every decision is appended to logs/decisions.jsonl.

What is sent to jev

  • The latest user text (up to 2000 chars, <system-reminder> blocks removed).
  • The text of up to 4 prior messages (500 chars each).
  • The client name and whether tools are present.

Tool results, tool inputs, and the system prompt are never sent. logs/decisions.jsonl stores the first 100 chars of user text plus jev's raw answers locally.

Offline / test mode

Set judge.kind: fixed in router.yaml; no jev key needed. scripts/integration.sh runs an end-to-end check with fake backends.

Layout

router/state.py   RouteState, AnthropicMessagesExtractor
router/judge.py   Decision, Judge, FixedJudge, JevJudge
router/policy.py  ThresholdPolicy
router/sticky.py  InMemoryStickyStore
router/config.py  RouterConfig, load_config
router/log.py     DecisionLog
router/hook.py    Router (LiteLLM CustomLogger), build_router
router/entry.py   router_instance

Extension points

  • New judge (rules, local jev): implement Judge, select it in build_router.
  • New client format (OpenAI chat): implement StateExtractor.
  • More lanes: add to model_list, extend Policy.

Verified

  • scripts/integration.sh passes: call_type for /v1/messages is anthropic_messages, payload carries messages at top level.
  • Live jev call: JevJudge returns calibrated scores (trivial prompt → 0.17, refactor prompt → 0.63) — 2026-09-20.
  • Not run: a live Claude Code turn through real cloud/local endpoints. Manual steps: set CLOUD_API_BASE/CLOUD_API_KEY/LOCAL_API_BASE, start the proxy per Run, point Claude Code at it, send one trivial and one hard prompt, then tail logs/decisions.jsonl.

License

MIT