Cost & model routing

July 16, 2026 · View on GitHub

The single biggest lever on agent cost is which model does which job. Token counts dwarf per-token price differences, so routing the bulk (mechanical, parallel work) to cheap models while keeping deep reasoning on Opus wins on both cost and speed.

The three tiers

TierModelWhere it's setJobs
CheapHaiku 4.5test-runner subagentrun tests, parse failures, triage logs, mechanical edits, "find all callers" sweeps
StandardSonnet 4.6code-reviewer, go-engineer, rust-engineer, docs-writer, k8s-operatormost feature coding, refactors, reviews, docs
DeepOpus 4.8driver session + architect, security-auditor, debuggerarchitecture, security, subtle debugging

The deep tier tracks the newest Opus — bumped to Opus 4.8 (claude-opus-4-8, shipped 2026-05-28) day one. Change a subagent's tier by editing the model: line in its claude/agents/<name>.md. Accepted values: a model ID (claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5-20251001) or inherit.

How the savings actually happen

  • Delegation keeps the driver's context small. A subagent reads 20 files and returns a 5-line conclusion; the expensive driver context never holds those 20 files. Less context = fewer tokens per subsequent turn.
  • Cheap models do the volume. Test runs and log triage are high-token, low-reasoning. On Haiku they're nearly free.
  • /cost re-plans a task into a delegation table before you spend.

Effort level

effortLevel: "high" in settings.json buys deeper reasoning per turn (worth it on the Opus driver). Subagents on cheaper models implicitly cost less per turn; you can also dial Codex reasoning per profile (deep/standard/cheap).

/effort ultracode (Opus 4.8+) goes further: xhigh reasoning plus automatic dynamic-workflow orchestration — Claude plans a fan-out workflow for each substantive task instead of working turn-by-turn. It's the most expensive setting (several workflows can run for one request), so reach for it on genuinely hard, wide problems and drop back with /effort high for routine work.

Prompt caching (automatic)

Claude Code prompt-caches the system prompt, tool definitions, and conversation prefix automatically — you don't toggle it. compass is structured to maximize the hit rate rather than fight it:

  • Stable system prefixes. Each orchestrate.sh step passes a fixed role file via --append-system-prompt-file (sdlc/roles/*.md), and the operating manual is one unchanging file — a long, identical prefix that caches across steps and sessions.
  • Byte-identical loop prompts. The converge loop (SDLC_CONVERGE=1) re-issues the same reviewer prompt every round, so each re-review reuses the cached prefix — a big reason iterating to green stays cheap.
  • Keep your project CLAUDE.md lean (philosophy §3): a smaller stable prefix is both cheaper to cache and higher-signal. Building on the API directly? The claude-api skill bakes in cache_control so apps cache by default.
  • Prefix-stability discipline (append, don't insert). The fastest way to lose the cache is to change something early in the prompt: loading a skill or tool mid-run, or editing a role file's preamble, relocates the cache boundary and busts everything after it. So when authoring roles/subagents/skills — keep the shared preamble byte-identical, put volatile content (task args, dates, freshly-loaded memory) at the tail, and load the union of likely skills up front rather than lazily mid-conversation. A later edit then costs a partial re-write of the tail, not a full cache miss. (This is the load-bearing lesson behind the two bullets above; it generalizes to any agent loop, not just compass's.)

1-hour cache TTL (extended)

The default cache lives 5 minutes; the extended TTL is 1 hour (GA). 1h costs ~2× on the write but reads stay at 0.1× — a net win only when the same large prefix is reused across calls more than 5 minutes apart, which is exactly the SDLC pipeline + converge loop (review → fix → review can span minutes-to-an-hour). It's not a win for one-shots (compass onboard, the router's tiny LLM fallback) — those would just pay the write premium.

Claude Code exposes this only as an env var (ENABLE_PROMPT_CACHING_1H=1, v2.1.108+) — no CLI flag, no settings.json field. compass applies it where it pays:

  • sdlc/orchestrate.sh exports ENABLE_PROMPT_CACHING_1H=1 for its claude -p steps (opt out with SDLC_CACHE_1H=0).
  • GitHub SDLC loop (claude-code-action): set it as a repo/Actions env var or in the workflow's env: if you want the hosted converge loop on the 1h TTL too.
  • Interactive sessions: add export ENABLE_PROMPT_CACHING_1H=1 to your shell rc if your day is long, context-heavy sessions (the common case); skip it for short, bursty ones.
  • It does not apply to the raw-API path in router/fallback-llm.sh — that prompt is below the 1,024-token cache minimum, so caching is a no-op there regardless of TTL.
  • Bedrock caveat: on the Claude-via-Bedrock path (CLAUDE_CODE_USE_BEDROCK) this env var may be silently ignored — Claude Code has historically pinned the Bedrock cache TTL to 5m (anthropics/claude-code#32671). Bedrock itself supports 1h on the 4.5 models (a cachePoint with "ttl":"1h" in the Converse API), but that's the raw API, not the harness path. Treat 1h as a no-op on Bedrock until that's fixed; the default 5m cache still works, and reads still drop to 0.1×.

Bring your own model — local LLMs & cost routers (Codex side)

The cheapest token is one you don't pay for. Codex talks to any OpenAI-compatible endpoint, so a tier can run on a local model (free, private) or a cost router:

ProfileBackendUse
codex --profile localOllama / LM Studio (localhost:11434/v1)grunt work on a free local model — zero API cost
codex --profile routerOpenRouter (one key → many models)route to the cheapest capable model per task

These ship inert in codex/config.toml ([model_providers.ollama|openrouter] + the local/router profiles) — the default tiers are unchanged; opt in per command. Set OPENROUTER_API_KEY for the router; pull a coding model (e.g. ollama pull qwen2.5-coder) for local. Edit the example model names to ones you actually have.

Honest limit: this lever is Codex-side. Claude Code's core agent runs Claude (or Claude via Bedrock/Vertex for enterprise: CLAUDE_CODE_USE_BEDROCK / ..._VERTEX) — it does not run arbitrary local/OpenRouter models. So the cross-tool play is: Claude Code for deep Claude reasoning, a local/router-backed Codex for cheap high-volume work + the independent audit. Cross-provider smart routing as a first-class compass layer is roadmapped (docs/10-roadmap.md §10).

Watching spend (pre/post budgeting)

  • Live (interactive): the status line shows estimated session cost ($x.xx) and context size, so you notice a runaway before it's expensive. /cost is the deliberate pre-plan.
  • Live budget ceiling (interactive hard-stop): set a per-session cap and compass enforces it — the budget-gate.sh PreToolUse hook blocks the next tool call once estimated session spend meets/exceeds the cap, so an agent can't quietly run up the bill while you're away. It's off by default (zero overhead); turn it on per-shell or persistently: 
    export COMPASS_MAX_USD=5             # this session hard-stops at \$5
echo 'max_usd=5' >> ~/.compass/config   # persistent default for every session
    How it reads spend: the status line is the only place the runtime hands us live session cost, so it drops a per-session breadcrumb (~/.compass/sessions/<id>.cost) that the gate consults — so the ceiling is accurate to the last status-line render, not to the cent. Headless mode (claude -p / CI): the status line never renders, so the gate falls back to reading session cost from the transcript JSONL that claude -p always writes — the cap works without the status line active. Honest limit: it fails open — if the cap is unset, or spend is unknown, it never blocks; a budget ceiling must not wedge a session on missing data. It's a cost guardrail, not a security boundary. To continue past a stop, raise COMPASS_MAX_USD or start a fresh session.
  • Daily ceiling (the unattended-loop circuit breaker): a per-day cap across every run, not just one session — set it before you let loops run unattended, on the assumption something will spin idle overnight. The same budget-gate.sh hook blocks once today's total (this session's breadcrumb + every loop/routine that logged to ~/.compass/spend.tsv) meets/exceeds it: 
    export COMPASS_MAX_USD_DAY=20            # halt once today's combined spend hits \$20
echo 'max_usd_day=20' >> ~/.compass/config   # persistent daily ceiling
compass spend --today --max-usd 20      # gate a routine on the day's cumulative spend (exit 2 = over)
    Like the session cap it fails open (unknown spend never blocks) and is independent of it — a day of many cheap sessions can't blow the daily budget even if no single one trips the session cap.
  • Per-run budget cap: every autonomous step is hard-capped — --max-budget-usd on each cloud workflow step and on each orchestrate.sh Claude step (SDLC_BUDGET/4 by default).
  • Pre-run estimate: orchestrate.sh prints the per-step cap and total budget hint before it starts — you know the ceiling up front.
  • Post-run analysis: when jq is present, orchestrate.sh captures each step's real cost (claude -p --output-format json → total_cost_usd) into .sdlc/run-*/costs.tsv, prints a per-step breakdown + total, and includes a Spend line in the PR body. (QA is free; the Codex audit isn't tallied.)
  • Aggregate across runs/repos: each step's cost is also appended to a global ledger ~/.compass/spend.tsv. compass spend [--today|--week|--month|--all] rolls it up by model and repo; set a ceiling with COMPASS_BUDGET_USD (or budget_usd= in ~/.compass/config) and it shows OK / over-80% / over.
  • Hard budget gate (CI/loops): --max-usd N (or COMPASS_BUDGET_USD for soft warnings vs COMPASS_MAX_USD for hard gates) exits 2 with a machine-greppable OVER_BUDGET line when total exceeds the cap — use it as a kill-switch in CI or scheduled loops: 
    compass spend --max-usd 5          # exits 2 + prints OVER_BUDGET total=$X cap=$Y if over
compass spend --all --json --max-usd 5   # adds over_budget/remaining fields to JSON
    Flag wins over COMPASS_MAX_USD env var, so scripts can override without env surgery.
  • Is it worth it? compass impact answers "how is compass benefiting me" — footguns blocked, files auto-formatted, spend by model, and an estimated $ saved vs running everything on Opus (a rough multiple-based estimate, labelled as such).

Cross-agent budget enforcement — compass gate (experimental)

The budget gate above works only for Claude Code sessions (where the status line or transcript is accessible). compass gate extends hard-stop enforcement to any agent that speaks the Anthropic or OpenAI API — Codex, SDK scripts, or any framework pointed at api.anthropic.com / api.openai.com.

It's a localhost reverse proxy (python3 scripts/compass-gate.py):

# start the gate (uses same COMPASS_MAX_USD / COMPASS_MAX_USD_DAY env vars)
compass gate                       # listens on 127.0.0.1:4141
compass gate --port 9000           # custom port

# point any agent at it
export ANTHROPIC_BASE_URL=http://127.0.0.1:4141
export OPENAI_BASE_URL=http://127.0.0.1:4141

# check live spend without interrupting the gate
compass gate --status

Once session or daily spend (the same caps from COMPASS_MAX_USD / COMPASS_MAX_USD_DAY, read from env or ~/.compass/config) reaches the cap, all further requests get a 402 response before hitting the upstream API. The gate logs cost to the shared ~/.compass/spend.tsv ledger so compass spend tallies gate-tracked runs alongside Claude Code sessions.

Honest limits:

  • Cost is computed from token counts in the response (input × price + output × price per model). Unknown models fall back to the conservative Opus rate — over-estimates, never under.
  • For OpenAI streaming responses, cost is only tracked when the client passes stream_options: {include_usage: true} — otherwise recorded as $0.
  • python3 is required. The gate is marked experimental: it's a simple HTTP proxy with no TLS termination (it forwards the auth header as-is to the real API — never expose it on a non-loopback interface).
  • Fails open: if caps are unset, all requests pass through.

Auto-routing the model — now measured

compass route "<task>" maps a task to the cheapest-correct tier (haiku/sonnet/opus) by a deterministic keyword heuristic. orchestrate.sh uses it for the Builder step only when SDLC_AUTOROUTE=1 (off by default; the default stays Sonnet).

Cache-aware routing (router/ module, ADR-0004). The deterministic router adds a cache-aware cost-min stage that folds Anthropic prompt-cache economics (read 0.1×, write 1.25×@5m / 2×@1h) + a session warm-set into the pick — riding an already-warm tier when it's cheaper than cold-loading a cheaper one (upgrade-only, never a quality drop). Plus a budget governor, a latency ceiling, domain quality floors, and spec validation + ReDoS lint — each off unless its signal is set. router/bench.sh --cache reports the effective $ saved; see router/README.md.

The honest caveat used to be "no eval set, so a wrong route can hurt quality more than it saves." That gap is now closed: a labeled ground-truth set lives at scripts/route-evalset.tsv, and

compass route --eval        # score the router; per-tier recall + accuracy

scores the router against it. CI gates on it — a routing change that drops below the floor (COMPASS_ROUTE_MIN_ACCURACY, default 90%) fails the build, so accuracy is a checked claim, not a vibe. The set also documents the heuristic's limits honestly: a couple of context-dependent tasks (e.g. "a backward-compatible proto contract change") are irreducible misses for keyword routing — which is exactly why SDLC_AUTOROUTE stays opt-in. Add real mis-routes to the set as they surface; that's how it earns its keep.

Advanced router engine (opt-in)

The keyword router is the default and the CI accuracy floor. An advanced engine — a 9-stage pipeline with cache-aware cost-min, confidence scoring, budget bias, and domain quality floors — is available opt-in:

compass route --engine advanced "refactor the auth module"
COMPASS_ROUTE_ENGINE=advanced compass route "write unit tests for the parser"

The local classifier inside the advanced engine is not yet trained — classify.sh abstains and the pipeline falls through to the LLM judge for the ambiguous tail. See router/README.md for the full pipeline description and how to build the classifier from logged routing decisions.

Rules of thumb baked into CLAUDE.md

  • Don't re-read a file you just edited.
  • Don't re-run a search you already delegated.
  • Prefer one well-scoped subagent over many main-thread file reads.
  • Reserve Opus for where a wrong answer is expensive.