Atlas Benchmarks

August 13, 2026 · View on GitHub

Reproducible A/B benchmark of Atlas — indexing throughput and agent-flow token efficiency — driven by the opencode CLI running the opencode-go/deepseek-v4-flash model.

Status: v4 prompt methodology run landed (2026-08-13). Numbers below are hand-curated from the raw JSONL run logs under packages/atlas/benchmarks/results/.

Methodology (in brief)

Atlas is a semantic code intelligence layer (SQLite knowledge graph + tree-sitter extraction + optional embeddings). Two claims we want to measure honestly:

  1. Indexing is cheap enough to be a default. Files/sec and MB/sec on real repos, cold cache.
  2. An agent with atlas is more token-efficient — fewer tokens, fewer tool calls, less wall time, lower cost — than the same agent without atlas.

Our vision: token efficiency is the norm. The with arm should not just answer correctly; it should do so with materially less context churn, because one atlas_explore call returns verbatim source + call graph + blast radius that would otherwise take a long grep → read loop.

To measure (2) we run opencode run --format json per question, twice per arm. The only variable between arms is the MCP config: one arm wires in the atlas MCP server, the other gets an empty MCP config. Both arms keep the built-in Read, Grep, Glob, Bash and Task tools, so the "without" arm has a real fallback path.

Arm prompts (v4):

  • with — atlas_explore is the mandatory first exploration call; Grep/Read/Bash are allowed afterward only for gap-filling. This measures atlas's ceiling while keeping a realistic fallback.
  • without — explicitly forbidden from delegating to a task/sub-agent, so it must do its own Read/Grep/Bash (no hidden sub-agent effort).

The harness is a small PowerShell + Node pipeline (index → run A/B → judge) — see packages/atlas/benchmarks/scripts/.

Corpus

RepoLangFrameworkSize
microsoft/vscodeTypeScriptElectronXL
excalidraw/excalidrawTypeScriptReactM
django/djangoPythonDjangoL
tokio-rs/tokioRustAsync runtimeL
square/okhttpKotlin+JavaHTTP clientM
gin-gonic/ginGoHTTP routerS
Alamofire/AlamofireSwiftHTTP clientM

Full URLs and SHAs in packages/atlas/benchmarks/corpus.json. One canonical architecture question per repo — the full list lives in ground-truth.json.

Repos are cloned to $env:TEMP\atlas-bench\repos — a temp directory that 1-index.ps1 -Clean reclaims when you're done.

Reproduce

# 0) One-time: build atlas
cd packages/atlas && npm run build

# 1) Clone the corpus and time atlas init on each
cd benchmarks/scripts
.\1-index.ps1                     # add -Fresh for a truly cold run
                                  # add -Only <id,id> to subset

# 2) Run the two agent arms per question (opencode engine)
.\2-agent-ab.ps1 -Engine opencode -Model opencode-go/deepseek-v4-flash

# 3) Score answers (raw metrics land in results/judged-summary.json)
node .\3-judge.mjs

# 4) Reclaim disk (deletes the temp repo dir; keeps results/)
.\1-index.ps1 -Clean

See packages/atlas/benchmarks/README.md for the full authoring guide (ground-truth format, environment neutralization, cost estimate).

Machine

  • Intel Core i7-9750H @ 2.60 GHz, 7.9 GB RAM
  • Windows 11 Home (10.0.26200)
  • Node v24.11.0
  • opencode 1.18.18, model opencode-go/deepseek-v4-flash
  • Atlas: packages/atlas/dist/mcp/server-entry.js (direct MCP mode, ATLAS_NO_DAEMON=1)

Interpretation notes

  • Token efficiency is the win — and it is consistent. Across all 7 repos the atlas arm cut all-tokens by 60–86%, tool calls by 63–92%, and reads/greps almost entirely (the graph lookup replaces the grep → read loop). Cost fell 40–71% in 6 of 7 repos (vscode tied).
  • Best case — django, okhttp, tokio: atlas used just 2–3 atlas_explore calls per run at 80–86% fewer tokens and 57–71% lower cost.
  • One atlas_explore replaces the grep → read loop. Median reads dropped from 8–21 to 0–2; median greps from 2–10 to 0.

Numbers below are per-repo medians over 4 valid runs per arm, taken from results/judged-summary.json on 2026-08-13.

Headline

Across 7 repos with valid paired arms (28 valid with-atlas runs, 28 valid without-atlas runs), atlas cut file reads by 95%, greps by 100%, fresh input tokens by 43%, billable tokens by 45%, all tokens by 76%, and wall time by 50%.

Indexing throughput

Cold-cache index of each corpus repo from a clean .atlas/ (or warm re-index if noted). One process, Windows 11, Node 24.

RepoLang / FrameworkSizeFilesSource MBWall sFiles/sMB/sDB MBPeak RSS MBCold?
vscodeTypeScript / ElectronXL12693148.682.35590.265.48371.3361.2warm
excalidrawTypeScript / ReactM6647.310.8811.28.9338.4360.7warm
djangoPython / DjangoL304120.261.03027.420.17130.7760.8warm
tokioRust / Async runtimeL7935.591.6483.53.4131.6860.9warm
okhttpKotlin+Java / HTTP clientM6464.361.6405.12.7351.0560.8warm
ginGo / HTTP routerS990.681.663.00.436.3660.7warm
alamofireSwift / HTTP clientM1082.141.199.01.9612.7961.0warm

Agent A/B - with atlas MCP vs without any MCP

Same opencode CLI, same model, same prompt. The only variable is whether the atlas MCP server is wired in (atlas config vs empty MCP config). Both arms keep built-in Read / Grep / Bash / Task so the without-arm has a real fallback. The with-arm is prompted atlas-first (v4); the without-arm is forbidden from delegating to a sub-agent. Numbers below are per-repo medians over valid runs only; infrastructure/rate/session-limit failures are excluded from comparisons and listed in the final column.

Format: with / without in each cell. All tokens includes cache reads for context-volume inspection; Fresh input excludes cache reads.

RepoValid runs w/woReads w/woGreps w/woAtlas calls w/woTotal calls w/woFresh input w/woCache read w/woBillable tokens w/woAll tokens w/woWall w/woCost w/woInvalid runs
alamofire4 / 40 / 12.50 / 5.54.5 / 05.5 / 17.534585 / 49816145984 / 43360036020.5 / 52626183644 / 492695.537s / 59s$0.003 / $0.005
django4 / 40 / 15.50 / 6.52 / 02 / 24.521301 / 3807455744 / 37452822201.5 / 4167377945.5 / 41620122s / 79s$0.002 / $0.004
excalidraw4 / 41 / 210 / 4.54 / 05.5 / 33.537133.5 / 74765.5172992 / 109676838168.5 / 79281209863.5 / 1190781.534s / 107s$0.003 / $0.008
gin4 / 40.5 / 4.50 / 22 / 02.5 / 8.522673.5 / 3914357216 / 20524823312 / 40676.583599 / 243109.514s / 30s$0.002 / $0.003
okhttp4 / 40 / 8.50 / 3.52 / 02 / 21.520118.5 / 4551044416 / 36844820928 / 48529.572842 / 416977.514s / 62s$0.002 / $0.004
tokio4 / 41 / 190 / 32.5 / 03.5 / 2629729 / 90286.587232 / 92640030839 / 93842.5119398 / 1023165.530s / 92s$0.003 / $0.009
vscode4 / 42 / 12.50 / 103.5 / 09 / 24.548447 / 39978.5256960 / 62086449878.5 / 42929.5306838.5 / 675589.5148s / 170s$0.004 / $0.004

Appendix — per-question call traces

Every raw run log (ab-<repo>-{with,without}-<run>.jsonl) is preserved under packages/atlas/benchmarks/results/ for anyone who wants to audit the exact tool sequence. These files are gitignored by default — commit aggregated summaries only.