Benchmarks

The two methods agree. On Go, the hand-graded audit below read 29/30 = 96.7% for us and 29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges on the same repository, says 97.6% and 97.2%. Two unrelated methods, one person reading source and one type checker, land within about a point on both arms. That is the result we care about most, and it is not a competitive one. It also removes the n=30 ceiling: hand-grading caps a precision estimate at roughly ±13 points near a 95% rate, while an oracle grades every edge, so n becomes the size of the repository.

Where our own miss goes, decomposed rather than waved at. On syft without tests we miss 3,846 of the oracle's 7,898 edges: 44% of that miss is dynamic dispatch alone and a further 39% is dispatch with a closure at one end, and the two buckets overlap so neither is the whole gap. Interface dispatch is the ceiling and nobody in the comparison has cleared it, at 6.5 distinct possible targets per call site. Matching that recall means emitting six edges where one is right, which is the behaviour the precision table charges other tools for.

Do not compare recall across rows. It swings from 0.03 to 0.97, driven by how many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by tool quality. Only within-row comparisons carry meaning, and a pooled recall over these cells would be meaningless.

Further limits:

• Two languages, seven cells, five repositories. This is not a nine-language claim and must not be quoted as one. The hand-graded audit below is the nine-language number.
• A contradicted edge is very strong evidence, not proof. RTA is unsound under reflection and go:linkname, so a genuinely dynamic edge can land in that bucket. It applies to all five arms equally and the gaps are far too large to be explained by it, which is why the metric is named precision against the oracle rather than precision.
• Edges the oracle cannot speak about are charged to nobody and reported at full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and 16% to 46% on the TypeScript ones, where dependencies are not installed in the pinned corpus.
• A library has no main, so RTA has no roots; cobra is analysed through its test binaries only, and the two variants of a repository answer different questions, so report both or say which you used.
• Two oracle languages, and the programme stops at two. The other corpus languages either need a per-repository build that does not exist on the measurement machine, have no sound call-graph tool (Rust), or admit no oracle even in principle (Python, Ruby, PHP). The hand-graded audit below is the permanent method there, not a stopgap.
• The TypeScript with-tests variants are void and quoted nowhere. The pinned corpus installs no dependencies, so a third of call sites go unresolvable.
• The three newer competitors are priced on these two languages only, and none was ever entered into the hand-graded audit in either direction. That audit is a two-tool claim and this one a five-tool claim; neither transfers to the other's languages.
• Two of the five arms are read through an adapter we wrote, and both now beat us on precision somewhere, so the reading matters. Graphify tags 93% of its call edges INFERRED and we score all of them, the choice least favourable to us. code-review-graph is scored on resolved rows only, the choice most favourable to it. Both are argued in the benchmark and either can be recomputed from the artifacts.

Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol required, and the graded pre-registration including the two predictions that missed: graph/experiments/g4-oracle-anchored.

One repository goes clearly to them. On seastar CodeGraph reads 6/10 against our 5/10, the only repository in the audit, on any language, where they beat us on a clear margin. Our misses there are chained calls on an untyped receiver; they infer the callee's declared return type and validate against it, so a failed inference costs them an edge instead of buying them a wrong one. On aria2 both sides read 10/10 and they resolve 24,950 distinct call edges to our 9,486. Precision is not the only reading.

This audit was graded by us, on both sides. That is the strongest form of a weak thing, and it is exactly what the oracle section above exists to check. The two agree where both exist, to within about a point, but only the oracle has an answer key we did not produce.

All nine cells are measured at one commit, and the resolver has moved since. Every corpus repository's call population was diffed site by site against the current tip: three cells are byte-identical, six moved, and of the 280 graded rows twelve moved — every one of them a row graded wrong, with no row graded correct moving at all. Re-reading at the tip can therefore only raise the number, so 85.7% is a floor. The table is not redrawn here because a cell whose population moved needs a fresh proportional draw rather than a twelve-row patch. No cell was measured on the 0.44.0 release.

All 600 graded rows are published, one file per cell, each row carrying the call site, the declaration the tool bound it to, the verdict and the reason it was given. A script in the same directory rebuilds every table above from them and fails if it disagrees. 560 of those rows enter the pooled figures: C++ is read at 50 rows per side and enters at a seeded 30 so that it carries the same weight as every other language. Every row on the page now carries its own verdict.

Precision is one of two halves. We lead no recall cell against the oracle, and lose cross-file coverage on 15 of 35 repositories in the same bench. A precision win is a claim about the edges a tool draws, never about how many.

graph/experiments/g1-edge-precision/rows

Why this is not benchmark tuning. We first ran this and came last, at 0.228. We published that. The cause was a query-time gate discarding most candidates before ranking ever happened. Fixing that path is what moved the number, and it is a fix any user gets rather than a change shaped around these questions. The check on that claim is the split, and it points the right way:

Overfitting makes the unseen half score worse. Ours scores better, on both tools. CodeGraph, which nobody tuned against either half, scores the same on both to three decimal places, so the two halves are equally hard and the gap is about the tool rather than the questions. We do not quote a pooled 112-instance figure, though it is easy to compute and would be 0.835. Averaging the halves loses the only number that matters.

The cocoindex row is not from the same sitting, and it is printed rather than smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to 2026-08-06, same instances, same gold spans, same deterministic grading. Its n is 41 because one named instance never answered even when queried alone, so it is excluded rather than counted as a zero, which makes its row better than counting it would (0.361 against 0.353). Its placing was pre-registered before its index existed, including the prediction that it would come last.

This is retrieval, not task success. It says we find the right files, not that an agent using us writes better code.

Measured on repowise 081a59fa (between v0.37.0 and v0.38.0). Raw cells: rung8.

5 of the 48 questions are missing from every arm equally because the run hit an API usage cap. Paired comparisons are unaffected; the figures are over the 43 all six arms completed.

The saving grows with the work. Splitting at the median by how much the bare agent needed: easier half 27.2%, harder half 34.3%, correlation +0.379. Pre-computed structure replaces exploration, and harder questions contain more exploration to replace. That split is post-hoc and the pre-registered comparisons are the stronger evidence.

Second harness: 15 questions on Claude Code (claude-sonnet-5).

The "agent used it" column is the interesting one: under Claude Code most of these tools were barely called at all, code-review-graph never once and Graphify three times in fifteen. Nothing differed about the servers, questions or indexes between the two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has to go looking before it can call anything, and frequently never does. Treat that column as unstable, including our own 15 of 15. Reruns on later days returned 4 of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph. Whether the collapse is the model or the harness is inconclusive: the same 15 questions on Opus, against bands fixed before the run (12+ means the model, 6 or fewer means schema deferral, 7 to 11 inconclusive), came back at 7. That run's token column fails its own control (-9.3% when the tool was called against -10.7% when it was not), so no token figure is quoted from it for any tool including ours.

Third harness: qwen3:8b under Ollama via opencode, same 15 questions, same seed. Every cell called its tool, 15 of 15 on both rows.

Two repowise rows, never combined. get_answer writes its answer using a hosted model, so a row using it is not a local-only result. The second row switches it off and leaves only tools that run against the local index. We verified the restriction holds rather than assuming it: instructed directly and repeatedly to call get_answer, that agent could not reach it in any of its 15 cells. repowise roughly doubles the tokens fed in on a single step while cutting steps from 3.3 to 2.1 and halving what the model writes, which is why the win shows up as time: reading a large payload once is cheap on a GPU, generating text token-by-token across several rounds is not.

What this section will not claim:

• This is work saved, not quality. A blind judge scored every tool in the field, ours included, a fraction below the bare agent on the 48-question run, in a range of 0.04 to 0.25 points on a 10-point scale, all smaller than the 0.69 points by which this benchmark moves when rerun unchanged. No tool here measurably changed answer quality in either direction. Ours is at the low end of that band and we will say so if it becomes a real effect. "No significant difference" is not parity, and an equivalence claim needs a test we have not run. On the local run the full surface scored +1.32 against the bare agent, above the judge's measured 0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and removing the single best question drops it to +0.99; the local-only row is +0.20, a null. The defensible statement is narrower: the local-only configuration answers about as well as a bare agent in roughly half the wall clock, on hardware you already own.
• The quality columns cannot be compared across tables, because grading a model with a judge from its own family is a known bias, so the judges differ.
• Not a universal saving. One repository, one commit, one prompt. Three harnesses, which is more than most published numbers in this category, and still not a constant.
• Adoption is not a stable property of a tool. Whether an agent calls a codebase server at all depends more on the harness than on the server. Any adoption figure, ours included, is only meaningful with its harness and its date attached. Ordering across four instruments was Sonnet 3 to 4, Opus 7, Codex 15, local qwen3:8b 15.
• No dollar figure, and the control that retired it is the most useful thing we ran. code-review-graph, having never called its server once in 15 questions and carrying 28,118 extra characters of schema that should cost more, measured 43% cheaper than the bare agent. The cause is prompt caching: whichever arm runs first warms it. An arm's position in the cycle correlated -0.487 with its dollar cost. Output tokens are never cached and correlate +0.010, so those are what we publish. If you see a token claim anywhere that does not say whether it controls for cache state and arm ordering, ask about that first.

Raw data, every cell including failures: rung9 (48-question Codex) and rung6 (the 15-question runs). What producing them cost in runs, machine hours and API spend, plus the per-tool setup traps that were most of the real effort: head-to-head.

It is not better than raw file size at discrimination. LOC-only scores 0.742 against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt +0.134 (95% CI +0.080 to +0.198): the same discrimination as counting lines, much better at ordering a fixed review budget, and unlike a line count it says why.

The signal is weak among files of similar size. Holding size fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so it survives cleanly only in the largest quartile, and a positive control confirms that collapse is a real absence rather than too few positives to detect one. A prior-defects baseline also still beats us on Popt by 0.085 even while losing on AUC.

Our AUC edge over CodeScene is marginal, and so is our raw defect-density lead (16.9x against 14.2x, p = 0.65 on a heavy tail); the size-normalized version is the one that reaches significance and the one to cite.

Marker weights are calibrated against a real defect corpus, not hand-tuned: every file scored at a commit preceding the bug window so nothing leaks backward, and an L2-logistic fit with file size as an explicit control, so a marker only earns weight for defect lift beyond being big. Only 26 of the 49 detectors are permitted to move the defect number, because that is the number carrying these accuracy claims.

Reports: BENCHMARK_REPORT.md · COMPARISON_REPORT.md

The two tables have different denominators and must never be quoted against each other. The graph table is construction only: no documentation, no embeddings, no health pass. The index table is a full repowise init.

One caveat runs in our favour on the graph table and is stated rather than buried. Our graph is built in memory and discarded, so serialisation cost is excluded from our column, while CodeGraph writes a real SQLite index (115 MB on one repository). Our memory figures come from a subprocess arm built precisely so peak RSS could be read the way every competitor's already was.

CodeGraph's own headline speed claim is incremental re-sync, roughly 0.3s to fold one file into a 4,400-file project. That is unmeasured here and we expect to lose it.

Seconds alone is not a quality claim. CodeGraph produces more distinct call edges than we do on several repositories.

The corpus caps near 2,000 files, so genuinely large-repository behaviour is measured on the top 10 of 35 rather than at scale. Single machine, single OS: the memory ratios are unlikely to invert but absolute numbers will move.

A fitted build-cost curve for five tools across a 12x repository-size range, which nobody in this field had published, is in head-to-head. Our exponent is sublinear at 0.906, and we publish that as sublinear work rather than efficiency, because symbol density halves across the range. That is a product finding against us.

Full per-repository tables: graph/experiments/g6-build-cost.

There is a comparable tool and we have not run it. RTK does this and essentially only this. It is also the tool whose advertised 60 to 90% came back 7.6% more expensive under JetBrains' rerun. So this table is a before-and-after of our own output, which is the weaker design. Only a paired run would settle it.

The two 0% rows are the net-positive guard working: distill never inflates small output. One run per command on one repository, so these are point measurements. Reduction is also not comprehension: the bytes removed are measured, the evidence they were safe to remove is narrower.

Full guide: docs/agent/DISTILL.md

Lead with the pooled figure. Pooled is sum over sum, so it weights each commit by the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit where get_context returns 40 tokens contributes a huge ratio that counts equally against one saving a hundred thousand. That is how a 35x becomes a 209x in a press release.

Raw CSV: token_efficiency_flask30

• Every retrieval number here is Python or Go. The graph sections are wider, at nine languages hand-graded and two compiler-graded, and they are the only sections that are. A JavaScript/TypeScript corpus (mui/material-ui, six tools, a 12x size range) is built and half graded, but its sealed half is unrun and nothing from it is quoted here. Publishing the development half is what that split exists to prevent, so the row arrives when the sealed half is evaluated, once.
• We index code files only, which depresses our own coverage on repositories with documentation. Deliberate: on doc-heavy repositories the docs outweigh the code, and indexing them polluted the index and degraded retrieval for the code questions the tool exists to answer. On the JavaScript corpus above, 21% of gold files are .md or .json, unreachable to us by construction rather than by ranking. cocoindex makes the opposite trade, is the only arm of six to retrieve any documentation gold, and pays for it on code gold.
• The agent-loop section is one repository, django/django at one commit, which is in every model's training data, and it measures single-session questions of roughly four to nine turns each. Nothing on this page measures a long multi-hour engineering task, and we will not imply a number for one.
• Command-output compression has no head-to-head and one row that needs re-measuring.