Methodology

Every rule here was paid for. Most of them exist because an earlier measurement produced a number that looked fine and was not.

The measurement rules

1. Resolved edges is the result. A coverage percentage never is. A change that raises coverage and lowers precision is a regression wearing a win's clothes. G2 exists to reproduce someone else's coverage metric, not to be scored on.

2. Fold to distinct (file, line, target) before comparing anything. Capture is not injective on either side. Our java.scm and ruby.scm mint a receiver-less call site for every member call in addition to the member site, so caffeine emits 110,740 call sites for 75,963 call expressions. CodeGraph likewise emits 39,079 raw calls edges on caffeine against 38,609 distinct. A comparison between one side's raw count and the other's distinct count is off by whatever the author needed.

3. Never mix the raw and distinct bases in one sentence. Report both side by side or commit to one. Every table in this directory labels which it is using.

4. A sampled precision rate already includes the wrong-edge classes you counted. Subtracting a known-bad class and applying the sampled rate double-counts. Pick one. Our audits use sampled rates alone.

5. Both arms in one process, behind a toggle. Never two checkouts. A second checkout admits an unrelated variable, and it has already cost one session's results.

6. A verdict inferred from a name is worthless. Precision rows are read from source: the call site with its imports and enclosing scope, then the target declaration. ambiguous is an honest verdict and some are expected.

7. Predict before you measure, from an instrument that shares no code with the thing being measured. A prediction that lands within ~10% is what makes a result trustworthy; one that misses by 3x means the mechanism is not understood yet, whatever the measurement says.

8. State recall on both denominators, including when it falls.

9. Refusing is a result. A mechanism that was built, gated on 15 repositories and then refused is a finding, and it gets written up like one.

10. Peer indexes under test-repos/*/.codegraph/ are frozen. Read-only, via a file:...?mode=ro URI, always. Published baselines reconcile against those exact bytes and a regeneration silently invalidates every prior number. New indexes for new experiments are built in a scratch copy, never in place.

11. Every repository is pinned, and the pin is captured as it is cloned. corpus/corpus.lock records a full 40-character SHA, remote and file count for every checkout the benchmark can touch, generated by corpus/capture_pins.py rather than typed. corpus/fetch.py materialises any subset at exactly those pins and refuses a directory sitting at a different commit rather than repairing it. Rule 10's frozen indexes live under those directories, and a helpful git checkout would invalidate every baseline silently. A rerun on a different pin is a different measurement and says so.

12. A cached artifact may supply coverage. It may never supply a cost. A competitor index depends only on (tool, tool_version, repo, pin) and not at all on our commit, so it is built once and restored afterwards -- otherwise thirty-five repositories times five arms is not affordable and the corpus stays at six. Two guarantees make that legitimate, and both are enforced rather than asserted. A restore reproduces every set the protocol exposes, byte for byte, which smoke.py checks by storing and restoring a real index. And a restored artifact carries from_cache with the date its cost was really measured, so a run using the cache is stamped publishable: false with that reason recorded; G6 and the determinism gate pass fresh=True, which bypasses the cache in both directions. No cost number that has ever been published came out of a cache.

13. No per-language claim from fewer than three repositories, and the three must be different kinds. Library, application, framework -- they stress different things, and three libraries is still n=1 about libraries. This rule exists because we nearly published a fact about zod as a fact about TypeScript: on the same arm, TypeScript reads 0.138 on zod and 0.573 on dub. Disagreement between the three is a finding, not a failed measurement, so G7 reports the spread with the repository named at each end rather than a mean, and a language below three repositories is printed as carrying no language-level claim instead of being averaged in with the rest.

14. Every control and every delta is reported per source language, not per repository. A repository is not a language. ktor lost 28 call edges between 2dcd2ac5 and 58576af0 and every one of them came from its 8 embedded .rs files -- its Kotlin-sourced edges were byte-identical across the pair. Read per repository that is "ktor changed", which invites the wrong explanation; read per source language it is "the Rust fix did exactly what it said and Kotlin was untouched", which is the truth. So a "control repository is byte-identical" claim means nothing on its own: at least one repository in this corpus is not monolingual, and the languages inside it move independently. Split every control and every before/after by the language of the calling file before quoting it.

Cost is measured on the graph, and only the graph

Our published indexing-time row in docs/BENCHMARKS.md §6 compares a full repowise index, which also generates documentation, computes health and builds embeddings, against tools that build a graph and stop. That is the right comparison for "how long until I can use this", and the wrong one for this directory.

G6 times graph construction alone on both sides: walk, parse, resolve, write edges. No documentation, no embeddings, no health pass. We expect to remain slower even so, and that row gets published at whatever it comes out at. A benchmark that only reports the columns its author wins is not evidence.

A cached artifact never becomes a cost number. Competitor artifacts are stored under artifacts/<arm>-<version>/<repo>-<pin8>/ and reused, because such an artifact depends only on (tool, tool_version, repo, pin) and on none of our commits. Reuse is what makes re-running our own column cheap. But every timed path asks for a real build, which bypasses the cache in both directions, and a restored artifact reports the cost measured when it was actually built, flagged from_cache with that date. No published second has ever come out of a cache.

Traps that have already bitten

• A dirty working tree. One audit's first pass measured another session's uncommitted work and read zod 15% high. Check git status, prefer a detached worktree at a named commit.
• A guard scoped to a path that does not exist passes everything. The dirty-tree check above was, for a time, applied with the package directory as the repository root, so it asked git about a packages/packages that does not exist, got an empty answer, and read it as clean. Every run it gated was stamped publishable without the check doing anything. The trees involved were in fact clean, so no published number is known to be wrong. The guarantee was simply absent, which is the whole failure mode the guard exists to prevent. provenance now refuses a scope matching no files, and --allow-dirty cannot suppress that. Applies past the graph bench: any check narrowed to a subtree should be tested by making it fail.
• The measurement worktree is shared with whatever is already running. Our arms import from a detached worktree, so moving it mid-run changes what is being measured, silently, with no trace in the result. Never check out another commit there while a run is in flight.
• The peer's unresolved_refs table is not calls-only. It carries a reference_kind column with references, imports, instantiates, extends and decorates. Filtering to 'calls' is the difference between a real recall figure and one that is roughly 5x too pessimistic.
• The peer's nodes.language is the caller's language. caffeine's index carries kotlin, python and c callers. Restrict deliberately, and say which way.
• Repositories that look like a mis-resolution and are not. gitleaks vendors its own regexp wrapper that its tests import instead of the stdlib. caffeine has a guava/ compatibility subtree mirroring Guava names on purpose. zod has parallel v3/v4/mini trees plus a zod4 alias to a published npm package. Read the imports before grading a row wrong.
• The benchmark-tuning trap. A mechanism can score well on a benchmark repo for a reason that will not reproduce for a user. The live example: inferring a decorator's type by reading the decorator's own declaration works on celery only because celery is both the framework and the repository under test. In an application that imports celery, that declaration is external and unreadable. Any framework-shaped result is run against repositories that use the framework without vendoring it before it is quoted.
• Two heavy parses at once will exhaust memory on the measurement machine. Parse serially.

How results are reported

The measurement rules above stop a number being wrong. These stop a page of correct numbers reading as an advertisement, which is a separate failure and a more likely one, because the author is also the tool's author.

1. Every headline number ships with three things: the mechanism that explains it, the caveat that runs against us, and what we did not measure. A number with only the first is marketing. The clearest example on this page is build cost: we are faster to build the graph, we do not persist an index while CodeGraph writes 115 MB of SQLite, and we have not measured incremental re-sync, which is their actual claim and the one we expect to lose.

2. Reproduce a competitor's metric faithfully before criticising it. G2 is CodeGraph's own coverage metric, computed on their index by our script, and we went looking for the reading that does reproduce their published table rather than stopping at "it does not". A critique that cannot first reproduce the thing it attacks is not evidence.

3. Never claim a win on a metric that is saturated. Five of six repositories sit above 0.9 on the either-direction reading. A lead there is noise, and reporting one would be the same sin this page documents in others.

4. Ties are called ties. Overlapping 95% intervals are never written up as a lead, and this is enforced in code rather than remembered: stats.overlaps is the gate, and stats.diff_significant is the sharper test where overlap is too blunt. gitleaks and caffeine are ties. The pooled precision gap is not.

5. Untestable is not a pass. An arm that resolved nothing cannot be fooled by a decoy, cannot lose an edge to a rename, and cannot drop one at a shadowed site. G5 reports vacuous and never counts it as a pass, because the alternative ranks a tool that resolves nothing above one that resolves almost everything.

6. State what our own normalisation did to a competitor's number. Two choices in this benchmark move a competitor's figures substantially: we filter code-review-graph's CALLS rows to those that resolve, because it stores unresolved callees in the same table, and we run Graphify --code-only because no other arm calls a language model. Both are argued in arms/, and the counts either way are in every result file, so a reader who disagrees can recompute rather than take our word.

7. Name the cells where we are worst, in the same table as the wins. zod, Java precision, M3 shadowing and incremental re-sync are all ours to lose and all appear by name.

8. A correction goes in the body, not a changelog. Two of this benchmark's results moved against us on the day they were checked properly. The caffeine coverage cell and the claim that we lose on build cost. Both are in the README's result sections where a first-time reader meets them, not in a footnote. This is the single most useful thing on the page for a skeptical reader, because it is evidence the harness is not tuned toward a conclusion.

9. If a table has us ahead in every column, treat that as a bug in the table. No honest comparison of four real tools comes out clean, and a sweep is far more likely to mean a denominator is wrong, a competitor is misconfigured, or a metric is measuring what we happen to do. The caffeine row looked like our clearest win until the denominator was checked.

Known weaknesses we publish rather than wait to be caught on

• Java is our least precise language and our largest edge count. Roughly 67% edge precision on caffeine, against roughly 52,000 edges. The cause is known: our java.scm mints a receiver-less twin of every member call, which then resolves by bare name at roughly 60% precision. Removing it costs caffeine 35% of its resolved calls, so it stands. About 17,000 of caffeine's edges are suspect. This is the single largest correctness exposure in our graph and it is stated here rather than discovered by a reader.
• Overload blindness on both sides. Roughly a third of the wrong rows on each side are a call bound by name with no regard for argument type.
• Chained receivers are unresolved everywhere. The one wrong edge that both tools produce on gitleaks is the same shape.
• Grading choice worth disclosing. "Right class, wrong overload" is graded wrong alongside class-level errors. Grading it separately moves the peer's Ocelot and caffeine cells up by 3 and 4 rows. Stating the choice beats letting a reader find it.
• zod is our worst cell against any arm. We find 269 symbol-bearing TypeScript files against a 401-file population; Graphify finds 401, code-review-graph 320, CodeGraph 291. Three of four arms beat us there, and unlike caffeine there is no denominator story that explains it away.
• We fail G5's shadowing mutation. After a local identifier shadows an imported package, we still bind the call through it. CodeGraph fails identically, but "nobody passes" is not "we pass".
• We have not measured incremental re-sync, which is CodeGraph's headline claim, roughly 0.3s to fold one saved file into a 4,400-file project. We expect to lose it. Quoting our cold-build result against that claim would be answering a question nobody asked.
• Our own central argument currently indicts us too. This benchmark exists because nobody publishes a graph-correctness number against an external oracle. G4 is the experiment that would fix that, and it is designed and not built, so today we are in the same position as everyone we are describing.