Reproducing the bakeoff

Every number in the repowise OSS docs/BENCHMARKS.md comes from a run in this repository. This directory says how to re-run each one, what it costs, how long it takes, and which ones a third party can actually reproduce, because some of them need credentials or paid API access that we cannot hand over.

Raw output for every run is under results/bakeoff_2026_08/, tracked in git and kept permanently, including the runs we later invalidated. Agent stdout logs are the one exclusion: 49 MB of transcript that no rerun is graded against.

What each claim costs to reproduce

Section numbers below refer to the OSS docs/BENCHMARKS.md.

Claim	Where	Reproducible by a third party?	Cost	Wall clock
Retrieval, file coverage (§1)	results/bakeoff_2026_08/rung8/	Yes, with an OpenAI key	Embeddings only, a few dollars	~12 h serial, ~1.5 h at 8 workers
Agent loop, Codex, 48 questions (§2)	results/bakeoff_2026_08/rung9/	Yes, with a Codex subscription	~$18	~5 h
Agent loop, Claude Code, 15 questions (§2)	results/bakeoff_2026_08/rung6/	Yes, with a Claude subscription	~$19 for the stratified draw	~3 h
Agent loop, local qwen3:8b (§2)	results/bakeoff_2026_08/rung6/	Yes, and this one needs no account at all	$0, Ollama on your own GPU	~2 h
Loading one commit's context (§3)	results/bakeoff_2026_08/rung1/	Yes, no credentials	$0	minutes
Distill compression (§4)	OSS repo	Yes, no credentials	$0	minutes
Code health defect prediction (§5)	health-defect/	Yes for the cross-project and PROMISE arms	$0	~1 h
CodeScene head to head (§5)	health-defect/	Partly. Needs a CodeScene account	licence cost	~1 h
Indexing time (§6)	results/bakeoff_2026_08/rung4/	Yes, no credentials for the competitor arms; ours needs an OpenAI key with prose on	$0 to a few dollars	~1 h
Cross-function performance bugs	perf-detection/	Partly. The clippy arm was never run end to end	$0	~2 h

The honest summary: §3, §4 and most of §5 need nothing but a checkout. §1 and §6 need an embeddings key. The hosted §2 rows need a paid coding-agent subscription, and no amount of packaging on our side changes that. The local qwen3:8b row is the one third party reproduction that needs neither, which is part of why it was worth running.

Before any run

Two rules that produced wrong numbers here before we caught them, both of which will produce wrong numbers for you too:

1. No timed build runs while another process pool is alive. Contention inflated one of our own index timings by 65%. scripts/ has the preflight check that enforces this.
2. Prove the arm was alive before recording a zero. A dead server, a wrong tool name, or a broken output extractor all score exactly like a bad tool. The harness records isError, the served tool list, and response size per call so that a zero is always distinguishable from a failure. Do not remove those fields.

Retrieval (§1)

ContextBench, 112 instances, split into a 70-instance dev half and a 42-instance sealed half. The split is pinned by instance id in results/bakeoff_2026_08/dev_test_split.json and generated by make_split.py. It was committed before any tuning work started, which is the property that makes the sealed half worth anything. Check the commit date yourself rather than taking our word.

Arms and their tool allowlists are in configs/arms.yaml. Every arm is given its full advertised tool surface. This rule exists because we found we had handicapped Graphify to 1 of the 10 tools it serves, Serena to 3 of 29, and code-review-graph to 1 of 30, while our own arm ran a surface we had picked. An arm you handicapped is not an arm you measured, and the rule applies to us too.

Grading is deterministic against ContextBench gold spans. No LLM judge is involved anywhere in this number.

Agent loop, cost and adoption (§2)

Pre-registration: configs/layerb_stratified_django.PREREGISTRATION.md, committed 83d970b before the run. A pre-registration nobody can date is worth nothing, so the commit hash and its timestamp are part of the artifact.

Config: configs/layerb_stratified_django.yaml. Runner: harness/run_experiment.py. Adoption is computed by harness/served_surface.py and harness/check_tool_calling.py; per-slice reporting by harness/report_by_shape.py.

Every arm receives a byte-identical prompt. The bare-agent control is verified free of the operator's own hooks by harness/env_isolation_probe.py, which carries a positive control, because a flag whose name says it isolates the environment is not evidence that it did.

Judge agreement is measured, not assumed, by harness/judge_agreement.py. The noise floor on django, measured 2026-08-05, is 0.69 points of disagreement between graders on the same answers, which is larger than every per-arm quality effect in the run. That is why we publish the work-saved result and not a quality result. An earlier 0.46 was quoted here and in several result files for months and is superseded; anything still citing 0.46, or Go's 1.235, predates that measurement. results/bakeoff_2026_08/rung6/ carries the repeats it came from.

One consequence binds every future quality run and is worth stating here, because it is the thing most likely to waste someone's money: a paired test is governed by the paired-delta standard deviation, measured at 2.23 on one arm, not by the judge's repeat noise. Sizing a run off 0.69 understates the required n by roughly 10x. A 0.50-point effect needs about 156 pairs, not 15.

Indexing time (§6)

results/bakeoff_2026_08/rung4/ holds the install and index matrix, with the self-reported index statistics for every arm in its own vocabulary.

Node and edge counts are not comparable across arms and no ratio should be computed from them: each tool defines a node differently. The column is there for completeness, not for scoring. The comparable figure is wall clock, and the column that belongs beside it is what each tool built in that time.

Index repowise for a benchmark with all of these flags, every time:

--no-prose --embedder openai --max-file-pages 0 --no-workspace \
  --no-editor-setup --yes

plus REPOWISE_SKIP_EDITOR_SETUP=1 and DO_NOT_TRACK=1 in the environment. There is one global repowise MCP key, and an unguarded init will repoint the editor config of whoever runs it.

What we will not claim

• Retrieval results are retrieval, not task success. Finding the right files is not writing better code, and we do not have a number for the second one.
• The quality column in §2 sits inside the judge's own noise. Failing to find a difference is not parity, and an equivalence claim needs a TOST, which has not been run.
• Every agent run is on django/django or pallets/flask at one commit, both of which are in every model's training data.
• Python and Go only. There is no TypeScript or JavaScript row anywhere.