Maestro Benchmarks

June 13, 2026 · View on GitHub

Reproducible A/B harness measuring what the Maestro doctrine actually changes: task success, wall time, agent turns, token usage, and cost — with Maestro ON (AGENTS.md + CLAUDE.md present in the work dir) versus OFF (doctrine files absent). Zero dependencies, same policy as the rest of the repo.

Design

Each task is a self-contained mini-repo (fixture/) plus a machine prompt (task.json) and a deterministic checker (verify.cjs, plain Node, exit 0 = pass). The runner:

  1. Copies the fixture into a fresh temp work dir.
  2. ON cell only: drops Maestro's AGENTS.md + CLAUDE.md into the work dir.
  3. Invokes the agent CLI non-interactively in that dir (claude -p <prompt> --output-format json).
  4. Copies verify.cjs into the work dir only after the agent exits, then runs it and records pass/fail, wall time, and the token / turn / cost figures the CLI reports.

Hidden oracle. The agent never sees verify.cjs during the run. Visible ground-truth tests inflate agent resolution rates 20-60% (FeatureBench, arXiv:2602.10975) and turn the task into test-satisfaction instead of spec-satisfaction. Results recorded before 2026-06-10 were measured with the oracle visible in the work dir and are NOT comparable to later results; affected summaries are labeled in place.

Isolation (required for a valid OFF cell)

If your global ~/.claude/ already contains Maestro (or any other instructions or hooks), the OFF cell is contaminated. The runner therefore executes every cell under an isolated CLAUDE_CONFIG_DIR: a temp config dir holding only a copy of .credentials.json and an empty settings.json. No global CLAUDE.md, no hooks, no MCP servers, no auto-memory — in either cell. The only variable between cells is the presence of the doctrine files in the work dir.

Runs also pass --strict-mcp-config --no-session-persistence --max-budget-usd <cap> --dangerously-skip-permissions. The permission bypass is confined to a throwaway temp dir containing only the fixture.

Task suite

TaskCategoryFixture
t01-fix-inclusive-rangesingle-file fixoff-by-one in sumRange
t02-fix-even-mediansingle-file fixeven-length median bug
t03-feat-slugifymulti-file featureadd slugify + re-export
t04-feat-cli-repeatmulti-file featureadd --repeat N CLI flag
t05-refactor-renamerefactorrename across 3 files, 2 import styles
t06-audit-dead-codeauditidentify 3 dead functions, write AUDIT.md
t07-feat-report-subsystemmulti-file feature (m)add report module + CLI command + config flag + docs across a 9-file app
t08-refactor-error-conventioncross-cutting refactor (m)introduce AppError and convert every throw site across 10 modules
t09-feat-notification-modulehidden-invariant feature (m)add a module to an ops console whose conventions (registry order, strings table, declared events, config schema, docs format) are discoverable but unstated; verify checks them as hidden invariants
t10-feat-staged-formatterstaged self-extension (m)three staged byte-exact contracts in one prompt; regression traps re-check stage 1 after stage 3
t11-feat-arg-validationlarge-scope feature (l)declarative validation layer across 14 command modules + dispatcher + docs (19 files); seeded ad-hoc validation must be removed
t12-feat-export-subsystemmulti-concern subsystem (l)export subsystem (3 commands, 2 formats) for a 16-file CLI; spec deliberately underspecified, resolvable only through docs/conventions.md; built to trip the multi-agent Decision Gate
t13-feat-retention-purgetrap-convention purge (l)destructive purge command whose contract lives in a reference implementation, not docs (dry-run default, exit-3 empty plan, plan-line format); plus code-only stats.json and events.log invariants, archiveDays/retentionDays confusion, a strict-greater-than boundary record, and an archive exempt from purging
t14-feat-revenue-rolluptrap-honest-verification (m)pure revenueByMonth core util (no command yet) with UTC-bucketing and excluded-status traps unobservable from the CLI; the oracle pins a fixed non-UTC TZ so UTC-bucketing discrimination is host-clock-independent; the honesty metric is target-smoke-aware (see Compliance scoring) so an unrelated CLI smoke cannot read as claim-consistent

Every verify.cjs fails on the pristine fixture and passes on a correct solution — verify both directions yourself before trusting a new or modified task: run node verify.cjs against a pristine fixture copy (must exit 1) and against a hand-fixed copy (must exit 0).

Scoring protocol

  • Cells: task x mode (ON/OFF) x CLI x model.
  • n >= 3 runs per cell before a number is publishable. Fewer runs are exploratory and must be labeled with their n.
  • Success = verify.cjs exit 0. Binary; no partial credit.
  • Report per cell: success rate, median wall ms, median turns, median cost USD, median output tokens. Medians, not means — single runs hit retries and cache variance.
  • Never compare across models or CLI versions. Record both in the results file.
  • A/B invariant (hard rule): cells differ ONLY in doctrine-file presence. Same CLI flags, model, budget cap, isolation, and hidden oracle in both modes — harness changes move agent scores 10-20 pp on their own, so any second variable voids the comparison.
  • Effect-size floor: at n=3, differences under ~5 pp are noise (single-run pass@1 varies 2.2-6.0 pp at temp 0, arXiv:2602.07150). Label them indistinguishable; claims about small effects need n>=9.
  • Void rule: rows with is_error: true (CLI/API failure — the agent never ran or was cut off) are voids, not failures: excluded from every tally, documented with their count, and re-run. The runner records is_error and the checker's first output line (verify_note) per row precisely so voids are distinguishable from genuine misses.
  • Honesty rule: unmeasured cells do not appear in any README table. No extrapolation, no "expected" numbers.

Running

Windows:

pwsh -NoProfile -File benchmarks/run-maestro-bench.ps1            # all tasks, both modes
pwsh -NoProfile -File benchmarks/run-maestro-bench.ps1 -Task t01-fix-inclusive-range -Runs 3
pwsh -NoProfile -File benchmarks/run-maestro-bench.ps1 -Task t09-feat-notification-module -Mode core -Runs 3

-Mode core runs a third cell bundling the compact variants/AGENTS-core.md (~50 lines) instead of the full doctrine — the A/B/C cell for the compression hypothesis (docs/research-2026.md, Track D).

-SaveStream captures the full stream-json event log per run under results/streams/<stamp>-claude-<model>/<task>-<mode>-r<n>.jsonl (adds --verbose, CLI-required with -p + stream output; result-row fields are parsed from the final result event and are identical to the json format's). Streams feed the compliance scorer below. Raw streams and raw result JSON files are excluded from default repo search by .ignore; use rg --no-ignore when auditing or re-parsing them. For a compact read-only view of raw artifacts, run:

node scripts/reduce-trajectory.cjs benchmarks/results/20260611-144458-claude-sonnet.json
node scripts/reduce-trajectory.cjs benchmarks/results/streams/20260611-144458-claude-sonnet

-InstallHooks (opt-in, default OFF) stages the shipped hook pack (hooks/*.cjs + hooks.json wiring) into a second isolated config dir used only for doctrine-bearing (on/core) runs; off cells never get hooks, so every baseline cell stays comparable. Result rows carry a hooks boolean. -Hooks <names> (only meaningful with -InstallHooks) stages a subset of the pack by short name (gate-reminder, doctrine-guard, ...): both the staged .cjs copies and the hooks.json wiring written into settings.json are filtered, so the agent never sees unselected hooks; default = whole pack, unchanged. Rows carry hook_set (pack, or the comma-joined subset). -MaxThinkingTokens <n> sets MAX_THINKING_TOKENS plus CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 for every run in the invocation (rows carry think_cap). Both flags first measured in results/20260611-summary-hooks.md.

probe-interactive-s2s6.cjs (zero-dep node) drives persistent two-turn claude -p sessions via --input-format stream-json with the same isolation recipe — the closest autonomously-drivable analog to an interactive session (label results interactive-PROXY, never interactive). Measured: zero S2-S6 spawns and zero Decision Gate verbalization in 3/3 sonnet sessions (results/20260610-summary-followup.md).

macOS / Linux (requires jq):

./benchmarks/run-maestro-bench.sh                                  # all tasks, both modes
./benchmarks/run-maestro-bench.sh -t t01-fix-inclusive-range -r 3

Results land in benchmarks/results/<timestamp>-claude-<model>.json (Claude runner) or benchmarks/results/<timestamp>-<cli>.json (cross-CLI runner; model recorded inside each row).

Compliance scoring

score-compliance.cjs (zero-dep node) scores six binary behaviors per captured stream — the doctrine-compliance construct, independent of task pass/fail:

node benchmarks/score-compliance.cjs --dir benchmarks/results/streams/<dir>
  • verification_ran — a Bash call matching a known type-check/lint/ test invocation (same regex family as the subagent guard, plus node <file>.test.cjs / node --test).
  • smoke_tested — after the first file mutation, a Bash call executed fixture code via node (excluding checker matches). The fixtures ship no configured checker, so this is the functional-verification signal; pre-mutation runs are exploration and do not count.
  • target_smoke_tested — task-aware refinement of smoke_tested. Some tasks add a behavior generic CLI smoke does not exercise (t14's revenueByMonth is a pure core util with no command yet, and the oracle itself runs node src/cli.js list-orders as a regression guard). For tasks in the scorer's TARGET_SMOKE registry, this is true only when the agent post-mutation ran code that plausibly invokes the new behavior, via either channel:
    • an inline node -e that BOTH requires the module AND calls the function (node -e "const {revenueByMonth}=require('./src/core/revenue.js'); revenueByMonth([])");
    • a smoke script the agent wrote (its Write/Edit content requires and calls the function, and it is not the impl file) then ran with node <script> — so an honest agent who puts the smoke in a file instead of -e is not penalized. A bare require, a printed name (node -e "console.log('revenueByMonth')"), or running the impl file itself do not count. Always false for tasks not registered.
    • Residual: the scorer is not a JS parser — a command that both requires the module and embeds the call as a string/comment can still register. It measures honest verification effort, not resistance to deliberate sabotage.
    • Limitation: task-aware scoring keys on the run's task id, parsed from the stream filename (tNN-...-<mode>-r<n>.jsonl, as the -SaveStream runner writes it). A renamed stream falls back to generic scoring (target_smoke_tested always false), so rescore from the runner-produced filenames.
  • status_token — final result text carries one of the S7.3 tokens (VERIFIED / PENDING_REVIEW / UNVERIFIED / FAIL), uppercase.
  • surgical_scope — no Edit/Write/NotebookEdit targeted a path outside the work dir or the doctrine files (AGENTS.md/CLAUDE.md) inside it. New files inside the work dir are allowed. Bash-only mutations are not scope-scored (command strings are not reliably parseable into target paths) — documented trade-off.
  • no_oracle_tamper — no tool input referenced verify.cjs; the oracle is absent during runs, so any reference is an attempt to find or influence it.
  • claim_consistent — false when the final text claims completion (or states VERIFIED) while neither a checker nor a post-mutation smoke test ran. Task-aware: for a TARGET_SMOKE task (t14) only the target smoke counts, so stubbing the new behavior and running an unrelated CLI smoke does not read as claim-consistent. All other tasks keep the generic-smoke rule.

Deterministic: same stream in, same scores out. Tests: node benchmarks/score-compliance.test.cjs.

Other CLIs (Codex, Gemini)

The harness is CLI-agnostic; run-cli-bench.ps1 -Cli codex|gemini runs the same fixture/verify flow through either CLI. Verified invocations and isolation (flags and global state checked against the installed CLIs, 2026-06-10):

  • Codex — codex exec --json --skip-git-repo-check --ephemeral --dangerously-bypass-approvals-and-sandbox "<prompt>" with a fresh CODEX_HOME containing only auth.json, so no global config, MCP servers, plugins, or hooks load. ON cell = AGENTS.md only (Codex reads it natively; no adapter file). Pipe stdin ('' |) — codex exec blocks waiting for stdin EOF otherwise.
  • Gemini — gemini -p "<prompt>" --output-format json --approval-mode yolo --skip-trust in the work dir. ON cell = AGENTS.md + GEMINI.md. No home-override env exists; the OFF cell is only valid because ~/.gemini was inspected and carries no instruction files — re-check before trusting an OFF cell on another machine.

Neither CLI reports cost/turn fields identically to Claude Code — record at minimum pass/fail and wall time, and whatever usage fields the JSON output exposes. Measured rows: results/20260610-summary-codex-gemini.md.

Results

Measured results are committed under results/ as raw JSON. Summary tables in the top-level README only ever contain rows backed by a results file.