Measurement tools

Read the consolidated results first. This directory contains measurement code; runtime code lives in src/.

One source of published numbers

results/index.json registers the two current v12 three-arm studies and preserves the historical v11 reference matrix, Claude regression and Codex smoke separately. Each entry has immutable results, a structural audit and SHA-256 hashes. Cost estimates are frozen with their historical basis. bun run bench:report verifies hashes and rebuilds docs/BENCHMARKS.md, docs/benchmark-report.html, docs/benchmark-results.json and the benchmark block of the root README.md without inference. It retains unknown costs and narrower later studies rather than selecting best percentages. A folder named final does not confer validity. bun run bench:screenshots renders the HTML figures into docs/assets/benchmarks/ with a local headless Chrome; set YOSHI_CHROME when the browser is not in a default location.

New runs do not replace these numbers automatically. Before registering a run: finish every planned trial; preserve failures and missing usage; run the protocol and fixture audit; record model, policy, source/fixture hashes and price basis; and explain whether it replaces a reference matrix or adds a separate study. Never pool different models, task definitions, policies or smoke/regression runs.

Explicit real-provider runs

These commands incur actual model usage and require the existing authenticated CLI and Gateway key. Choose a fresh output directory each time. They are not part of bun test or bun run bench:report.

# Claude reference workload. Frozen local source/research fixtures must exist.
bun run bench:claude --out .yoshi/runs/claude-example \
  --arms baseline,jev-context --trials 1 \
  --scenarios operations,agent-reports,evidence,history --judge-timeout-ms 45000
bun run bench:audit .yoshi/runs/claude-example
bun run bench:claude:report .yoshi/runs/claude-example .yoshi/runs/claude-example/report

# Codex uses the selected Claude manifest and the same frozen fixture workspace.
bun run codex:bench --out .yoshi/runs/codex-example --trials 1 \
  --fixtures .yoshi/focused-v11-validation-2026-09-17
bun run codex:report .yoshi/runs/codex-example .yoshi/runs/codex-example/report

Codex reporting also audits preserved envelopes and fixture hashes. Per-run reports stay inside their run directory. Both report tools require an explicit run, so an omitted argument cannot silently publish a superseded experiment.

The source snapshot is .yoshi/references/jev-eval-agent; reviewed historical fixtures are .yoshi/history-study/webmcp. bench:prepare prepares the source reference; history:prepare TRANSCRIPT DESTINATION extracts reviewed Claude history. Codex's selected fixture manifest also refers to an isolated temporary workspace: if it no longer exists, prepare a new Claude fixture run and pass its manifest directory. Do not replace a missing historical fixture with synthetic data under the same study name.

Diagnostic and historical tools

Command	Purpose
bun run codex:profile	Local historical character-distribution audit, not token savings
bun run jev:calibrate --phase development --out .yoshi/runs/calibration	Earlier tool-only classification experiment, not calibration of the current conversation policy
bun run jev:report DEVELOPMENT HELDOUT OUTPUT	Historical classification report
bun run summary	Mixed runtime log accounting; no baseline means no savings claim

Earlier methodology and results are in docs/archive. Old raw directories were relocated into .yoshi/archive/; its index.json maps original paths to current locations. Historical manifests and source snapshots retain their original provenance strings. The four selected raw studies and live fixture directories stay at their original paths so current reproduction and audits continue to work.

Sticky lifecycle and native comparison

The jev-sticky arm uses the current arrival lifecycle. jev-context explicitly selects the historical snapshot lifecycle. The native arm forwards the original history with Anthropic's clear_tool_uses_20250919 strategy (two tool uses kept, 1,000-token minimum clearing); it never uses Jev. The common relay records input_transformations, native context_management, cache creation/read usage and actual model identity. No missing cost or reasoning field is inferred.

session is a real Claude Code loop alternating four fixture reads and four local archive writes. Every arm gets a separate output directory. The existing exact JSON oracle grades required identifiers; the input fixture hash check remains mandatory. --padding-chars adds a disclosed synthetic irrelevant system archive to exercise long-context behavior; it is not a natural long project.

# Small-session diagnostic: deliberately bypass the production gate.
bun run bench:claude --out .yoshi/runs/sticky-session \
  --arms baseline,jev-sticky,native --trials 1 --scenarios session \
  --min-context-tokens 0 --native-trigger 1000 --judge-timeout-ms 45000
bun run bench:audit .yoshi/runs/sticky-session

# Production size gate, with explicit synthetic context padding.
bun run bench:claude --out .yoshi/runs/sticky-long \
  --arms baseline,jev-sticky,native --trials 1 --scenarios session \
  --padding-chars 190000 --judge-timeout-ms 45000

# Strict signed-thinking check: mismatch is an error, never silently dropped.
bun run bench:claude --out .yoshi/runs/sticky-fable \
  --arms baseline,jev-sticky,native --trials 1 --scenarios evidence \
  --model claude-fable-5-1 --effort high --strict-thinking true \
  --min-context-tokens 0 --native-trigger 1000 --judge-timeout-ms 45000

The audit verifies source-preserving authorized edits and compares every unchanged original prefix against the previous forwarded prefix. It also checks that sticky mode never enables thinking drops. Prefix comparisons with zero removals are vacuous for omission replay; report removal counts alongside them. Zero reported thinking drops proves transport acceptance only when the strict run includes signed blocks and actual prior omissions, and never proves the model used its reasoning. Each run freezes source hashes in its manifest. Keep all failed and invalid diagnostic runs, with their disposition, in the separate validation report.