Live evidence suite

task-suite.json is the pre-registered cross-client task set. It covers command recovery, repeated reads, noisy search, cross-session recall, cross-project transfer, staleness, harmful-findings quarantine, active-model harvesting, a no-benefit control, and long-session/compaction behavior.

Copy runners.example.json outside the repository and replace every model, version, and baseline-isolation placeholder with the exact local CLI values. Do not weaken the baseline: it must load no Token Optimizer MCP server, hooks, plugin, rules, or user instructions.

First inspect the counterbalanced schedule without launching a model:

npm run eval:evidence:plan

Then run one client/model cohort:

node scripts/run-evidence-eval.mjs \
  --runner C:/private/token-optimizer-runners.json \
  --client codex \
  --model gpt-5.6-sol \
  --repetitions 8

Each arm receives a fresh fixture copy, graph, and hook-state directory. The aggregate result is a schema-v2 eval-run record. Use --include-transcript only when raw model output is appropriate to retain locally.

Published, redacted live cohorts:

• Claude Code 2.1.225 / Claude Sonnet 5 / command recovery / five pairs

Cross-client mistake transfer

handoff-suite.json tests a stricter chain: one live producer must encounter a dead end, capture the durable lesson itself, and a fresh consumer must receive that exact finding before its first tool call. Empty, natural, oracle, irrelevant, and stale graph arms all use the full integration and identical post-producer workspace snapshots.

Inspect the pre-registered cyclic order without launching a model:

npm run eval:handoff:plan

Run a direction with explicit, locally isolated runner profiles:

node scripts/run-handoff-eval.mjs \
  --runner C:/private/token-optimizer-runners.json \
  --producer codex \
  --consumer claude-code \
  --producer-model gpt-5.6-sol \
  --consumer-model claude-sonnet-5 \
  --repetitions 5

The schema-v3 artifact stores hashes and behavioral outcomes by default, not transcripts. A missing or semantically wrong natural harvest remains a failed natural arm and is never replaced with an oracle finding.