Model and Harness Comparisons

T3MP3ST model comparisons are built from committed benchmark receipts. The matrix command aggregates existing artifacts; it never starts containers, calls a model, or contacts a target.

Reproduce the committed comparison

npm run bench:model-matrix
npm run verify:model-matrix

The committed Cybench matrix compares Claude Opus 4.7 and 4.8 on the same 15 task IDs, live-tools harness label, artifact schema, and pass@1 policy. Its JSON contains every source artifact path; its Markdown output provides the human-readable table.

Outcome categories stay separate:

• success: the committed strict oracle verdict detected the expected flag;
• failure: the benchmark ran but did not satisfy the oracle;
• abstention: the artifact explicitly records a refusal, abstention, or no-action outcome;
• infrastructure_error: the run failed because of a timeout, API, spawn, or equivalent harness error;
• skipped: the expected artifact is unavailable.

These are system results: model, provider, harness, runtime, tools, target class, and attempt policy all matter. Historical dates and model versions also differ. Do not interpret the table as an isolated model-quality ranking.

Run arbitrary model variants

The refusal-frontier harness accepts any comma-separated OpenRouter model IDs without source edits:

# Offline: prints the exact shared corpus and call count; spends no API quota.
node scripts/refusal-frontier.mjs \
  --models anthropic/model-a,openai/model-b \
  --classes crypto,memory \
  --n 3 \
  --dry-run

# API-backed and cost-bearing: requires OPENROUTER_API_KEY.
node scripts/refusal-frontier.mjs \
  --models anthropic/model-a,openai/model-b \
  --classes crypto,memory \
  --n 3 \
  --at comparison-2026-07

Every selected model receives the same classes, five-rung corpus, sample count, judge, and concurrency settings. JSON output retains raw responses and judge rationales; Markdown output renders model-by-model and cross-model deltas.

Use the fields in CONTRIBUTION_RECEIPTS when adding new results. Never commit credentials or private target data, and never present live external runs without a target-specific authorization receipt.