Agent task comparison runner v1

Status: implemented execution adapter and deterministic fixture-mode demonstration. No comparative model observations. The paid 18-trial paired pilot (task x available-lane x repetition) is issue #105 and is not run by this contract.

Audience: benchmark operators and reviewers of the Agent Task Comparison v1 contracts.

This closes the execution-adapter gap issue #104 identifies: scripts/agent-task-comparison.py validates and summarizes externally supplied evidence but does not itself run anything. This contract adds exactly one thing beside it: scripts/agent-task-comparison-runner.py, an external-harness adapter that turns a (task, available lane, trial) selection into a real run, a typed event ledger, and a ledger record — without modifying a single byte of the existing plan/task/manifest/ledger/ observation/audit/report contracts it reuses unchanged.

Non-goals

This contract does not:

• redefine the ledger, observation, audit, or report schemas — it imports scripts/agent-task-comparison.py as a library and calls its existing make_plan_from_loaded, trial_from_loaded, make_observation, and make_audit functions verbatim;
• run or claim any real coding-agent trial. Every ledger this repository's test suite produces is generated by the deterministic fixture backend described below, and every artifact and field it writes says so;
• produce or admit a timing claim. This host runs many concurrent builds during this round of work; any wall-clock number measured on it is contended and not admissible evidence. The fixture backend's validation_wall_ms / review_wall_ms values are fixed constants read from a checked-in scripted transcript, not measurements of anything that ran on this host.

The pluggable runner interface

scripts/agent-task-comparison-runner.py run takes the same --manifest --task --lane --trial selectors as the existing trial command, plus --runner {fixture,live}:

python3 scripts/agent-task-comparison-runner.py run \
  --task signature-migration-v1 \
  --lane semaprax-graph-operational \
  --trial 1 \
  --runner fixture \
  --fixture-script benchmarks/agent-task-comparison-v1/fixture-runner/signature-migration-v1.semaprax-graph-operational.json \
  --evidence-dir benchmarks/agent-task-comparison-v1/evidence/<run-id>

Owned-signature compiler-evidence probe

owned-signature-migration-v1 has an additional independent oracle. It only accepts its semantic, ownership, runtime, and review rows when the caller supplies an explicit compiled semaprax binary with --compiler. The runner records that binary's SHA-256 before and after bounded check, test, run, and project graph commands for both the immutable baseline and the isolated candidate. A timeout, spawn failure, output-cap failure, nonzero status, or binary drift fails the relevant row.

Run both available fixture arms into a unique local log directory with this one-shot command. Set binary to the exact already-built compiler image being audited; this command does not build the compiler or contact a model/provider.

binary=/absolute/path/to/target/private/debug/semaprax
run_id=".agent-logs/owned-signature-probe-$(uuidgen)"
for lane in semaprax-graph-operational semaprax-source-first; do
  python3 scripts/agent-task-comparison-runner.py run \
    --task owned-signature-migration-v1 \
    --lane "$lane" \
    --trial 1 \
    --runner fixture \
    --fixture-script "benchmarks/agent-task-comparison-v1/fixture-runner/owned-signature-migration-v1.$lane.json" \
    --evidence-dir "$run_id/$lane" \
    --compiler "$binary"
done

For cleanup evidence, the runner copies the exact src/core.spx bytes into a separate operating-system scratch directory and appends only a synthetic entry function there. It asks graph for that projection and records the exported cleanup plans for benchmark.owned.select and benchmark.owned.call, bound to the original core-byte digest. It never writes the immutable baseline or candidate while collecting evidence. The projection proves those core functions' compiler plan shape; it is not a claim that the projection is the project entry closure. The separately captured project graph and baseline / candidate command results supply the project-level evidence.

This is an offline fixture-harness result. It is not a live coding-agent pilot, a blinded human-review timing result, a provider/model observation, or a comparative-performance claim.

A backend is anything that can turn a task/lane/trial selection into an ordered action stream (model usage, context presentation, tool calls, failed attempts, stale detection/recovery, validation/review intervals, human interventions) against an isolated sandbox copy of the task fixture. Two backends are wired:

• fixture replays a checked-in, scripted transcript (semaprax.agent-task-comparison-fixture-runner.v1, under benchmarks/agent-task-comparison-v1/fixture-runner/) deterministically: no randomness, no wall-clock measurement, no network access, no model or tool credentials. This is the only backend this round of work exercises, per the coordinator's no-model-spending scope for this issue.

• live is wired to the same pipeline but refuses to place a provider call under any circumstance in this repository state. --runner live without --provider-config fails closed immediately. With --provider-config (a JSON object naming provider, model, api_key_env, and budget_ceiling_usd) it still refuses unless invoked with --dry-run, in which case it validates the configuration and prints exactly what it would invoke, with network_calls_made: 0. Without --dry-run it raises:

  HUMAN_BLOCKED: approved live-model budget for the #105 pilot. This build
  accepts --runner live only with --dry-run; it never places a real provider
  call.
  

  Adding a real provider call is out of scope here and requires the operator-supplied credentials and spending ceiling issue #105 owns.

A third-party backend can be added by implementing the same shape the fixture backend has: given a task's independent-acceptance-bearing sandbox directory and the frozen task prompt/setup/fixture bytes (never the task's acceptance rubric, drift patch, or manifest — see below), return the ordered action stream. The runner script, not the backend, is responsible for containment, drift application, ledger assembly, and calling the existing unmodified observation/audit validators.

Protected state: what the backend never sees, and what happens if it tries

Issue #104 requires that "an agent under evaluation must not be able to edit its own acceptance." This is enforced at three independent points, so that losing any one of them does not silently remove the guarantee:

1. The backend is never given a path to protected state. create_sandbox copies only the task's fixture_files bytes (source files and semaprax.toml) into a fresh, ephemeral directory under the OS temp directory. The task JSON (which carries the acceptance rubric and review protocol), the manifest, the drift patch, and the evidence directory this script itself writes the ledger into are never copied into the sandbox and their paths are never passed to the backend.
2. Every backend-declared write is containment-checked before it touches disk. resolve_write_target rejects an absolute path, a .. segment, and any path that walks through an existing symlink, and requires the final resolved parent directory to stay inside the sandbox root. A fixture script (or a future live backend) that tries to point a write at ../../../benchmarks/agent-task-comparison-v1/tasks/signature-migration.json, at an absolute path, or through a symlinked escape hatch is rejected with RunnerFailure before any byte is written, and the target file is provably unchanged (scripts/test-agent-task-comparison-runner.py's ProtectionTests compares its SHA-256 before and after the rejected attempt).
3. Acceptance is never read back from the backend. The ledger's acceptance rows come only from independent_acceptance, which inspects the resulting sandbox files against each task's acceptance criteria (one checker per task, keyed by task id) after the backend has finished. A fixture script may carry an arbitrary claimed_outcome / claimed_acceptance field — run_fixture_backend and build_ledger never read either key, and ProtectionTests .test_candidate_backend_cannot_claim_its_own_acceptance runs a "lying" script that leaves the signature unmigrated while claiming every criterion passed, and asserts the independently computed ledger reports signature: failed regardless.

Once the harness finalizes the ledger and its transcript artifact, it chmods both read-only as defense in depth (ProtectionTests.test_candidate_cannot_write_the_finalized_ledger_after_the_run_completes asserts a same-user write attempt raises PermissionError and leaves the file unchanged). This is process-level, same-user defense in depth, not a substitute for OS-level sandboxing (a container or VM); a live backend running an actual coding agent as a subprocess needs that additional isolation layer, which is out of scope for the fixture-only demonstration required this round and is noted as a limitation below.

Drift injection

For a task with drift_injection != "none" (stale-signature-recovery-v1), the fixture script marks the harness-recognized point with one {"kind": "identifying_inspection"} action. run_fixture_backend:

• reads the task's actual committed drift patch bytes (already authenticated against the manifest's recorded SHA-256), not a hardcoded copy of its effect;
• applies its single hunk via apply_unified_diff_single_hunk to the sandbox's src/core.spx the moment the marker is reached, before any later action in the script runs;
• rejects a script that raises the marker zero times for a drift task, more than once, or that raises it at all for a task with no drift patch.

Because both lanes' sandboxes are built from the same fixture_files bytes recorded in the plan, and each lane's own fixture script independently triggers the same one-time drift application, ZeroLaneAndDriftTests .test_drift_is_applied_exactly_once_and_both_lanes_start_from_identical _fixture_bytes asserts the two lanes' resolved trial contracts bind byte-identical starting fixture bytes and that each lane's run applied the drift exactly once.

Zero lane and cross-trial reuse

The runner does not reimplement lane availability or trial-range checking: run() calls the existing, unmodified trial_from_loaded, which already raises Failure for zero-graph-native (availability: external_unrun) and for a trial number outside the manifest's repetition count. ZeroLaneAndDriftTests.test_zero_lane_is_refused exercises this through the runner's own entry point. Each run() invocation builds one fresh, ephemeral sandbox scoped to its exact task/lane/trial selection and deletes it in a finally block; sandboxes are never reused or shared across selections, so one trial cannot observe another's candidate state.

Determinism

scripts/test-agent-task-comparison-runner.py's DeterminismTests run the same (task, lane, trial) selection twice into two separate evidence directories and assert the resulting ledger.json and observation.json files are byte-identical, and separately recompute the twelve required metrics directly from the checked-in fixture script (without calling any runner internals) and assert they equal the derived observation's values.

Evidence layout and why nothing here is checked in as a static golden file

--evidence-dir must resolve inside the repository tree, because agent-task-comparison.py's observation/audit commands (reused unmodified) resolve the ledger path against the repository root and recompute the plan's repository_head as git rev-parse HEAD at the moment they run. A ledger's plan_sha256 therefore only revalidates while the working tree's HEAD still equals the commit it was generated against — this is the existing contract's deliberate "source drift fails closed" behavior, not something this runner changes. Consequently this runner never checks in a permanent ledger/observation/audit snapshot as a golden fixture: doing so would go stale the moment a later commit advances HEAD, and a stale-but-present file that nobody re-generates is worse than no file. Instead:

• benchmarks/agent-task-comparison-v1/fixture-runner/*.json (the scripted, recorded-response transcripts) are checked in, because they do not depend on HEAD at all.
• benchmarks/agent-task-comparison-v1/evidence/ is created on demand by run() and is gitignored (its own checked-in .gitignore keeps everything under it out of version control except that file itself); regenerate it at the commit you want to audit.

What this demonstrates versus issue #105

This round exercises the fixture backend across both available lanes for two of the three tasks (signature-migration-v1 and stale-signature-recovery-v1, trial 1), which is enough to prove the full select -> run -> capture -> ledger -> observation -> audit path, drift injection, and lane pairing end to end. It does not attempt the complete paired report (which requires the full 3-task x 2-lane x 3-trial matrix), and it makes no model observation, superiority claim, or the Zero-lane comparison — those remain exactly as unavailable/unclaimed as the existing contract already states. Producing the real 18-trial paired pilot with a live coding agent is issue #105. The --runner live --dry-run path above prepares a plan; it does not establish an implemented live observation path.

Explicit OpenCode tuple transport

scripts/opencode-agent-task-pilot.py adds a separate, opt-in macOS transport for one source-first tuple using the fixed free OpenCode profile. It requires an explicit absolute --semaprax executable, copies that compiler into private host state, and binds the copy to streamed SHA-256 checks of both original and copied bytes. The child sees the private compiler through PATH.

The actual run and export use the same saved sandbox-exec profile as the preflight probes. The profile denies reads beneath the current checkout, original Git checkout, evidence parent, and user home; writes are confined to the disposable candidate and private host state. Probes exercise direct and symlink access, permitted candidate reads, and denied outside writes. Both processes use private home/config/data/cache/temp directories. Bounded output, deadlines, and process-group cleanup apply to run and export.

Every result remains explicitly ineligible: blinded review, complete ledger metric mapping, the other lane, and the full 18-trial experiment are unfinished. The ten offline tests in scripts/test-opencode-agent-task-pilot.py exercise local stubs through the actual wrapper, including copied compiler resolution; they are not live-provider or productivity evidence. No real pilot tuple was run to validate this transport slice.

Criterion mapping

See the "Criterion-by-criterion mapping" section of the issue #104 closing report for how each required test/failure case above maps to its exact evidence (test name, command, and observed result).