YYLO Benchmark

yylo.dev · YYLO CLI · YYLO Ledger · npm

YYLO Benchmark provides two deliberately separate evaluation lanes:

• Isolated v2 (default): task prompts and ordinary Workflow Runner YAML execute in private fresh-repository attempt workspaces.
• Governed workflow: explicitly authorized production-touching workflows execute through a reviewed boundary with selected-step model overlays, typed exclusive locks, per-step envelopes, and blinded per-step judging.

The governed lane does not weaken or add production authority to isolated v2.

• Package: @yylo/benchmark
• CLI: yylo-benchmark (also delegated unchanged by yy benchmark)
• Source: yylo-dev/yylo-benchmark

Boundaries

Benchmark owns case normalization, attempt isolation, retained evidence, evaluator provenance, recovery, doctor, and reports. The selected harness owns command/workflow interpretation, provider and model resolution, and process/session execution.

Model selectors are opaque. Benchmark has no provider allowlist, credential store, paid-command classifier, execution grant, reservation, or USD ceiling. Candidate and judge costs remain separate evidence; unavailable cost is never converted to zero. Running a plan grants no production or external authority.

Install and inspect

Node.js 20.10 or newer is required. Pin the release selected by your operator:

npm install --global '@yylo/benchmark@VERSION'
yylo-benchmark --version
yylo-benchmark --help
yylo-benchmark init --stdout

These commands do not dispatch a candidate or judge. Publication and global installation are separate release-owner operations.

Configuration

Create yylo-benchmark.config.json with yylo-benchmark init, then replace placeholders with project-approved profiles. The v2 shape is:

{
  "schema_version": "yylo_benchmark_config.v2",
  "yylo_version": "INSTALLED_YYLO_VERSION",
  "workspace": {
    "attempts_root": ".yylo-benchmark/attempts",
    "registry_root": ".yylo-benchmark/registry"
  },
  "default_candidate_harness": "candidate",
  "harnesses": {
    "candidate": {
      "kind": "yylo_pi",
      "prompt": "Complete the configured benchmark case."
    }
  },
  "default_evaluators": [],
  "evaluators": {}
}

Supported harness kinds are yylo_pi, workflow_runner, and project-approved command adapters. Candidate and LLM-judge profiles select harnesses independently.

Evaluator profiles support deterministic commands and configurable LLM judges. The immutable plan binds both the initially selected profiles and the complete evaluator catalog available for later generation-only re-evaluation. Judge configuration binds prompt, rubric, selected evidence, byte limit, blinded or visible identity, single/reference/pairwise mode, settings, repetition/aggregation, and strict JSON or legacy VERDICT: PASS|FAIL parsing.

Keep workspace and registry roots ignored and private. The registry must be outside every candidate repository.

A legacy juno_benchmark_config.v1 file belongs to the governed-workflow lane; changing only its schema string is not a migration. Inspect configuration without mutation:

yylo-benchmark workflow migrate-config --input yylo-benchmark.config.json

When the input is isolated v2, pass an explicit unused --output path to create a governed template for review. Existing files are never overwritten.

Governed production workflows

Use the explicit namespace for a tracked project workflow and policy sidecar, including paths under .juno_task:

yylo-benchmark --config yylo-benchmark.config.json workflow setup
yylo-benchmark --config yylo-benchmark.config.json workflow readiness \
  --models zai/glm-alpha,zai/glm-beta
yylo-benchmark --config yylo-benchmark.config.json workflow plan \
  --workflow .juno_task/workflows/daily_product_ops.yaml \
  --steps-file .juno_task/specs/benchmark/daily-ops-policy.yaml \
  --steps first,second,third \
  --models zai/glm-alpha,zai/glm-beta \
  --attempts 1 --output governed-plan.json --dry-run
yylo-benchmark --config yylo-benchmark.config.json workflow run \
  --plan governed-plan.json \
  --steps-file .juno_task/specs/benchmark/daily-ops-policy.yaml --dry-run

Planning and dry-run report dispatch_count: 0. Planning preserves canonical selected-step order and expands model × attempt × selected step. It hash-binds workflow and policy bytes/semantics, variables, selected scope, compiled per-model workflow bytes, source tree, installed YYLO version, and reviewed boundary identity. The compiler injects the exact selector only at recognized yy pi dispatches; the trusted boundary owns exactly one --execution-envelope transport and reconciles requested identity against observed child evidence.

Live workflow run requires the exact hash-pinned boundary and consumer-owned credentials. Credentials remain in child-process memory and must not be written to plans or evidence. Production resources execute under typed exclusive locks in strict plan order. Completed missing/malformed envelopes are durable harness failures; only genuinely unknown effects require manual recovery.

yylo-benchmark workflow recover --plan governed-plan.json --steps-file POLICY
yylo-benchmark workflow rejudge --plan governed-plan.json --steps-file POLICY
yylo-benchmark workflow doctor --plan governed-plan.json --steps-file POLICY
yylo-benchmark workflow report --plan governed-plan.json

Recovery reuses retained terminals, rejudge dispatches no candidate, doctor verifies the retained chain, and report derives per-step/per-model outcomes. Paid/provider and production execution always requires separate consumer authorization; package tests use synthetic boundaries only.

Plan

Both case kinds produce yylo_benchmark_experiment_plan.v2 and nested yylo_benchmark_attempt_plan.v2 objects:

yylo-benchmark plan \
  --task benchmark/task.md \
  --models vendor-a/model-1,vendor-b/model-2 \
  --attempts 2 \
  --output task-plan.json

yylo-benchmark plan \
  --workflow .juno_task/workflows/evaluate.yaml \
  --models vendor-a/model-1,vendor-b/model-2 \
  --controlled-model-variable candidate_model \
  --var run_date=2026-08-31 \
  --output workflow-plan.json

Workflow YAML is hash-bound and passed unchanged to the configured Workflow Runner. String variables remain literal; object, array, boolean, numeric, and null variables are forwarded as canonical JSON so runner argv remains lossless and plan-bound. Benchmark does not parse command admission, reject ordinary cwd/environment/executable/managed-agent fields, or rewrite model arguments. With --controlled-model-variable, the report classifies the matrix as model_only; otherwise it is an agent_system comparison.

Planning is always zero-dispatch. --dry-run remains an explicit compatibility affirmation.

Run and recover

yylo-benchmark run --plan workflow-plan.json --dry-run
yylo-benchmark run --plan workflow-plan.json
yylo-benchmark recover --plan workflow-plan.json
yylo-benchmark doctor --plan workflow-plan.json
yylo-benchmark report --plan workflow-plan.json

Every candidate receives a dedicated fresh repository plus private home, temp, cache, and config roots. Configured default workspace-root names are logical namespace inputs: candidate roots resolve to separate per-attempt OS-temporary locations, while the private registry resolves to a source-bound host state location outside the source repository. Candidate environments discard inherited values and PATH entries that disclose source, controller, registry, sibling-control, task, Git-routing, or credential handles. Default candidate subprocesses additionally run behind a selective filesystem sandbox that denies those protected roots (sandbox-exec on macOS or Bubblewrap on Linux); an unsupported or unprovisioned host fails closed before dispatch. The snapshot has a private .git database with no remotes, alternates, worktree links, future refs/objects, controller route, hidden graders, reference solutions, sibling paths, or registry path. Receipts distinguish that selective boundary from custom trusted-host workspaces and do not claim a container boundary.

Intent is durable before candidate or judge dispatch. After candidate execution and before terminal publication, Benchmark retains a deterministic post-execution manifest covering Git HEAD/tree/refs/index/status and every worktree file. Intentional edits, additions, staging, and commits are therefore accepted as the bound candidate result; later repository drift is rejected. Recovery reuses hash-verified terminals. Retained state is rejected unless its state, plan, attempt, initial workspace receipt, post-execution manifest, candidate harness/model, terminal, evidence, evaluator configuration, and evaluation IDs form one exact linkage. Reuse, recovery, doctor, report, and re-evaluation verify that complete chain. An attempt directory without terminal state is ambiguous and remains manual; it is not blindly dispatched again. Doctor and report fail closed unless every attempt in the immutable plan has the complete retained chain; zero or partial experiments are never reported as complete.

Candidate and evaluator subprocesses run in isolated POSIX process groups. A timeout records measured wall time only after bounded TERM grace, process-group KILL, direct-child close/reap, and confirmed process-group disappearance within an explicit cleanup bound, including when descendants ignore SIGTERM. Failure to confirm cleanup fails closed rather than publishing terminal timeout truth. Windows is rejected before dispatch because this package does not provide an equivalent job-object/tree-termination guarantee.

Re-evaluation

yylo-benchmark regrade --plan workflow-plan.json --profile checks-v2
yylo-benchmark rejudge --plan workflow-plan.json --profile judge-v2

Regrade and rejudge consume retained AttemptEvidence; they do not dispatch a candidate. The immutable plan binds the stable evaluator profile configuration; each repeated operation under that same profile ID derives and appends exactly the next generation, rejects same/non-monotonic replay, and aggregates deterministically with retained applicable records. Existing records are immutable, and historical required deterministic correctness/safety failures remain authoritative. A required deterministic correctness or safety failure cannot be overridden by judge prose. Missing or malformed deterministic protocol fields, malformed judge output, timeout, privacy failure, missing identity/session, and evaluator-harness failure produce an invalid infrastructure evaluation with unknown quality.

Evidence and reports

New durable objects use yylo_version and retain:

• requested, resolved, and observed candidate/evaluator identities;
• initial workspace receipt plus post-execution repository manifest hashes linked through terminal, evidence, and state;
• candidate truth separately from harness and evaluator validity;
• candidate and judge cost completeness separately;
• evaluator profile, prompt, rubric, output, generation, and provenance hashes;
• evidence/evaluation IDs behind every aggregate;
• valid resolved rate, invalidity, comparison classification, and generations;
• authoritative runtime evidence in milliseconds: per-attempt wall time with evidence ID/timestamps, per-evaluator measured time with evaluation ID/generation, and candidate/evaluator/combined aggregates.

Runtime remains separate from candidate_cost and judge_cost; no duration is inferred from token, billing, judge prose, or command-harness protocol claims. The launcher overwrites command-harness timing fields with its measured start, end, and elapsed wall time before evidence persistence.

The library exports a bounded hash-verifying projection for immutable v1 objects containing juno_version. It never rewrites historical bytes. Historical Daily Ops discrepancies are documented in docs/historical-daily-ops-evidence-inventory.md and are not v2 scoring oracles.

Delegation

When compatible packages are installed independently, YYLO delegates argv, cwd, streams, signals, and exit status:

yy benchmark --help
yy benchmark plan --task benchmark/task.md --models vendor/model --output plan.json

The delegated and standalone commands use the same Benchmark executable and v2 contracts.

Development

npm ci
npm test
npm run typecheck
npm run build
npm pack --dry-run

Tests and packed acceptance use local deterministic harnesses. They do not make paid/live provider calls, mutate production, publish, install globally, push, or deploy.