Frozen walker instruments

September 3, 2026 · View on GitHub

Four instrument shapes. Each is reproducible against the named blob; none is pooled with another.

Shape 1—lane EXECUTION (Run 11's EN frame)

Walks how a lane's rails execute (gates, prohibitions, fail-routes, writes) from a frozen contract prompt. Common wrapper, byte-identical across every dispatch; only [SCENARIO BLOCK] differs.

Contract walk — READ THE SKILL AND REPORT WHAT YOU WOULD DO, IN WHAT ORDER. DO NOT ACTUALLY DO ANY OF IT.

## What to read (read all of it before answering)
<path to the SKILL.md under test>
If it points at another file and you judge that file necessary to decide the situation below, you MAY read it — then say which file you read and why.

## The situation to walk
[SCENARIO BLOCK]

## What to answer — answer as numbered items EXACTLY as listed. Do not add headings. Do not skip an item.
1. Does this skill's activation condition apply? — yes/no, and which sentence in SKILL.md made you decide that (quote it).
2. The sequence of steps you would take — a numbered list to the end; walk ONE path; if information is insufficient, follow the skill's own unknown/default rule and say where you got stuck.
3. Points where you must ask the user's permission first — how many, where, what each asks.
4. Points that are absolutely forbidden — list every prohibition SKILL.md states.
5. Where the output goes — which file, which folder.
6. If something goes wrong mid-way — what does the skill tell you to do.
7. Any grade / level / mode you must choose — if so, what did you choose and why.
8. What in SKILL.md you read and were NOT sure about — quote the ambiguous sentence, or write "none".

## RAILS
- Never actually do the work — no spawn, no write, no edit, no state-changing command. Read-and-report only.
- Do not search the web. Do not read any other repo.
- Answer from what you read in the file — if SKILL.md does not say it, write "SKILL.md does not say".
- Answer in English, except technical terms and text quoted from the file.

Full scenario blocks (4 lanes: routing-OFF, sensitive/never-down, escalate-up, route-to-self) and the per-run results live in results/2026-08-03-skill-determinism-walks.md §Run 11 (this benchmark's detail file consolidates the prior lab record—the canonical working copy is TheColliery/scratchpad/longrun/SKILL-VARIANCE-WALK.md, machine-local and not part of this public repo).

Shape 2—lane SELECTION (Run 12's shape)

Walks the decision UPSTREAM of execution—given a situation, which entry does the reader pick at all. 4 scenarios (one real, three controls), EN, ~1.1k chars, questions A (routing pick) / B (enumerate the trigger list) / C (free-text criticism). Not pinned at fire time (declared gap in the original record)—reconstructable from the scenario summaries in the detail file, not byte-reproducible the way Runs 13-15 are.

Shape 3—hook-injected coordination cue (Runs 13, 14, 17, 18)

Walks a UserPromptSubmit-injected cue, not a SKILL.md body—the load layer is two hook-emitted lines. 5 scenarios, EN, answer format A (which leads) / B (next action) / C (surface to user Y/N) / D (enumerate the stakes set) / E (self-reported uncertainty). Same 5 situations and answer format reused across all four waves—the clause text is the only thing edited between runs, which is what makes the runs like-for-like; it also means the situation set has now been iterated against three times by the same author (see RESULTS.md §Publishability verdict for what that bounds).

  • Run 13 (before carve): results/raw/run13-prompt-v1.txt, blob 3b4ff7080d7ccf935a32aa254bcc1c3bdd32b91c, 3,760 B.
  • Run 14 (after wave-1 carve, hook line A substituted, scenarios untouched): results/raw/run14-prompt-v1.txt, blob cd7c82405ffa9ad4c40ac9ade85ad471ce904d52, 3,999 B.
  • Run 17 (after wave-2 carve, both hook lines substituted—one-flock means CoalTipple's copy now carries the same clause too, so this is NOT like-for-like with Run 14 on that one axis): results/raw/run17-prompt-v1.txt, blob e82b0693ec2ea28998ecd8b93da6522474ee8cc4, 4,322 B. Clause walked: 594 ch, sha 23fed45403bb, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 18 (after wave-3 carve, both hook lines substituted, generated by reading the clause out of the live shipped conductor rather than hand-pasted): results/raw/run18-prompt-v1.txt, blob 70a3ecd9a33efb5f0d9b8ac8c0b855d74089b855, 4,408 B. Clause walked: 637 ch, sha 2416c04309ce, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 19 (wave-4 clause, generated by substituting the live shipped clause into both hook lines, three fail-loud guards asserted wave-4 present / wave-3 absent before any walker spawned): results/raw/run19-prompt-v1.txt, blob 93e42c03d5b2d60e2390c78a1411350d9aad6472, 4,504 B. Clause walked: 685 ch, sha 4a18bb6f8f03, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 20 (wave-5 clause, two additive phrases on top of wave 4, four fail-loud guards): results/raw/run20-prompt-v1.txt, blob 4da42a7f2275337b3b3db8ca993ac8e4373328d5, 4,842 B. Clause walked: 854 ch, sha1 e72fec15dffe, byte-identical across both conductors and both plugin/ dists at fire time. Reverted the same day (CoalBoard 2b99b46, CoalTipple 85f1312) back to the wave-4 clause—this prompt is retained as the frozen record of what was walked, not of what ships.

Shape 4—this benchmark's prose-index hypothesis test (Run 15)

Same shell as Shape 1 (8 numbered questions, read-only rails), scenario constructed so every branch of the target skill's own deterministic tier rubric is satisfied by the prompt's own wording—a clean test of whether the walker COMPUTES the branch rather than guessing it.

  • Run 15 (CoalMine rot-canary, prediction locked before firing): results/raw/run15-prompt-v1.txt, blob c9ccd10f9f644796d77f853b5006e58f95696437, 2,874 B.
  • Target walked: installed coalmine@coalmine 3.14.0 skills/rot-canary/SKILL.md, blob 57406d5a42f55474672bbbed3d31a62a86cadd59.

Reproducing a run

  1. Fetch the target blob at the pinned hash (the installed plugin cache, or git cat-file blob <hash> against the tool repo's history if the blob is a commit-reachable object there).
  2. Dispatch the frozen prompt to N fresh, memory-less, no-spawn leaves per tier (this record used blind-ic under Claude Code; any zero-context, zero-tool-beyond-read leaf qualifies).
  3. Score each rail's modal answer and membership per the formula in PREREGISTRATION.md.
  4. Walker transcripts are collected into the detail record, then destroyed (standing lab-hygiene rule)—only the frozen PROMPT and the scored TABLE persist; a stranger re-running this gets a fresh, independent reading rather than an answer key.

Scorer note (fifth instrument defect, Run 20, fixed)

A line-anchored scorer (matching on an S<n> label) silently drops any walker whose answer uses a different shape and reports the aggregate over the survivors—this happened five times across the series (Runs 17, 18, 20 caught; two more shapes seen in raw output but not mis-scored). Run 20's instance was the most dangerous: it dropped exactly the two weak-tier walkers who could not follow the text, which manufactures an apparent improvement on the rail that measures whether the weak tier follows the text at all—the direction opposite of a random drop. The replacement scorer is format-agnostic (order-of-appearance, not line anchors) and refuses to print an aggregate unless all N walkers yield exactly the expected number of answers per rail — a parse failure fails loud instead of silently shrinking the denominator. Use the replacement for any re-run; do not revert to line-anchoring for convenience.