claude-code-cold-tester

June 28, 2026 · View on GitHub

Test your docs with an AI that has zero context — like a real new hire.

A Claude Code skill that runs a document — onboarding guide, runbook, CONTRIBUTING, tutorial — through a separate model with a blank slate, and surfaces every place a newcomer would get stuck.

cold-tester demo

The problem

You can't test your own documentation. You wrote it, so you already know what every step means — you fill the gaps without noticing, and it reads perfectly. That's the curse of knowledge, and it's why "looks clear to me" docs strand real newcomers on step 3.

Your main AI agent has the same problem: it's been in the conversation, it knows the project, it'll glide past the same gaps you do.

The fix

Hand the doc to an AI that genuinely has no context — a separate process, a different (cheap) model, given only the document, a persona ("first-time contributor"), and a kickoff. It tries to actually do the task, step by step. Wherever it stalls, guesses, or writes "UNCLEAR — need to confirm" is a hole in your doc.

main agent (approver) ──doc + persona + kickoff──▶ isolated model (zero context)
       ▲                                                   │
       │            submissions + "where I got stuck"      │
       └───────────────────────────────────────────────────┘
   rules pass / conditional / reject per gate, lists the doc fixes

Your main agent stays the approver — it never plays the tester (it knows too much). It reads what the cold tester produced and judges each gate.

Setup

pip install openai pyyaml
export COLD_TESTER_API_KEY=sk-...                       # required
export COLD_TESTER_BASE_URL=https://api.openai.com/v1   # optional (default OpenAI)
export COLD_TESTER_MODEL=gpt-4o-mini                    # optional

Any OpenAI-compatible endpoint works — get a key from OpenAI, DeepSeek, Together, OpenRouter, or point COLD_TESTER_BASE_URL at a local server. Use a cheap model on purpose: the tester's value is being naive, not smart. A run is a handful of short completions, so it costs cents.

Verify your setup with a dry run — point it at any short doc; if it prints submission produced (...) for each gate and writes a report, you're set:

python3 cold_tester.py --docs README.md --persona "newcomer" \
  --task "follow the readme" --gates "setup" --output /tmp/check.md

Then drop the skill in:

git clone https://github.com/<you>/claude-code-cold-tester.git
cp -r claude-code-cold-tester/skills/cold-tester ~/.claude/skills/cold-tester

Run it

The script lives once at its install path; you point it at any repo with --base — you never copy it into the target project. Run it from anywhere:

python3 ~/.claude/skills/cold-tester/cold_tester.py \
  --base /path/to/the/repo/you/are/testing \
  --docs README.md,CONTRIBUTING.md \
  --persona "First-time contributor who has never seen this repo" \
  --task "Open your first pull request" \
  --kickoff "Set up the project and open a PR fixing a typo." \
  --gates "setup,make-change,open-pr" \
  --output results/run.md

--docs paths are resolved relative to --base, so the same installed script tests any project. With a config file (examples/contributing-test.yaml):

python3 ~/.claude/skills/cold-tester/cold_tester.py \
  --config ~/.claude/skills/cold-tester/examples/contributing-test.yaml \
  --base /path/to/repo

It produces a report: each gate has the cold tester's attempt and a blank verdict for the approver to fill (pass / conditional / reject), plus a summary of the doc fixes.

What "follows the doc" means here: the cold tester doesn't execute commands or touch your filesystem. It reasons through the document as a newcomer would and reports, step by step, where the instructions run out — what it would have to guess, look up, or give up on. The gaps it surfaces are real; the execution is simulated. That's deliberate: you want the reading comprehension test, not a sandbox.

Proof: it found holes in this README

Pointed at an earlier draft of this very README, the cold tester (running DeepSeek) flagged, among others:

"UNCLEAR — no instructions about virtual environment setup. I'll assume global pip is acceptable."

"I need an API key. UNCLEAR — I don't have one. Where do I get one? The document doesn't mention this."

"The repo URL has <you> as a placeholder. I don't know the actual GitHub org/user… The document doesn't have a verification step. How do I know if setup succeeded?"

Every one of those is real, and every one is invisible to the author — we knew where to get a key and what <you> meant, so we never saw the gaps. That's the entire point of the tool, demonstrated on itself.

Why it works

The highest-value finding is a gate the cold tester marked UNCLEAR that you'd have known how to do. That gap is invisible to you and to your context-rich main agent — it only shows up when someone with zero context tries to follow the doc literally. Onboarding guides, runbooks, SOPs, API quickstarts, tutorials: anywhere the author's knowledge silently fills the blanks, this makes the blanks visible.

This is honest about its scope: it's a focused harness around an OpenAI-compatible API, not magic. It needs a key, it costs a few cents per run, and the tester is only as naive as the persona you give it. That's the whole design — a cheap, blank-slate tester you can point at any "follow these steps" doc.

License

MIT — see LICENSE. Not affiliated with Anthropic or any model provider.