Running your own experiments

This folder is a measurement rig, not a library. The probes in probes/ ask a typed oracle real questions against real data and write raw rows; separate report scripts judge them. Everything here is set up so you can point the same machinery at your question and your data.

Read FINDINGS.md first for what has already been measured — it carries the recommended configuration and a list of retracted claims. CLAUDE.md carries the summary of what TypeSafe is. For the short version to hand someone else, there is a three-minute walkthrough at When a number beats a sentence.

Setup

Node 20.6+ (the scripts use node --env-file), then:

npm install
cp .env.example .env     # then fill it in

.env needs:

variable	needed for	notes
TYPESAFE_KEY	every Jev arm	from console.typesafe.ai/settings/keys. TYPESAFE_API_KEY also works
ANTHROPIC_KEY	the Haiku baseline arms only	ANTHROPIC_API_KEY also works
ANTHROPIC_WORKSPACE_ID	only with an org-scoped Anthropic key	a workspace-scoped key must not set it

You can run the whole Jev half with TYPESAFE_KEY alone. Skip the Anthropic key until you actually want a baseline to compare against.

Reproduce what's already here

npm run dataset          # build the binary corpus -> .sandboxes/episodes.json
npm run run              # arm A (Jev) + arm B (Haiku via `claude -p`)
npm run run:api          # arm C (Haiku via the API — the honest baseline)
npm run report           # accuracy, confidence separation, calibration, cost

npm run run:repeat       # 5 independent runs of arm A
npm run stability        # per-item churn across those runs

npm run wording          # the broad/narrow x Choice/Noul 2x2
npm run wording:report

npm run ladder           # the 5-rung ordinal ladder -> .sandboxes/ladder.json
npm run score            # Jev Score, 3 runs
npm run score:haiku      # the same rubric asked of Haiku
npm run score:report

Run data is regenerable and lives gitignored in .sandboxes/. Nothing in probes/ judges anything — runners record verdicts, confidences, latencies and tokens; the *-report.mjs scripts do all the judging. Keep that split when you add your own: it means you can re-judge a run without paying for it again.

Knobs, without touching code

Every runner reads the same environment variables.

variable	default	what it does
PROBE_REPO	a local repo path	source repo for build-dataset / build-ladder
PROBE_FOREIGN	a local repo path	the "different project" rung of the ladder
PROBE_N	60	declarations sampled
PROBE_DIFF_CHARS	6000	per-diff truncation budget — the main cost lever
PROBE_SEED	20260916	deterministic sampling; same seed rebuilds the identical corpus
PROBE_OUT	.sandboxes/episodes.json	where the builder writes
PROBE_DATASET	.sandboxes/episodes.json	which corpus a runner reads
PROBE_TAG	—	suffixes result filenames, so a second corpus sits beside the first
PROBE_LIMIT	0 (all)	truncate the item list — use this for a smoke run
PROBE_CONCURRENCY	6	parallel in-flight requests
PROBE_HAIKU_VIA	cli	api reaches Haiku directly; cli spends the subscription

A second corpus, end to end:

PROBE_REPO=/path/to/other-repo PROBE_OUT=.sandboxes/episodes-two.json npm run dataset
PROBE_DATASET=.sandboxes/episodes-two.json PROBE_TAG=two npm run wording
PROBE_TAG=two npm run wording:report

Caveat: probes/report.mjs reads fixed filenames (results-typesafe.jsonl, results-haiku.jsonl, results-haiku-api.jsonl) and does not honour PROBE_TAG; only wording-report.mjs does. For a tagged run through run-arms.mjs, either move the files aside between corpora or teach report.mjs the tag.

Always smoke-test first: PROBE_LIMIT=6 npm run run costs cents and catches a malformed question before you pay for 173 items three times.

Running a new experiment

1. Get labels you didn't write

The single most load-bearing choice in this folder. Both corpora are labelled by construction, not by hand — a commit's message paired with its own diff is a match; paired with a different commit's diff it is not. No judgment under test could contaminate them, and there is no annotator to disagree with.

build-dataset.mjs also builds in a difficulty axis it never tells the model about (hard negatives share a file, easy ones don't). That axis is what makes confidence testable: a number that tracks a difficulty it was never shown is carrying information, and that argument does not depend on the labels being right.

If you can find a pairing, a fixture, or an existing outcome that generates your labels for free, use it. Hand-labelling is the fallback, not the default.

2. Ask everything in one call

The runners bundle every question into a single systemOne call. This is the intended shape, and it is measured: four questions cost 6% more input tokens than one, because the state dominates and ships once, and bundling perturbs a shared question's answers by under 2%. So ask the extra questions. wording.mjs is the template — four cells, one request, all seeing identical state.

3. One factor per comparison

Four claims in FINDINGS.md were retracted, and every one came from comparing against a convenient arm rather than a correct one. If you want to know whether the primitive matters, hold the wording fixed, or run the full factorial as wording.mjs does. A quantity measured on one arm's plumbing is a fact about that plumbing until a second arm confirms it.

4. Copy a probe, don't extend one

The probes are deliberately small and duplicative. To ask a new question:

1. Copy the closest runner — run-arms.mjs for a binary judgment, score-probe.mjs for an ordered one, wording.mjs for a factorial.
2. Replace the question constants at the top and the stateFor() shape. Keep the question text as a module-level constant; a question edited in place mid-run makes the results uninterpretable.
3. Write one JSONL row per item to .sandboxes/results-<name>-run<N>.jsonl, recording verdict, confidence, raw probabilities, ms and usage. Record everything; decide what matters later.
4. Write a matching *-report.mjs that reads those rows and prints the judgment.
5. Add both to package.json scripts.

5. Test three things, in this order

What the existing reports look for, and what a new one should:

1. Separation — mean confidence when right against when wrong. Flat means confidence-gating is dead and you are back to a bare classifier.
2. Difficulty — mean confidence on easy against hard items, on an axis the model was never shown.
3. Churn — run the same items 3-5 times. Where verdicts flip, and whether the flips concentrate in low-confidence items. This is the strongest test available, because it doesn't depend on your labels at all.

For an ordered judgment add adjacent-level AUC. A Score will merge two levels it cannot resolve and report mean scores that look perfectly monotone while doing it — there is no error and no warning. Confidence is what flags it.

6. Budget before you run

At the prices in FINDINGS.md, a 173-item corpus costs roughly $0.01 per Jev run and ~$0.40 per Haiku-API run; the claude -p arm is ~8x the API arm and exists only to show what CLI harness overhead costs. Multiply by repeats. PROBE_DIFF_CHARS is the lever if a corpus is too expensive — it truncates the diff, which is nearly all of the token count.

The Jev price constant is hardcoded in probes/report.mjs (JEV_PRICE) and was quoted, not published. Re-check it before quoting any cost figure.

Where results go

file	holds
FINDINGS.md	the standing summary — claims, evidence, retractions. Update this when a probe lands
artifact	When a number beats a sentence — the plain-language summary for a reader outside this folder
notes/YYYY-MM-DD-*.md	one entry per working session: what was measured and what it means
probes/	runners and reports
.sandboxes/	corpora and raw rows. Gitignored, regenerable, never a source of truth

Findings format is claim → evidence → confidence in the claim. Name the probe that backs a claim; where nothing does, say so.