run/

Zero-dependency Node.js (≥20) scripts that talk to the OpenRouter API. Everything is designed to be idempotent and resumable: interrupt anything, re-run the same command, and it continues where it left off. All scripts are pure ES modules; shared code is in lib.js.

Setup

cp .env.example .env    # fill in OPENROUTER_API_KEY (PI provides it)

Workflow (in order)

Step	Command	What it does
1	npm run fetch-models	Snapshot the OpenRouter catalog → data/models-catalog.json. Re-run before every main run (pricing/availability drift).
2	npm run assess-models	Filter usable models, guess families, estimate cost → config/models.selected.json + console report. Then review that file by hand (see config/README.md, rule 3).
3	npm run pilot:dry	Print the pilot plan (request counts per model), no API calls.
4	npm run pilot	Execute the pilot → data/runs/pilot-01/. Re-run to retry failures.
5	node run/validate-run.js --run-id pilot-01	Completeness/failure report.
6	(stats) npm run stats:all then read results/pilot-report.md	Go/no-go decision before spending on the main run.
7	npm run experiment:dry, then npm run experiment	Main run → data/runs/main-01/.
8	node run/validate-run.js --run-id main-01	Validate, retry until ≥95% complete.

Key mechanics (read before modifying)

• Cell identity. Every request is one repetition of a cell (model × task × language × temperature × rep index). Its key is sha1(model|task|lang|temp|rep|prompts-major-version). Successful responses are appended to responses.jsonl; on restart, keys already present are skipped. This is what makes runs resumable and lets multiple partial invocations merge cleanly.
• Failures are not data. After max_retries with exponential backoff, a failure is logged to failures.jsonl and the cell stays pending — the next invocation retries it. Failures never enter analysis.
• Design freeze. The runner stores SHA-256 hashes of prompts.json and run.config.json in the run's manifest.json and refuses to resume if they changed. New design ⇒ new --run-id.
• Budget guards. --dry-run prints the plan; --limit N caps API calls per invocation (useful for a first cautious execution); frontier-priced models get expensive_reps_factor × reps in the main run.
• Metadata. Each response records UTC timestamp, latency, serving provider (OpenRouter routes to multiple upstream providers — this is an analytical variable, see paper1 robustness section), reported model string, token usage, and reported cost.

Rate limits / etiquette

Concurrency is capped in config/run.config.json (request.concurrency). 429s are retried with backoff automatically. If a specific provider keeps failing, lower concurrency rather than raising retries.

Adding a script

Keep the conventions: ESM, parseArgs from lib.js, no npm dependencies, every output file carries a generated_utc/manifest, console output states the next step.