PAMELA

Low-cost, large-scale behavioral measurement of LLMs through single-token output distributions, collected once via OpenRouter and analysed

Repository map

Folder	Content	Start with
config/	prompt battery (15 tasks × 4 languages), run configuration, curated model list	config/README.md
run/	Node.js data collection: catalog fetch, model assessment + cost estimate, resumable experiment runner, validation	run/README.md
data/	append-only raw responses (JSONL + manifests), derived normalized data	data/README.md
stats/	analysis pipeline: Node (normalize, distributions, JSD) + R (clustering, classification, ROC/EER)	stats/README.md
results/	machine-generated statistics and figures; pilot go/no-go report	results/README.md

Quick start

cp .env.example .env            # add OPENROUTER_API_KEY
npm run fetch-models            # snapshot the catalog
npm run assess-models           # filter models + cost estimate → review config/models.selected.json
npm run pilot:dry               # sanity-check the plan (no API calls)
npm run pilot                   # collect the pilot (resumable; re-run to retry failures)
node run/validate-run.js --run-id pilot-01
npm run stats:all               # full analysis pipeline
npm run stats:pilot-report      # → results/pilot-report.md = GO / NO-GO for the main run

Requirements: Node ≥ 20, R ≥ 4.0 (packages auto-installed by npm run stats:r-deps), no npm dependencies.

Principles (bind all future work)

• Reproducibility: every response carries the prompts-file hash, git commit, UTC timestamp, serving provider, and cost; raw data are append-only; analyses are regenerable end-to-end.
• Budget discipline: dry-runs and cost estimates before any paid call; --limit caps; frontier models at reduced reps.
• No silent exclusions: refusals and off-format answers are coded and counted, never dropped.
• Pre-registration discipline: pilot data calibrate the design but are excluded from confirmatory analyses.