Jev Arena

September 19, 2026 · View on GitHub

A calibration arena for TypeSafe Jev: pose many questions about hidden stochastic processes whose exact probabilities are analytically known, collect Jev's native probability for every event in one fan-out request, and measure how honest those probabilities are (Brier, log loss, ECE, a reliability curve, and a confidence-gated risk–coverage table).

What it is, and why it's interesting

Every existing LLM-calibration repo measures verbalized confidence from a chat model ("I'm 80% sure") and then rescues it with temperature scaling. Here the model's native output is the distribution — Jev answers typed noul / choice / score questions — so this demo exists purely to measure it, not to patch it.

The worlds are analytically exact (a coin with a stated bias, integer-count urns, a standard deck, a uniform hidden integer), so there is no sampling noise: the oracle stub scores a perfect Brier/ECE of 0.0000, and every deviation is Jev's.

A live smoke test showed Jev assigns 0.89 to face "1" of a fair die, so real miscalibration is visible — the demo is genuinely informative, not a victory lap.

Why Jev / TypeSafe

Native probabilities. noul returns a probability in [0, 1]; choice and score return a full normalized distribution plus a confidence. Nothing has to be parsed out of prose.
Fan-out cost. All 200 questions are answered in 2 requests (~150/batch, ~0.5 s/request) for ~19k tokens total — the cost of measuring calibration is one call per batch, not one per event.

Quickstart

cd ~/jev-arena
uv sync --dev

# Offline (default; no key, no network) — a perfect oracle stub:
uv run jev-arena --offline --trials 200 --seed 7 --results-dir results

# Miscalibrated stub — proves the ECE detector has teeth:
uv run jev-arena --offline --stub miscalibrated --trials 200 --seed 7 --results-dir results

# Live (real API; needs TYPESAFE_API_KEY):
uv run jev-arena --live --trials 200 --seed 7 --results-dir results

The API key is read at runtime from the environment, then <repo>/.env, then ~/jev-ultrafast/.env. It is never printed, logged, or written.

How it was verified

uv sync --dev
uv run ruff check .                 # All checks passed!
uv run pytest -q                    # 25 passed, 1 skipped (live test needs an explicit TYPESAFE_API_KEY)
uv run pytest -m live -q            # 1 passed (real API; requires TYPESAFE_API_KEY exported in the environment)
uv build                            # sdist + wheel
uv run jev-arena --offline --trials 200 --seed 7 --results-dir results
uv run jev-arena --live --trials 200 --seed 7 --results-dir results

Live run headline (--live --trials 200 --seed 7, model jev-1.13.0, the run committed under results/): over 145 binary (noul) events, Jev scored Brier 0.0059, log loss 0.5393, ECE 0.0620 in 2 requests / 710 ms (~3.6 ms/event, 13,566 in / 5,468 out tokens). Baselines: always-0.5 Brier 0.0766 / ECE 0.1371; oracle Brier 0.0000 / ECE 0.0000. (Jev is stochastic, so a re-run moves these by a few thousandths — e.g. an earlier run scored Brier 0.0063 / ECE 0.0642.)

Reliability (predicted → empirical) is systematically overconfident in the low bins and near-perfect in the high bins:

bin	mean predicted	empirical rate	count
[0.1, 0.2)	0.166	0.096	18
[0.2, 0.3)	0.236	0.140	22
[0.3, 0.4)	0.336	0.269	46
[0.4, 0.5)	0.430	0.364	13
[0.5, 0.6)	0.555	0.492	15
[0.6, 0.7)	0.649	0.638	7
[0.7, 0.8)	0.743	0.686	10
[0.8, 0.9)	0.845	0.840	12
[0.9, 1.0)	0.920	0.950	2

Categorical questions (choice / score, 55 events) are where Jev is least calibrated: multiclass Brier 0.1915, cross-entropy 2.0571, mean API confidence 0.4905. The per-world breakdown shows the damage concentrated in the weighted-die and deck worlds (ECE 0.109 / 0.084), matching the "0.89 on a fair die" smoke test.

Risk–coverage (gate = noul decisiveness d = |p - 0.5| * 2, since noul has no API confidence): acting only when d >= 0.25 covers 76.6% of events at 100% accuracy; d >= 0.10 covers 94.5% at 99.3%. The HIGH/MEDIUM/LOW bands are >= 0.8 / >= 0.4 / else.

The committed results/ in this repo is from the live run. Re-running offline overwrites it with the oracle stub's perfect scores.

Offline mode

--offline (the default) never touches the network and never reads TYPESAFE_API_KEY. Two stubs:

oracle — returns each question's exact ground truth in Jev's schema. Brier/ECE 0.0000, the unit-test anchor that the scoring pipeline is wired right.
miscalibrated — sharpens toward 0/1 (T = 1.5) and adds +0.45 bias, an overconfident forecaster that must trip the ECE detector. Measured ECE 0.3565 at seed 7 (and > 0.25 across seeds). This is the teeth test.

Metrics

All pure stdlib, no numpy, hand-verifiable:

binary Brier mean((p - y)^2), log loss -mean(y log p + (1-y) log(1-p)) with p clipped to [1e-12, 1-1e-12];
ECE with configurable equal-width bins, plus a reliability table and an ASCII calibration curve;
risk–coverage sweep over a gate threshold, and a 3-band HIGH/MEDIUM/LOW policy;
categorical multiclass Brier mean(sum_c (p_c - y_c)^2) and cross-entropy -mean(sum_c y_c log p_c).

tests/test_metrics.py hand-checks each against tiny fixtures (e.g. ECE of [(0.9,1),(0.8,1),(0.3,0),(0.2,0)] is exactly 0.2).

Safety rails

No secrets. .env is gitignored; .env.example holds empty placeholders. The key is resolved at runtime and never printed, logged, or written.
Validated output. Every answer's keyset must match the offered criteria, all probabilities must be finite and in [0, 1], and they must sum to 1 within 0.02, or the batch is retried.
No network by default. --offline is the default; --live is explicit.
Ground truth is exact. Probabilities come from the world definitions, not from sampling, so the oracle's zero error is real.

CI / no secrets

.github/workflows/ci.yml runs lint, tests, and a build on Python 3.11 and 3.12. .github/workflows/demo.yml runs the offline demo and uploads results/ as an artifact. Both run without any secrets: the offline path never reads TYPESAFE_API_KEY, and no workflow references it.