Does Jev know when it doesn't know?
September 19, 2026 · View on GitHub
An independent calibration test of TypeSafe's Jev on a task it cannot have seen, alongside three public benchmarks it probably has.
Jev's pitch is epistemically honest probabilities: you set a confidence threshold, automate above it, and route the rest to a human. That only works if the probabilities stay honest on inputs the model has not been trained on. This repo measures that directly, on ~$0.06 of API calls, and publishes every raw response.
Short version. On three public benchmarks Jev is very accurate and its probabilities need almost no correction. On a rule-based task it has never seen, Jev is as accurate as a strong zero-shot model can be — and confidently wrong where the rule is unknowable, with the same magnitude of overconfidence a fine-tuned classifier shows out of distribution. The sign of the error also flips by question type.
Results
All calls through Vercel AI Gateway (model id typesafe-ai/jev; the typesafe-ai/jev-latest id in the AI SDK docs returns "Model not found" on the Gateway), AI SDK 7.0.107 experimental_evaluate, zeroDataRetention: true, 2026-09-19. 3,721 public-benchmark items + 900 synthetic items, 0 failed calls. The Gateway does not expose a model version; every response resolves to canonical slug typesafe-ai/jev and declares rounding: {probabilityDecimals: 2, scoreDecimals: 2}, which is where the 0.01 quantisation below comes from.
Refit T is a diagnostic: the single temperature that minimises NLL on that set after the fact. T ≈ 1 means the probabilities were already calibrated; T > 1 means overconfident; T < 1 underconfident. It carries the sign that a scalar ECE does not. Reported ECE is at T = 1 (as returned), 15 equal-width bins on max-probability.
Public benchmarks (likely in Jev's training mix)
| Dataset | n | Accuracy | NLL | ECE | ECE / noise floor | Refit T |
|---|---|---|---|---|---|---|
| OpenBookQA (val) | 500 | 94.2% | 0.172 | 0.024 | 1.0× (0.024) | 0.96 |
| CommonsenseQA (val) | 1,221 | 88.1% | 0.395 | 0.032 | 1.7× (0.019) | 1.35 |
| HellaSwag (val, 2k) | 2,000 | 86.1% | 0.420 | 0.029 | 1.6× (0.018) | 1.00 |
These accuracies are far above what a 3–8B model scores zero-shot on the same splits and consistent with these datasets' train splits being in the training mix (they are in every public instruction collection). We cannot verify contamination either way. The calibration numbers should therefore be read as in-domain calibration.
Synthetic support tickets (cannot be in Jev's training data)
900 tickets generated by a rule-based script on 2026-09-19 (data/), three questions each. Each question sent as one evaluate call with the option descriptions as criteria.
| Question | Type | n | Accuracy | Chance | NLL | ECE | Refit T | Reading |
|---|---|---|---|---|---|---|---|---|
| Which queue? | choice (4) | 300 | 89.0% | 25% | 0.697 | 0.082 | 3.29 | Right, but wrong answers also get 1.00 |
| Is the customer angry? | boolean | 300 | 91.7% | ~50% | 0.275 | 0.079 | 0.66 | Underconfident |
| Priority (org rule) | score (4 levels) | 300 | 44.7% | 25% | 1.331 | 0.325 | 3.40 | Cannot know; mean stated probability 0.74 |
| All | 900 | 75.1% | 0.768 | 0.107 | 2.74 |
ECE noise floor for the 900-item set (perfectly calibrated model, same predicted distributions, 200 re-samplings): 0.024. Measured 0.107 is 4.4× the floor.
The priority label is defined by a rule that is not in the text (see below). No zero-shot model can recover it; 44.7% is chance plus common sense. The question is whether Jev's probabilities reflect that it cannot know. They do not: the chosen level carries 0.74 probability on average, and the temperature needed to make the probabilities honest is 3.4.
TypeSafe also returns a separate confidence statistic. Read as a probability of being correct, its ECE is 0.035 on OpenBookQA, 0.078 on HellaSwag (worse than the max-probability), and 0.18 on the synthetic set.
Relation to other independent measurements
This is not the first independent calibration test. The jev-exploration ledger (Samuel Sacco) tracks claims about Jev and re-analyses other people's benchmarks; the numbers below are what those repositories state as of 2026-09-19:
- jev-spam-eval: ~9.9k real emails (5,733 ham/spam/phishing + 3,300 fresh + 853 recent phishing), 98.6% accuracy with context enrichment and no task-specific fitting. It does not report ECE; jev-exploration's re-analysis of it describes a reliability curve that overstates low probabilities and understates high ones.
- jev-phishing-bench: 2,000 emails, Jev 62.6% [60.5, 64.7], ECE 0.154 (10 bins) against Claude Haiku 4.5's 0.097.
- jev-exploration's own 800-item difficulty-gradient set (contamination-free): ECE 2.1–2.5× its noise floor across all four difficulty tiers, with error concentrated in the middle of the range; p ≥ 0.9 gave a 1.000 hit rate in every tier at 21.5–32.5% coverage, so thresholding at 0.9 was safe there. Its recommendation: treat the output as a monotone score, not a probability, and calibrate locally.
Those studies all use boolean (Noul) questions on tasks whose labels are recoverable from the text. This repo adds two things:
- An unknowable task. The label is a policy, not a property of the text. This tests "does the model know it doesn't know", which is the case the human-routing use case actually depends on.
- Sign by question type. Our boolean result (T 0.66, underconfident) independently reproduces the direction found in the difficulty-gradient set. Choice and Score go the other way (T 3.3–3.4, overconfident). Whether a fixed threshold is safe therefore depends on the question type, not just on the task — for Choice and Score it is not.
What Jev does well
- Semantic judgments on unseen text are strong: 91.7% on "is the customer angry", including tickets with Korean text.
- Schema never broke: 0 invalid responses in 4,621 calls.
- Speed and cost were as advertised; the whole study cost about six cents at list price.
- On the three public benchmarks, the probabilities needed essentially no correction.
Footnotes that matter for anyone thresholding on Jev's output
- Probabilities are quantised to 0.01 and frequently exactly 0 or 1. In OpenBookQA, 1,051 of 2,000 option probabilities are exactly 0. One item assigned probability 0.00 to the correct answer (1.00 to a wrong one, confidence 0.99). A probability of exactly 0 cannot be repaired by temperature scaling; that single item contributes 0.028 of the 0.172 mean NLL. This is also a counterexample to the "no misses at confidence 1.000" observation reported elsewhere.
- The sign of miscalibration differs by type (choice/score overconfident, boolean underconfident on the same inputs), which suggests type-specific post-processing rather than one calibrated distribution. Calibrate per question, not per model.
- Don't threshold on the
confidencefield. On these sets it was never better than the max probability and sometimes much worse.
The synthetic task
data/val.jsonl — 900 records, JSONL schema (state, type, question, options/levels, label). data/generate.py regenerates the set; change --seed to get a fresh, guaranteed-uncontaminated test set.
Each ticket is {channel, customer_tier, subject, body}, drawn from 20 templates across four queues, with random order numbers, amounts, days, and a calm or angry closing line (30% angry; some lines in Korean). Labels:
queue— the template's queue (billing / shipping / technical / general).angry— whether an angry closing line was appended.priority— template base urgency (0–2) + 1 if angry + 1 if tier is gold/enterprise, clipped to Low / Normal / High / Critical. The tier bump is the point: it is an organisational policy, absent from the text, of the kind every real routing, approval and escalation rule contains.
5% of labels are randomly corrupted (documented in the generator) to mimic real operational data, so no model can exceed ~95% on any question.
The exact question and option text sent to Jev is in scripts/jev_eval.mjs.
Reproduce
npm i ai@latest
export AI_GATEWAY_API_KEY=...
node scripts/jev_eval.mjs --in data/val.jsonl --out results/jev_synth.jsonl --model typesafe-ai/jev --concurrency 8
python -m scripts.eval_dump --dump results/jev_synth.jsonl --fit-temperature --confidence-reliability
Python side needs torch (CPU is fine) for the metrics; pip install torch datasets. Concurrency 8 is safe on paid credits; the free tier rate-limits this model to a handful of requests per minute, and concurrency 32 triggered retry storms.
eval_dump prints accuracy, NLL, Brier, ECE with its noise floor (the mean ECE of a perfectly calibrated model with the same predicted distributions at this n, 200 label re-samplings), the measured/floor ratio, the refit temperature, per-type and per-source breakdowns, and full reliability tables. Public benchmarks: build the JSONL with python scripts/convert.py hf --preset openbookqa --out data/ho_obqa --val-limit 2000 (likewise commonsense_qa, hellaswag) and run the same two commands on data/ho_*/val.jsonl. data/generate.py --seed <new> gives a fresh synthetic set that no model can have seen.
results/ contains every raw response exactly as returned, written before any scoring.
Limitations
- One synthetic task family, n = 900 (300 per question type). Enough to place T far from 1; not enough for fine bin-level claims.
- Public-benchmark contamination is unknowable; those rows are context, not evidence about generalisation.
- The Gateway exposes no model version and
typesafe-ai/jevmay move silently. All numbers are from 2026-09-19; re-run before quoting them against a later date. - Calibration here is measured on Jev's returned probabilities as a user would consume them. It says nothing about the model's internal distribution before post-processing.
License
Code: MIT. Synthetic data (data/): CC0.
Citation
@misc{jev-ood-calibration-2026,
title = {Does Jev know when it doesn't know? Calibration of TypeSafe's Jev on an unseen rule-based task},
year = {2026},
url = {https://github.com/scienthoon/jev-ood-calibration}
}