jevcal

Your model says 0.90. Find out what it actually means.

Decision models hand you a probability. You write if p > 0.9: auto_approve(). That line is a bet that 0.9 means 90%. This tool settles the bet against your own labelled data in about thirty seconds, and if the bet is bad, learns the correction.

python3 jevcal.py check --data example/reviews.csv --threshold 0.9

Raising the threshold doesn't rescue you. That's the point.

---

Fix it

python3 jevcal.py fit --data example/reviews.csv --out calibration.json

  ── Held-out half, never seen during fitting ───────────────
                  Brier      ECE      AUC
  as shipped     0.0837   0.1601   0.9264
  platt          0.0248   0.0081   0.9264
  isotonic       0.0270   0.0159   0.9112

  Brier 0.0837 -> 0.0248  (70% better)
  AUC   0.9264 -> 0.9264  (a monotone rescale reorders nothing)

  ── Translation table ─────────────────────────────────────
   your model says     it really means
              0.50  ->              6%
              0.70  ->             11%
              0.90  ->             28%
              0.95  ->             42%
              0.99  ->             75%

  ── The threshold you should write ────────────────────────
  for a real 50%:  if p >= 0.965   (5 of 1600 judgments in this sample clear it)
  for a real 90%:  if p >= 0.997   (0 of 1600 judgments in this sample clear it)

Read that last line carefully. To genuinely get 90%, you would need p >= 0.997, and nothing in 1,600 judgments ever got there. The highest real confidence any judgment reaches after correction is 62%.

Then one line in production:

import jevcal

cal = jevcal.load("calibration.json")

if cal(p) > 0.9:       # now 0.9 means what you think it means
    auto_approve()

---

What you need

A CSV with two columns: the model's probability, and what actually happened.

p,y
0.93,0
0.12,0
0.88,1

Column names default to p and y; override with --p-col / --y-col.

About 100 labelled rows. Not thousands. Measured on real data — the correction is 94% of the way there at 100 rows and stops improving after that:

labelled rows	% of calibration error removed
25	69%
50	90%
100	94%
800	93%

Under 50 rows or 10 positives the tool warns you that its own output is not trustworthy, rather than printing confident-looking numbers anyway.

---

Three things it refuses to do

It won't invent a fix you don't need. If recalibration doesn't improve Brier score on held-out data, it says so and writes no file.

It won't score itself on data it trained on. Fit on half the rows, report on the half it never saw. A calibration scored on its own fitting data always looks perfect and means nothing.

It won't let you celebrate ECE. Expected Calibration Error is the number everyone reaches for, and on imbalanced data it is close to useless: a model that ignores its input and returns the base rate for every row scores a perfect ECE of 0.0000 with AUC 0.5. The tool prints that baseline next to your result and makes its own decisions on Brier, which can't be gamed that way.

---

What it can't do

Calibration changes the units, not the model. If your model can't reach 90% real confidence on your data, no threshold and no calibration will get you there. What you gain is knowing that, instead of nudging the threshold up and hoping.

It also assumes your labelled sample looks like your production traffic. The correction is largely a base-rate shift, so if your live positive rate is very different from your sample's, refit.

---

Commands

jevcal check --data labelled.csv --threshold 0.9     # the diagnosis
jevcal fit   --data labelled.csv --out cal.json      # learn the correction
jevcal apply --data new.csv --calibration cal.json   # apply it to fresh rows

Install

There isn't one. It's a single file and the only dependency is numpy.

curl -O https://raw.githubusercontent.com/Adilmp/jevcal/main/jevcal.py

Platt scaling and isotonic regression are implemented directly, about sixty lines each, rather than pulled from sklearn — so every number it prints is traceable to code you can read in one sitting.

---

Where this came from

An audit of TypeSafe's Jev against civil_comments, where every comment carries the share of human annotators who flagged it. Jev ranked well — AUC 0.91 — while its stated probabilities sat below the diagonal in every confidence band. The ranking was fine; the units were wrong. Full method, charts and raw data are in that repo.

TypeSafe's own docs say: "Start with conservative thresholds, test with your own data, and adjust as you observe results." This is the tool that does it.

Nothing here is Jev-specific. It works on any model that emits a probability you threshold against — an LLM judge, a classifier, a fraud score.

example/reviews.csv is the real Jev output from that audit, 1,600 judgments, so every number in this README reproduces from a clean checkout.