typesafe-local

Ask a local language model typed questions about a document and get back probability distributions instead of text. No parsing, no retries, no API key.

TypeSafe shipped a model that returns typed decisions instead of prose, and the idea was interesting enough that I wanted to know how much of it comes from the model and how much from the way it is served. So I built one, on top of an off-the-shelf local model, and measured it.

Most of the speed, it turns out, is serving structure — encode the document once, read one position, never generate. That part reproduces on a laptop with nothing trained.

The numbers are the hard half. Raw probabilities are close to meaningless until corrected, and on a judgement call the answer can turn on the order you listed the options in while still reporting confidence 1.000. Both are measured here rather than asserted.

Unaffiliated with TypeSafe. The API shape is deliberately theirs so the same client works against either.

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  -d @examples/support_ticket.json

{
  "model": "mlx-community/Qwen3-1.7B-bf16",
  "answers": {
    "refund_requested": {"type": "noul", "noul": 1.0, "raw_mass": 1.0},
    "department": {"type": "choice", "choice": "billing",
                   "probabilities": {"billing": 0.44, "technical": 0.56, "sales": 0.0},
                   "confidence": 0.34, "raw_mass": 1.0},
    "frustration": {"type": "score", "score": 1.0,
                    "probabilities": {"0": 0.0, "1": 1.0, "2": 0.0},
                    "confidence": 1.0, "raw_mass": 1.0}
  },
  "usage": {"state_tokens": 164, "question_tokens": 177, "request_peak_mb": 244.5}
}

That department answer is a good one to sit with: a duplicate charge is arguably either team, the distribution says so, and confidence 0.34 is the model declining to pick. That is the signal you are paying for.

Four questions over one document: ~200 ms on an M4 Pro.

How it works

Three ideas, in ots/engine.py:

1. The document is encoded once. Its KV cache is kept and every question is a short suffix appended to it, then rewound. Ten questions cost barely more than one, and no question can see another.
2. Nothing is generated. We read the logits at the one position where the answer would start and softmax over only the tokens that are legal answers. A wrong-typed answer isn't caught and retried — it's unrepresentable.
3. The raw probabilities are wrong, so we measure and correct them.

Three more diagrams — request flow, how logits become typed answers, and the calibration path — are in docs/diagrams.

Quick start

python3 -m venv .venv
.venv/bin/pip install -e '.[server]'
.venv/bin/ots-server

The model (~1.1 GB) downloads on first run into .hf_cache/ inside this folder. ./cleanup.sh --yes removes everything; nothing is written elsewhere.

OTS_MODEL=mlx-community/Qwen3-1.7B-bf16 OTS_MEMORY_LIMIT_GB=8 .venv/bin/ots-server

variable	default
OTS_MODEL	mlx-community/Qwen3-0.6B-bf16	any MLX causal LM
OTS_MAX_STATE_TOKENS	8000	largest document
OTS_MAX_QUESTION_TOKENS	4000	largest single question
OTS_MAX_RESIDENT_TOKENS	12000	the memory guard: state + largest question
OTS_MAX_TOTAL_TOKENS	200000	total compute per request, not memory
OTS_MEMORY_LIMIT_GB	6	passed to MLX; see the caveat below

The token limits are the real guard, and they are checked before anything reaches the GPU. OTS_MAX_RESIDENT_TOKENS is the one that maps to memory: because the cache is rewound between questions, peak residency is the document plus the single largest question, never the sum. That is why 900 short questions over a 1500-token document run fine at 712 MB.

OTS_MEMORY_LIMIT_GB is not a hard cap. MLX documents set_memory_limit as "a guideline", and it only raises once RAM and swap are exhausted — which is the outcome you wanted to avoid. Measured: with a 0.05 GB limit set, a request still completes having touched 1.2 GB. Size your requests with the token limits; treat this one as a hint to MLX.

Requests are serialised with a lock. The handler is a sync def, so FastAPI would otherwise run it in a 40-thread pool where every request passes its own budget check and they collectively exhaust one shared process-wide ceiling.

Read raw_mass first

Every answer carries raw_mass: the share of the model's probability that landed on the tokens we offered it, before renormalising. confidence is computed after renormalising and is therefore blind to it — an answer can report confidence 0.99 with raw_mass 0.02.

Near 1.0 — the format landed, the numbers mean something. Low — the model wanted to say something we never offered. Everything below it is noise rescaled to look like a distribution. This is not a warning you can skip; it is how you tell a decision from an artefact.

Calibration

.venv/bin/python eval/make_dataset.py    # 96 labelled examples
.venv/bin/ots-calibrate                  # fit, and report held-out before/after

Out of the box Qwen3-0.6B gives p(yes) = 0.96 to "Is this about cats?" asked of a payments complaint. On a held-out half:

	raw	calibrated
AUC	0.886	0.886
accuracy	0.396	0.875
ECE	0.527	0.196

The AUC doesn't move, and that is the point: the ranking was always fine, only the scale was wrong. Two fitted numbers (p = σ(0.97·gap − 3.38)) fixed it.

Check AUC before ECE. High AUC with bad ECE is a calibration problem and two parameters fix it. Low AUC means there is no signal and rescaling cannot help.

Calibration is not free: on Qwen3-1.7B, which is already well calibrated (ECE 0.118, bias ≈ 0), fitting on 48 examples made it slightly worse (0.128).

A calibration belongs to the model it was fitted on and calibration.json records which. Applying the wrong one is silently destructive — the −3.38 bias above, applied to the unbiased 1.7B, turned a p(yes) of 0.97 into 0.50 — so the engine refuses a mismatch rather than correcting the wrong thing.

Does any of that survive real data?

pip install -e '.[eval]'
.venv/bin/python eval/boolq.py --model mlx-community/Qwen3-1.7B-bf16

The 96-example set above is mine — I wrote the passages, the questions and the answers. It shows the machinery works and proves nothing about the world. BoolQ is someone else's: real yes/no questions over Wikipedia passages, 3,270 held-out examples, 62.2% yes. It maps onto noul exactly, so there is no room to make it easy by construction.

Qwen3-1.7B, full validation split, bootstrap 95% intervals:

	accuracy	ECE
always answer "yes"	0.622	—
raw	0.754 [0.740, 0.769]	0.211
corrected on my 96 toy examples	0.755 [0.740, 0.769]	0.054
corrected on BoolQ itself	0.780 [0.760, 0.799]	0.025
corrected using the 0.6B's parameters	0.691 [0.675, 0.706]	0.267

AUC is 0.852 throughout — calibration only rescales, so it cannot move it.

Three things worth taking from this:

A correction fitted on 96 synthetic questions about cooking and astronomy took ECE from 0.211 to 0.054 on real Wikipedia QA, most of the way to the 0.025 you get by fitting on BoolQ directly. So the miscalibration is a property of the model's readout, not of the set I happened to build. That is a stronger claim than the toy numbers alone could support, and I expected it to fail.

Using another model's parameters is worse than doing nothing — 0.267 against a raw 0.211. This is the pairing the engine refuses at construction, measured.

The two models are miscalibrated differently. The 0.6B needed a bias shift (b = −3.5, it says yes to everything); the 1.7B needs a sharpness fix (a ≈ 0.15, b ≈ 0, its log-odds are roughly six times too extreme). Same symptom, different cause, which is why one file cannot serve both.

For scale: 0.780 against a 0.622 baseline is a real signal, and BoolQ state-of-the-art is around 0.90. This is a 1.7B model on a laptop, not a competitive entry.

Option order changes the answer

.venv/bin/python eval/robustness.py     # every permutation of each option list

Reading a letter token is a multiple-choice prompt, and the order you write the options in is part of it. Measured on Qwen3-1.7B, all 24 permutations of a 4-option question:

case	widest spread	winner stable
language named in the text	0.000	100%
request stated outright	0.000	100%
cause not stated — a judgement	0.995	67%

When the answer is in the text, reordering does nothing at all. When it's a judgement, order takes over: the same option ranged from 0.005 to 1.000, and the winner flipped between all four options.

confidence does not warn you. It is computed within one ordering, so it cannot see the variance across orderings — the most confident single permutation of that judgement case reported 1.000. A well-typed, high-confidence answer can be an artifact of the order you happened to write the criteria in.

So: for anything that isn't stated outright, average over a few permutations, or freeze the option order and calibrate against that exact order. Changing the option list is changing the question.

Tests

pip install -e '.[test]'
pytest          # 37 tests, no model needed, ~1 s
pytest -m model # 11 more against a real model

The model-marked tests cover the claims the design rests on: that every answer label is a single token at the real answer boundary, that an answer does not change when other questions are added to the request or reordered, and that the budget admits many cheap questions because the cache is rewound between them.

Limits

• 26 options per choice, 10 levels per score. Answers must be single tokens. Two-letter labels (AA, AB) are single tokens and would allow 552, but the model ignores them and emits a single letter anyway (measured: p=0.997 on "A" for a 27-option question). For a larger set, narrow the options for that question — shortlist in code, or walk a taxonomy a level at a time, feeding each answer into the next question's options. Do not split one flat list across several questions and merge: each question renormalises over its own options, so the pieces are not on a common scale. Measured — asked about a power supply, the half-list that excludes the right answer confidently returns resistor at 0.855.
• Signal decays with document length. Asking "is this about software?" vs "about cooking?": separation +3.00 on a 25-token document, +0.25 on a 1225-token one. So a calibration fitted on short states is wrong on long ones.
• Calibration is the part training buys and serving cannot. Two fitted parameters correct a global bias; they cannot make uncertainty mean something per question. A model trained for this is calibrated before any correction, and its confidence varies with the question rather than shifting uniformly. That is the one gap here that is not an engineering problem.

MIT licensed. See docs/architecture.md for the internals.