typed-gate

A pattern for using a System One model without throwing away the number it gives you.

Code finds candidates. The model judges. Code decides what a human sees.

pip install -r requirements.txt
cp .env.example .env          # add your TypeSafe key
cd examples/documents
python generate.py --n 200 --seed 23
python run.py --n 100

The idea

Jev returns a typed answer and a calibrated probability. It does not generate text, so it cannot invent a value: it picks one of the options you hand it.

Nearly every demo built on this takes the argmax and acts. The probability is right there in the response and nothing reads it. This library is the missing half:

Step	Who	Why
1. Find candidate spans	your code, regex or a parser	the model cannot pick a span you did not find
2. Choose among them, and say whether the source supports the choice	the model	one choice and one noul per field, in a single call
3. Decide whether a human looks at it	typed_gate/gate.py	policy, not inference

Step 3 is the whole library. Everything else is plumbing.

The rule people get wrong

A yes/no probability near 0.5 is the model declining to answer. It is not a weak yes.

if support < 0.5:        # wrong. 0.51 is not a yes.
    send_to_human()

lo, hi = 0.40, 0.60      # right. the band is a refusal to commit.
if lo <= support < hi:
    send_to_human()

Change a threshold in Gate and nothing re-runs. The judgments are reusable; the policy is yours.

Results

100 synthetic freight documents, 11 fields each, 1,100 field decisions, run 19 September 2026 from the Netherlands. Every lane got the identical documents and the identical questions, and both lanes are graded by the same matcher.

Lane	Supplied a wrong value	Omitted a value that existed	Flagged for review	p50	$/1k docs
Jev + gate	0	0	117	260 ms	$0.09
Jev, argmax only	0	25	0	260 ms	$0.09
Claude Haiku 4.5	6	0	0	1,788 ms	$1.35
Gemini 3.7 Flash	0	0	0	6,789 ms	$3.33

Two harms are counted separately on purpose. A wrong value reaches the record and is invisible to whoever reads it later. An omission leaves a blank that somebody notices.

Correctness is a tie. Gemini also scored zero wrong values. Jev is 26x faster and 36x cheaper than Gemini on this task, and that is the honest claim. It is not more accurate.

The one result worth reading closely

All six of Haiku's wrong values landed on fields the document does not state. Four were Incoterms it worked out from prose.

Four documents carry an identical sentence: the shipper clears export and pays carriage to the named destination port, risk passes once the goods are on board. That is CFR, and the letters CFR appear nowhere on the page.

Haiku answered CIF, CIF, CPT, CFR. One right. Nothing in the JSON marks any of them as inferred. Re-run at a second pass, the same six fields failed, and doc-0002 changed from CIF to CPT.

Jev returned nothing on all four and flagged them.

Jev's own failures

25 omissions, every one of them ocr_noise, and every one of them the candidate generator's fault rather than the model's: a regex cannot match a mangled string, so no candidate exists. The gate caught all 25. That is the design working, not a rescue.

Point it at your own data

Everything domain-specific lives in one YAML file. typed_gate/ never changes.

name: your_document_type
fields:
  - id: invoice_number
    ask: "Which candidate is the invoice number issued by the supplier?"
    patterns: ['\bINV[-/]?[0-9]{4,10}\b']
gate:
  accept_choice_confidence: 0.80
  supported_min: 0.70
  band: [0.40, 0.60]

Write a labels.json mapping each file to its true values and run.py scores you.

Tune the thresholds on your data. The three numbers above are defaults, not findings.

What would make these numbers wrong

• The documents are synthetic and I wrote the generator. generate.py self-checks that every truth it claims appears verbatim in its own document, which caught three labelling bugs that had each manufactured a fake error rate. It cannot catch documents that are simply too easy.
• n=100, one seed per lane. Haiku was re-run and produced the same six failures, so that number is stable. The others were not re-run.
• Text input only. Real documents are PDFs and scans, and the candidate step is where that lands.

The interesting result is not what happens on documents I invented. It is what happens on yours.

Practical notes, measured rather than assumed

• Reuse the connection. A fresh TLS handshake cost 350 ms per call from Europe. With a requests.Session the same call is 249 ms p50.
• Latency is flat to about 25 questions, then it climbs. 4 questions 261 ms, 25 questions 275 ms, 100 questions 551 ms, 200 questions 710 ms. Keep a schema near 10 fields per call.
• There is a hard input ceiling of 65,536 tokens. 340 short questions passed; 345 returned 400 max_tokens_exceeded. A token limit, not a question-count limit.
• A thinking model needs output headroom. Gemini 3.7 Flash spends ~600 reasoning tokens per document. At max_tokens: 700 it returns empty content that looks exactly like a parse failure.
• It reads dates as text, not ordered quantities. Do not ask it to compare or sequence dates.
• It does not count reliably. Count in code.

Examples

• examples/documents — freight booking confirmations. Built and measured above.
• examples/prose — the same pattern over writing. Not built yet.

Licence

MIT. No real company, shipment, person or customer appears anywhere in this repository. Every value in examples/ is generated by generate.py from a seed.