reflex

September 21, 2026 · View on GitHub

A tiny "decision model" you can run on your own GPU.

You give it some information (a support ticket, a document, a photo) and a list of questions with fixed answer options. It answers all the questions at once and tells you how sure it is about each option, as percentages. It never writes free text, so it can never make up an answer that isn't on your list.

It is an open re-creation of Jev, the "System One" model TypeSafe released in September 2026, built on top of a normal open-weights model (Qwen3.5-4B by default).

state:     "My payouts have failed three times this week and nobody replied to my emails."

question   which team?        ->  payments 99.9%   account 0.04%   other 0.04%
question   escalate?          ->  yes 90%
question   how urgent (0-2)?  ->  1.9   (low 4%  medium 1%  high 95%)
question   refund requested?  ->  yes 2%

That whole answer comes back in about 200 ms, and faster once the state is cached. Your code then decides what to do with the numbers ("auto-route if above 90 %, otherwise ask a human").

Status

What it is. One HTTP endpoint, POST /v1/systemone, answered by a single forward pass over a frozen open-weights model. No decoding, no reasoning, no escalation: about 200 ms, and the slowest request is a small multiple of the fastest.

Where it stands. On the public items of JevBench, against Jev itself:

	easy	standard	hard	hard ECE	latency
reflex, frozen Qwen3.5-4B, two orders (`stable`)	1.000	0.917	0.685	0.081	~200 ms
reflex, frozen Qwen3.8-27B, two orders	1.000	0.958	0.766	0.061	~1 s
Jev 1.13.0 (official)	1.000	0.986	0.730	0.031

Ours are self-run numbers on a suite we have consulted throughout development. The official JevBench v1.2 runs (534 decisions including held-out items, run by the benchmark's author on an H100) placed reflex (4B, an earlier LoRA configuration) #5 of 36 with score 71.7, and reflex-27b #22 with the highest Intelligence (90.5) and Calibration (86.2) on the board and hard-tier 75.9 % against Jev's 74.1 %, held back by cost and speed. Details and what was actually run: docs/results/jevbench-official.md.

What is proven not to help. Fine-tuning, in every form we tried. Four LoRA mixes on public datasets and a distillation from a 27B teacher all won on data shaped like their training data and lost general judgement on long, ambiguous inputs. GEPA prompt optimisation did the same in miniature. Wording ensembles, more than two option orders, and a reasoning cascade behind a fitted escalation trigger were all measured and none of them ships. What did help was the readout: a lettered yes/no answer, Evidence / Criterion framing, and averaging two distinct option orders. Every run, with its verdict, is indexed in docs/results/README.md.

How to run it. git checkout stable && uv run reflex-serve --stable on any 16 GB CUDA GPU. The five-minute version is below.

Try it in your browser first

https://kshetrajna12.github.io/reflex/ runs the whole thing on your own GPU through WebGPU (Chrome, Edge, or Safari 18+) with a 650 MB Qwen3.5-0.8B model. Pick a preset, load the model once, drop in a photo, press Run. Nothing is uploaded anywhere. It is the same request format and the same readout as the Python version below, just smaller and less calibrated. Read it as a demonstration of the mechanism rather than of the served quality: a frozen 0.8B is not a usable judge, and the model that ships is the 4B (docs/results/weight-classes.md). The page lives in docs/: reflex.js is the inference module, app.js the UI.

Why would I want this?

Large chat models are great at reasoning but slow and expensive when all you need is a quick, structured judgment: which queue, is this spam, how angry is this customer, does this photo match its caption. A decision model:

returns numbers, not prose – nothing to parse, no JSON that fails to validate;
answers many questions in one pass – ask 20 questions about one document for roughly the cost of one;
tells you its confidence, honestly – after calibration, "85 % sure" really means right about 85 % of the time, so you can set thresholds;
works on images too – put a photo in the state and ask typed questions about it.

Get started in five minutes

You need a Linux machine with an NVIDIA GPU (8 GB of memory is enough for the default model), Python 3.12, and uv.

git clone https://github.com/kshetrajna12/reflex
cd reflex
git checkout stable                       # the commit and configuration we recommend
uv sync                                   # installs PyTorch (CUDA 13), transformers, etc.
uv run python examples/support_ticket.py  # downloads Qwen3.5-4B (~8 GB) the first time

stable is a git tag that moves with the configuration, and serving/stable.json at that commit says what it is: today the frozen Qwen3.5-4B, the default prompt, no adapter, no calibration file, every question read in two option orders. Stay on main if you would rather follow the experiments.

You should see the ticket example above printed as JSON, plus timings. The very first call takes ~20 seconds while GPU kernels compile; after that it is fast.

Try it on a photo:

uv run python examples/image_triage.py path/to/photo.jpg --caption "two cats on a couch"

Ask your own questions

from reflex import Engine, SystemOneRequest

engine = Engine.load("Qwen/Qwen3.5-4B")

resp = engine.answer(SystemOneRequest(
    state={"email": "Hi, I was charged twice for my subscription this month, please fix."},
    questions={
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {"billing": "charges, refunds, invoices",
                         "technical": "bugs, outages",
                         "sales": "pricing, upgrades"},
        },
        "angry": {"type": "noul", "instructions": "Is the customer angry?"},
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this?",
            "criteria": ["can wait a week", "should be handled today", "blocked right now"],
        },
    },
))

print(resp.answers["team"].choice, resp.answers["team"].probabilities)
print(resp.answers["angry"].noul)          # probability of "yes"
print(resp.answers["urgency"].score)       # 0.0 .. 2.0, probability-weighted

Engine.load takes the flags the server takes. To get exactly what the server serves, including the two option orders, load the manifest:

from reflex.serving import engine_kwargs, load_stable
engine = Engine.load(**engine_kwargs(load_stable()))   # or Engine.load(..., default_permutations=2)

The three question types

type	asks	you get back
`noul`	"is this true?"	`noul`: probability of yes
`choice`	"which one of these?"	`choice` (best option), `probabilities` for every option, `confidence`
`score`	"how much, on this scale?"	`score` (weighted position), `probabilities` per level, `legend`, `confidence`

instructions is the question. criteria are the options (choice), the ordered levels from low to high (score), or optional descriptions of what "yes" and "no" mean (noul).
state can be a string or any JSON. Put an image anywhere in it as {"type": "image", "source": "<file path, URL, or data: URI>"}.
All questions in one request are answered independently and in parallel.

Run it as a server

uv run reflex-serve --stable --port 8008

curl -s localhost:8008/v1/systemone -H 'content-type: application/json' -d '{
  "state": "The export button crashes in Safari but works in Chrome.",
  "questions": {
    "browser_specific": {"type": "noul", "instructions": "Is the bug browser-specific?"},
    "severity": {"type": "score", "instructions": "How severe is this?",
                 "criteria": ["cosmetic", "degraded but there is a workaround", "blocking"]}
  }
}'

The request and response shapes are the same as TypeSafe's hosted API, so client code written for Jev can point at http://localhost:8008 instead.

--stable reads the recommended settings from serving/stable.json; spell them out with --model and --permutations if you prefer. --api-key (or REFLEX_API_KEY) puts a bearer key in front of /v1/*, which you want on anything reachable from the internet.

If your GPU is already running SGLang, reflex can read the same label logits off it instead of loading the weights itself:

uv run reflex-serve --backend sglang --sglang-url http://127.0.0.1:30000 \
    --model Qwen/Qwen3.5-4B --permutations 2 --port 8008

Same wire format, same answers (the two backends' probabilities differ by 0.004 at the median). At 4B the in-process engine is faster, so this is for deployments that already run SGLang and for the 27B, where it is the better way to serve. docs/SERVING.md has the launch recipe and the caveats; docs/results/sglang-backend.md has the numbers.

On a Mac (Apple Silicon)

The server also runs on the Mac GPU through PyTorch MPS. Install without the CUDA wheel index, then pick the device:

uv sync --no-sources                      # plain PyPI torch, which includes MPS
uv run --no-sync reflex-serve --model Qwen/Qwen3.5-2B --device mps --dtype float16

Use --no-sync afterwards, or uv run re-resolves torch against the CUDA index. float16 rather than bfloat16. Measured on an M3 Pro (18 GB): Qwen3.5-0.8B answers the request above in about 130 ms and Qwen3.5-2B a 200-token state with two questions in about 500 ms. Pick a model whose weights fit in roughly half of unified memory: MPS memory is capped so that a model that is too large fails with an out-of-memory error instead of swapping the machine to a halt (Qwen3.5-4B in float16 does not fit in 18 GB).

Make the percentages honest (calibration)

Out of the box the numbers are roughly right. To make them trustworthy for thresholds, run the calibration check. It answers 1,200 exam questions and measures how well confidence matches accuracy:

uv run reflex-eval-mmlu --n 1200 --fit-temperature runs/calibration.json
uv run reflex-serve --calibration runs/calibration.json

What we measured (lower ECE = more honest; Jev reports 0.031):

model	accuracy	honesty (ECE) before	after
Qwen3.5-4B	72 %	0.090	0.039
Qwen3-8B	71 %	0.264	0.061

The stable configuration ships no calibration file, because a temperature fitted on one distribution does not transfer to another, and reading each question in two option orders already cuts the calibration error on never-trained external sets roughly in half for free (docs/results/order-averaging.md). Fit a temperature on data from your own workload, and refit it whenever the model, the prompt or the precision changes.

Fine-tune it on your own tasks

Read this as a tool, not a recommendation. Every adapter trained in this repo was rejected: each one won on data shaped like its training mix and lost general judgement on long, ambiguous inputs, so the served configuration is the frozen model (docs/results/frozen-vs-trained.md). Where it does pay is the case below: your own workload's labels, where the data you train on is the data you will see. The numbers in this section are in-distribution numbers.

Temperature fixes over-confidence but cannot make the model better at a task. For that you train it, and the recipe is simple: show it labelled examples and penalise it with a proper scoring rule (log loss or Brier), which is minimised only by the true probabilities. That is the supervised form of the "RLCD" training Jev uses.

You need labelled data in the same shape as a request, one JSON object per line:

{"state": {"comment": "..."}, "questions": {"toxic": {"type": "noul", "instructions": "..."}},
 "labels": {"toxic": 0.67}, "source": "civil_comments"}

Labels can be hard ("billing", true, 2) or soft (0.67, {"billing": 0.7, "sales": 0.3}) when annotators disagreed; soft labels are what a proper scoring rule wants.

reflex-data builds such files from eight public datasets, one recipe each, covering all three primitives (routing intents, exam questions, toxicity with soft labels, hallucination checks, passage relevance, response helpfulness, code-review chunks):

uv run reflex-data mix --out runs/mix_train.jsonl --eval-out runs/mix_eval.jsonl --per-source 800
uv run reflex-calibrate train --data runs/mix_train.jsonl --val runs/mix_eval.jsonl --out runs/lora-mix
uv run reflex-serve --adapter runs/lora-mix --calibration runs/lora-mix/calibration.json

What one epoch of that bought on Qwen3.5-4B (held-out, 200 items per source):

	accuracy	calibration error (ECE)
raw model	62.7 %	0.120
after LoRA	76.8 %	0.051
after LoRA + temperature	76.8 %	0.024

Toxicity went from 50 % to 94 %, hallucination checks from 78 % to 99 %, code-review "needs a comment" from 50 % to 74 %. Per-source numbers and caveats are in docs/results/lora-mix-qwen3.5-4b.md.

Training is LoRA by default: minutes on one GPU, base model untouched. --full updates every weight instead, which fits a 4B model on a large GPU but rarely helps for a few thousand examples. The trainer prints accuracy and calibration per source before and after, so you can see exactly what the training bought. Your own data plugs in the same way; src/reflex/train/recipes.py shows how each public dataset was mapped onto a primitive, which is the part to copy.

If you have the inputs but not the labels, reflex-distill is the other route: a stronger model answers your own states through the same prompt, and its distributions become the targets (docs/DISTILLATION.md). That teacher may reason (reflex-distill label --think N); the student it trains never does.

Example: triaging a pull request

examples/pr_review.py is a small AI PR-review triage built on this: a PR-level state (title, description, files) answers what kind of change, how risky, breaking, needs a migration, does the description match, and every diff hunk answers sensitive area, weakens error handling, debug leftovers, public API change, behaviour change, and how much a senior reviewer would want to look. Code aggregates the numbers and prints the hunks to hand to a reasoning model or a human.

uv run python examples/pr_review.py --repo pydantic/pydantic --pr 13824

kind            feature    feature   98%  bugfix    1%  chore    0%
risk            1.96 / 3   (max hunk needs-eyes 2.02 / 3)
breaking          29%      needs migration   27%      description matches   82%
hunks           40 reviewed, 0 skipped   logic hunks 20   test files changed 7
escalate to a reasoning model / human (P(needs a careful read) >= 50%):
    84%  pydantic-core/src/validators/counter.rs        @@ -0,0 +1,182 @@   +182/-0
    82%  pydantic-core/src/input/input_python.rs        @@ -487,6 +488,28 @@ +22/-0
    81%  pydantic-core/src/serializers/type_serializers/counter.rs  ...    +164/-0
1.2s PR level, 13.3s for 40 hunks

The raw model already orders things sensibly; the point of the design is that the outputs are numbers, so thresholds are yours, and with your own history of reverted or hotfixed PRs the calibrator can be trained so "80 %" means 80 % on your codebase.

The browser demo has the same triage at the bottom of the page: it fetches a public PR from the GitHub API and runs it on the 0.8B model on your GPU. Expect a rougher ordering than the 4B; it is the same questions.

How it compares

On the public items of JevBench, a benchmark for Jev-class decision models, the frozen Qwen3.5-4B with reflex's default prompt, reading each question in two option orders, scores 1.000 / 0.917 / 0.685 on the easy / standard / hard tiers, with hard-tier calibration error 0.081 and no calibration file. Jev itself scores 1.000 / 0.986 / 0.730 at ECE 0.031, and the strongest other open 4B rebuild 1.000 / 0.986 / 0.613, on the same items. If you have a 60 GB GPU, the same code on the frozen Qwen3.8-27B reaches 0.958 / 0.766 at ECE 0.061, in about a second per request (docs/results/weight-classes.md).

Those are our own runs. The public items have been consulted throughout development, so they are a development suite rather than an independent test; both configurations are filed on JevBench and queued by its author (issues #3 and #5).

Fine-tuning turned out to be a trap for general use: the adapters trained here improved data that looked like their training data and cost accuracy on long, ambiguous inputs, and prompt optimisation with GEPA did the same in miniature. The numbers, the controls and the two prompt changes that did transfer are in docs/results/frozen-vs-trained.md; the public-item comparison is in docs/results/jevbench-public.md.

Every experiment, in order, with its verdict: docs/results/README.md. That index is the honest version of this section: what was tried, what won, and what was thrown away.

How it works, in one paragraph

The state is run through the model once and its internal cache is kept. Every question is then run as a separate branch that can see the state but not the other questions, all in the same forward pass. Instead of letting the model write an answer, we look at what it would say next, keep only the answer labels (A, B, C …, and a lettered pair for yes/no), and turn those scores into percentages. Each question is asked twice inside that same pass, with its options in two different orders, and the two readings are averaged, which is the cheapest accuracy and honesty we found. A single "temperature" number, fitted on labelled data, can make the percentages honest on your own data. There is no decoding loop, no reasoning and no escalation anywhere on that path: one request is one forward pass, which is what keeps it under 300 ms. The details, the design trade-offs, and the mapping to the Jev write-ups are in docs/ARCHITECTURE.md; why the fast pass is the whole product is in docs/VISION.md.

Good to know

Works with Qwen3.5 (default, handles images), Qwen3, and Qwen3-VL checkpoints. Any size that fits your GPU; --model Qwen/Qwen3.5-0.8B runs on very small cards.
The browser demo runs all questions of a request in one batched forward pass, but re-reads the state for every question and does not cache it between requests, so it is fine for a handful of questions, not hundreds.
A choice question can have up to 26 options; score can have 2 to 10 levels.
--device mps runs the server on an Apple Silicon GPU, --backend sglang on an SGLang server; --permutations N sets how many option orders each question is read in.
The model is not magic: check its answers on a handful of your own examples before trusting it, and use the confidence numbers to route uncertain cases to a person.
reflex.think lets a model reason before the labels are read. It is an offline tool for teacher labelling and experiments only, never a serving mode; the server has no flag for it.
Tests: uv sync --extra dev && uv run pytest (the GPU tests download small models).

Credits

Built after reading TypeSafe's Jev announcement and docs, and Archer Hume's Jev's architecture, unmasked. Models by Qwen. Not affiliated with TypeSafe.

License

MIT. Do whatever you like with it. The Qwen model weights carry their own (Apache-2.0) license.