Architecture

A serving strategy around an off-the-shelf instruct model — by default Qwen3-0.6B (28 layers, d_model 1024, 16 Q / 8 KV heads, vocab 151936, tied embeddings, 0.596B params, of which 0.156B is the tied embedding matrix).

The question this was built to answer: how much of a purpose-built decision model's advantage is the model, and how much is the way it is served? The measurements below say most of the latency is serving structure and reproduces without training anything. Calibration is the part that does not.

The request path

POST /v1/systemone
  │
  ├─ build_prefix(state)   "<|im_start|>system … <document>{state}</document>"
  │  tokenize, check limits                              ← nothing has hit the GPU yet
  │
  ├─ model(prefix, cache)                                ← expensive, once
  │     28 layers write K/V into the cache.
  │     The logits from this call are discarded; we only wanted the cache.
  │
  └─ per question:
        build_suffix(q)    "\nQuestion: …\nOptions:\nA) …\n<|im_end|>
                            <|im_start|>assistant\n<think>\n\n</think>\n\n"
        model.model(suffix, cache)  → hidden states
        hidden[:, -1:, :]           → last position only
        lm_head(hidden)             → 151936 logits
        sum mass over each answer's token group, calibrate, softmax
        trim_prompt_cache(cache)    → rewind to the end of the document

The empty <think></think> block is prefilled so Qwen3 treats its reasoning as finished and the next token is the answer.

trim_prompt_cache is a correctness property, not an optimisation: question k+1 never sees question k's tokens, so adding a question cannot change another question's answer.

Why it is fast

No decode loop. The dominant cost in normal LLM serving is autoregressive decoding: one full forward pass per output token, memory-bandwidth-bound, arithmetic intensity ≈ 1. Here every answer is a single logit read, so the whole system stays in the prefill regime. Structured output via JSON schema still pays the decode loop; it only masks which tokens are legal at each step.

The document is encoded once. For a document of P tokens and Q questions of S tokens: P + Q·S instead of Q·(P + S). The ratio tends to Q as P ≫ S. Measured, P = 1266, S = 31, M4 Pro 16-core GPU:

questions	total	ms/question	tokens vs. one-call-per-question
1	269 ms	269.5	1.0x
4	331 ms	82.6	3.7x
16	585 ms	36.6	11.8x
32	882 ms	27.6	18.4x

≈ 249 ms fixed for the document, ≈ 20.3 ms marginal per question. Peak memory for the 32-question request: 1.9 GB, of which 1.2 GB is the weights. Prefill throughput 2·N·T/t ≈ 6.1 TFLOP/s. KV cache 112 KB/token.

The answer space is closed at the logit level. 151,934 of the 151,936 logits are never read.

raw_mass is read off the grouped answer mass before any correction is applied, which is why it tells you whether the answer format landed rather than whether the answer is right.

Two things that bit, and the fix

The LM head over every position. model(ids, cache)[0, -1] materialises a tensor of shape (1, S, 151936). At S = 2149 that is one 653 MB tensor, of which we keep one row. Across a 24-question request: 3.6 GB — enough, on unified memory, to push the machine into swap rather than raise an allocation error. Slicing the hidden state before the head makes it 14.6 MB total and also skips the ~26% of per-question FLOPs that live in the embedding matrix. The same applies to the document pass, where the logits are discarded entirely: call model.model, the body, not model.

Ignoring what trim_prompt_cache returns. It reports how many tokens it actually removed, and returns 0 for a cache it cannot trim. A short trim would leave one question's tokens visible to the next, silently breaking the independence the design rests on. The return value is now checked.

Reading one token per answer. Asked a yes/no question, Qwen3-1.7B put p=0.675 on "y" and only p=0.117 on "yes". Reading "yes" alone lost three quarters of the evidence, and since "no" scored 0.194 it inverted the answer. An answer is now a group of tokens (yes/Yes/YES/y/Y) and its probability is the total mass on the group.

Both were found by raw_mass, which is why it is reported on every answer.

What training would buy that serving cannot

1. Scoring is indirect. We read the identity of a letter token, so the model has to bind letter → option in-context and then emit the letter. A trained readout instead scores the option slots themselves from the decision position. Both approaches are listwise — the options are read together, so they can inform each other — which an external probe of Jev demonstrates: appending an irrelevant fifth option shifted the log-odds between two existing options by −0.28 (all ten randomised blocks), which fixed independent per-option logits under an unchanged softmax cannot do. The difference is the binding step, and it is why 255 options are workable there and 26 here.
2. No training objective for calibration. Post-hoc Platt scaling corrects a global bias; it cannot make uncertainty mean something per question.
3. Questions run serially. 20.3 ms marginal for 31 tokens ≈ 1500 tok/s against 5100 tok/s on the document — that gap is per-call overhead on a small matmul, not FLOPs. Stacking question suffixes along the batch dimension over the shared prefix is the largest remaining speedup.