Evaluation protocol

Fixed data and experiment

The public catalogue contains 24 invented specialist workflows. The dataset contains 72 synthetic requests with manually assigned labels: 24 clear routes, 12 near-neighbour routes, 12 ordinary no-skill requests, 12 ambiguous or unsupported tasks and 12 adversarial or distracting requests. Adversarial cases are harmless routing tests, not attempts to exploit an external service.

Twelve cases are marked development and 60 test before any live call. The labels, catalogue, category membership and split are fixed. A missing downstream artefact does not itself prevent choosing a workflow; the evaluation concerns routing, not execution. Unsupported specialist work requires review, whereas an ordinary reply needs no skill.

Run the development split first. Inspect every development case. Any policy change must be documented, preserving the original complete results and rerunning the development split. Then freeze the policy and run the entire dataset once, reporting development and test separately. Never tune on test failures without explicitly declaring that the test set has been reused. A small synthetic holdout is not evidence of production performance.

The initial policy is version 1.0 with jev-1.12, batch size 8, two survivors per batch, four workers, confidence 0.65, winner probability 0.70, margin 0.20, specialist need 0.70, no-skill need ceiling 0.30, ambiguity veto 0.50 and candidate fit 0.65. These values are conservative starting settings, not fitted calibration estimates.

The lexical baseline uses token-set cosine overlap between the request and each skill's name plus description. A fixed stop-word list removes common function words. A top score below 0.08 means no skill; a gap below 0.02 means review; otherwise it selects the best match. Scores are not probabilities. The baseline has no external calls and is not tuned using the evaluation labels.

Metrics

• Accuracy: exact agreement with the expected skill, no-skill or review label, divided by every case. API errors never count as correct, even on an expected-review case.
• Wrong-route rate: any selected skill unequal to the label, divided by every case. This includes needless loads and unsafe guesses on expected-review cases.
• Wrong specialist routes: a different skill loaded on a specialist-labelled request. A separate counter records specialist requests incorrectly marked no skill.
• Needless-load rate: selected skills on no-skill labels, divided by the number of no-skill labels.
• Abstention rate: all review outcomes divided by every case. API errors are included and also reported separately.
• Automatic accuracy: exact agreement among non-review decisions only. Read it alongside abstention and overall accuracy to avoid rewarding excessive abstention.
• Latency: elapsed wall-clock milliseconds around routing, including network, batch scheduling and local processing. Report median, nearest-rank p95, maximum and sum. Per-call timings are retained separately. The sum of parallel call times is not request latency.
• Tokens: sum API-returned input and output counts across every call. If any count is missing, the corresponding total is null and a separate known subtotal remains available.
• Estimated input cost: input tokens times USD 0.042 per million, based on TypeSafe's homepage price of USD 42 per billion read on 16 September 2026. This is a vendor price assumption, not a bill. Output, taxes, discounts and unreported usage are excluded. Incomplete usage produces a null estimate and a known-usage lower bound.

Every JSON artefact includes the dataset and catalogue snapshots and hashes, policy snapshot and hash, configuration, dates, requested and returned model identifiers, all cases, all call evidence, failures and metrics by category and split. Markdown is rendered from those captured values without inference. Repeating live commands produces a new experiment, not an exact replay guarantee.

The result validator checks case completeness and order, source hashes, selected-skill membership, distributions, finite timings and probabilities, token aggregation, model metadata and recalculated summaries. It checks internal consistency, not label truth or provenance authentication.

Recorded iterations

All runs took place on 16 September 2026. The initial jev-1.12 attempt returned service-boundary failures for all 12 development cases. A minimal diagnostic call reported HTTP 400; a separate jev-latest greeting probe returned the concrete identifier jev-1.13.0 with 273 input and 22 output tokens. Pinning that identifier gave 12/12 correct development results, using 42,091 input and 6,853 output tokens. Diagnostic calls are not dataset cases and are excluded from evaluation accuracy and run costs. The failed diagnostic had unknown usage.

The first full run, results/final.json, used policy 1.0 and scored 64/72 overall, including 52/60 on the initially untouched test split. It had eight unnecessary reviews, no wrong routes and no API errors. The lexical baseline scored 51/72. This original full artefact is preserved, despite its filename; it is not the final policy result.

After inspecting all failures, policy 1.1 revised only the specialist-need and ambiguity questions. It clarifies that document-level structured work can require a specialised procedure even when the output is prose, and that missing execution details do not necessarily prevent workflow selection. The thresholds, catalogue, labels and splits stayed unchanged. This is test-informed revision: subsequent results on the published dataset are exploratory and the split named test is reused, not held out.

The complete rerun, results/final-v1.1.json, scored 68/72, with 18 reviews, no wrong routes and no needless loads. Three reviews were unnecessary (c24, n06, a08). One batch response for n12 failed client validation and was not selectively retried; its raw response body was intentionally not retained, so the precise invalid field is unknown. Seven previous unnecessary reviews disappeared, two new unnecessary reviews appeared and one case suffered that validation failure. Changing prompts and varying responses prevent a clean causal or statistical improvement claim.

The final run's 287 attempted calls returned 255,897 known input tokens and 40,731 known output tokens. Missing usage from the validation failure makes complete totals and full input-cost estimates null. Known input usage corresponds to a lower-bound estimate of USD 0.010747674 at the stated vendor price. The earlier error-free full run returned 252,752 input and 41,082 output tokens, estimating USD 0.010615584 for input.

Scope of claims

Observed repository results measure only this router on these labels. TypeSafe's descriptions of calibrated decisions and its published cookbook results are company claims or separate experiments. They are not accuracy, calibration or latency findings from this repository. Typed output prevents invalid shapes, not incorrect valid decisions.

There is no downstream agent arm, no real skill execution, no production traffic, no billing reconciliation, no confidence interval, and no repeated-run calibration study. The live fixture is larger than the configured batch size but smaller than the API's maximum choice list. Only the offline tests exercise very large catalogues.