Exploratory evaluation
September 17, 2026 · View on GitHub
This English-only pilot was run on 2026-09-17 with Qwen3-4B-Instruct-2507 4-bit MLX on Apple M3 Pro: 24 calibration examples and 32 separate synthetic test examples. All 56 messages are in English and AI-authored; intended labels are not independently human-validated. Eight messages were translated before this fresh run. The scenarios were evaluated previously, so this is not a new unseen test set.
| Method | Correct | p50 | p95 | Mean generated tokens |
|---|---|---|---|---|
| Prompted JSON probabilities | 29/32 | 1524.3 ms | 1631.1 ms | 34.8 |
| One-token answer | 28/32 | 594.0 ms | 659.6 ms | 1.0 |
| Direct scores | 28/32 | 595.6 ms | 665.0 ms | 0.0 |
These are fresh measurements on the English-only dataset. Direct scoring reads scores for every supplied option without generating an answer probability object. The one-token baseline returns only a label. This small run establishes neither general accuracy gains nor robust latency differences. It covers local MLX only; no API-provider performance is inferred.
Probability quality and abstention
| Method | NLL (lower is better) | Brier sum (lower is better) |
|---|---|---|
| Prompted JSON | Not reported: invalid outputs | Not reported: invalid outputs |
| Raw direct scores | 1.725 | 0.235 |
| Calibrated direct scores | 0.523 | 0.205 |
Temperature scaling was fitted on the 24 calibration examples, with temperature 3.497785. It leaves winning labels unchanged. The fixed threshold of 0.8 accepted 31/32 test cases, with 3 accepted errors. Calibration and threshold application occurred after the scoring timer stopped. Scores do not guarantee correctness.
Protocol and limits
- Revision:
50d427756c6b1b2fe0c0a10f67fbda1fc8e82c1b. - Four categories, eight test examples each. One question per call.
- One warmed repetition per example/method; shuffled order with seed 1729. No persistent prompt cache or concurrency.
- Wall time includes prompt preparation, tokenization, inference, readback and interpretation. Loading, downloads and report serialization are excluded.
- JSON uses greedy prompting without grammar constraints, with a 192-token cap. Direct and one-token modes share their prompt; JSON has different instructions and longer input. The one-token control avoids an unused additional decode step.
- Invalid outputs: 1. Every observation is retained; the slowest recorded call took 24.6 seconds. Percentiles are not worst-case guarantees.
- The report slider explores saved outcomes; it does not validate a new threshold. Larger real tasks and repeated trials are needed for deployment or broad claims.
Evidence and reproduction
Recorded measurements · Calibration artifact · Offline report · Dataset
The report records the English-only dataset hash, raw predictions, timing, and
source hashes from this run, before the package was renamed to DecisionBridge.
The rename changes package paths and report branding, not inference prompts or
scoring logic. No historical scores were relabeled as measurements
on translated inputs. For the dependency environment, install
requirements-macos.lock.txt into Python 3.12, then install this project with
--no-deps -e ..
.venv/bin/decisionbridge benchmark --offline --dataset examples/issue_triage.json --output reports/rerun.json
.venv/bin/decisionbridge report reports/rerun.json --output reports/rerun.html
Use domain examples with separate calibration and test splits. Neither generated probability text nor token scores should be assumed reliable without evaluation.