MedCalc-Bench (Inspect AI)

An Inspect AI implementation of MedCalc-Bench, a benchmark for evaluating whether large language models can perform clinical calculations from free-text patient notes.

Khandekar, Tsoukalas, Lin, et al. MedCalc-Bench: Evaluating Large Language Models for Medical Calculations. NeurIPS 2024 (Datasets & Benchmarks Track, Oral). arXiv:2406.12036 · original code

This is an independent port of the published benchmark to Inspect AI. The scoring logic mirrors the authors' reference check_correctness implementation; see Scoring for the one intentional difference. It is contributed upstream as inspect_evals #1765.

Why this matters, and for whom

Clinical calculation is the rare medical task with a verifiable right answer — a MELD score, a Wells criterion, a creatinine clearance — and it is exactly where a plausible-sounding wrong number from an LLM is dangerous. As models are pulled into clinical documentation and decision support, "does it actually compute the right value from the note?" is a capability that has to be measured before deployment, not assumed. MedCalc-Bench showed even strong models land near ~50% here (Khandekar et al., NeurIPS 2024), so the benchmark is a live signal, not a solved one.

Who needs an eval like this: model developers gating releases, AI-safety evaluators (this is built for the UK AI Safety Institute's Inspect framework, the same harness used for frontier-model capability testing), and health-AI vendors under regulatory pressure to show evidence rather than vibes. The eval itself is treated as safety-critical infrastructure: the scorer never executes model output as code (the reference implementation calls eval() — see Scoring), so a model can't manipulate its own grade. A trustworthy evaluator is a precondition for trusting anything it reports.

What it measures

Each sample gives the model a patient note and asks it to compute one specific clinical value — a lab-derived score, a physical quantity, a risk/severity score, a drug dosage, or a date (e.g. gestational age). The test set contains 1,000+ manually reviewed instances spanning 55 calculators in seven categories (lab test, physical, date, dosage, risk, severity, diagnosis).

Calculators are of two kinds:

• Rule-based (lookup / decision tables) — scored by exact match.
• Equation-based (formulas) — scored within a tolerance band the authors derived from the precision of the input measurements.

Usage

# Clone and install the pinned commit
git clone https://github.com/azrabano23/medcalc-bench-inspect
cd medcalc-bench-inspect
uv sync

# Run (chain-of-thought is the default, matching the paper's main setting)
uv run inspect eval src/medcalc_bench/medcalc_bench.py@medcalc_bench --model openai/gpt-4o-mini

# Direct-answer (no reasoning) variant
uv run inspect eval src/medcalc_bench/medcalc_bench.py@medcalc_bench \
    -T prompt=direct --model openai/gpt-4o-mini

# Quick smoke test on a few samples
uv run inspect eval src/medcalc_bench/medcalc_bench.py@medcalc_bench \
    --model openai/gpt-4o-mini --limit 10

Task parameters

Parameter	Values	Default	Meaning
prompt	zero_shot_cot, direct	zero_shot_cot	Chain-of-thought prompting (the paper's primary setting) vs. answer-only.

Dataset

Loaded from nsk7153/MedCalc-Bench-Verified, pinned to revision 591157b3343b4dda247294f9d929da4c75026fa8. This is a public, verified mirror of the NeurIPS 2024 release with the identical 14-column schema; the original ncbi-nlp/MedCalc-Bench-v1.0 is gated. The test split is used for evaluation.

The relevant columns are Patient Note, Question, Ground Truth Answer, Calculator ID, Calculator Name, Category, Output Type, Lower Limit, and Upper Limit. The limits define the acceptance band for equation-based answers.

Scoring

The model is prompted to return a JSON object ({"step_by_step_thinking": ..., "answer": ...} in CoT mode, {"answer": ...} in direct mode). The scorer extracts the answer field — robustly, falling back through embedded JSON, a loose answer: pattern, and finally the last number in the text — and applies the calculator's rule, keyed on Calculator ID exactly as in the reference:

• Date (IDs 13, 68): exact calendar-day match.
• Gestational-age tuple (ID 69): (weeks, days) extracted by regex and compared.
• Integer (IDs 4, 15–18, 20–21, 25, 27–29, 32–33, 36, 43, 45, 48, 51): exact match after rounding.
• Decimal (all others): correct iff Lower Limit ≤ answer ≤ Upper Limit.

The reported metrics are accuracy and stderr.

Intentional difference from the reference: the original code calls eval() on answer strings. This port parses numbers safely (stripping units, commas, $, and %) rather than executing model output as Python.

Validation

The scorer is validated by self-consistency against the real dataset: feeding each ground-truth answer back through its own calculator's rule must score CORRECT. This exercises every output-type path (date / tuple / integer / decimal) and confirms the calculator-ID groupings and tolerance parsing match the data. See tests/test_medcalc_scoring.py (the dataset-marked test); unit tests cover the scoring rules and answer extraction without network access.

uv run pytest                       # unit tests
uv run pytest -m dataset            # + real-data self-consistency

License

MIT (this Inspect port). The underlying MedCalc-Bench dataset and original code are released by their authors under their own terms — see the original repository.