OpenVerification1 Eval

Model Cards

Starting with version 2.3.0.preview, the evaluation results are provided in the corresponding Model Card. The rest of this file is archived from earlier versions.

Output Predictions and Evaluations

See the model directory (in the GitHub Release binary) reexpress_mcp_server_model__v2_1_0/model_details/final_eval_output for the output predictions (all_predictions.jsonl), logs (version_2.1.0.log.txt), sorted possible label errors¹ (possible_label_errors.jsonl), and sorted predictions in the High-Reliability region ² (high_reliability.jsonl) for each eval split, as well as the model's calibration set. Graphs of the output are also saved in model directory.

We provide high-level summary statistics here. This is an evaluation of the underlying SDM estimator used in the MCP Server; the tool-calling LLM is not involved. In other words, the arguments to reexpress(user_question: str, ai_response: str) are directly the question and response from the benchmark dataset without translation by, nor composition with, a tool-calling LLM.

Version 2.1.0

Data Split	Marginal Accuracy	Admitted Accuracy	Admitted Proportion	Dataset Size
Calibration (not held-out)	0.93	0.98	0.73	120159
MMLU Validation (binary verification)	0.92	0.97	0.70	3036
OpenVerification1 5k Test	0.92	0.98	0.69	5000
MMLU-Pro-4-QA-GPT4o-Letters	0.88	0.95	0.54	5346
MMLU-Pro-4-QA-GPT4o-Explanations	0.87	0.94	0.60	5344

Version 2.0.0

Data Split	Marginal Accuracy	Admitted Accuracy	Admitted Proportion	Dataset Size
Calibration (not held-out)	0.92	0.98	0.72	74684
MMLU Validation (binary verification)	0.93	0.97	0.64	3036
OpenVerification1 5k Test	0.94	0.98	0.73	5000
MMLU-Pro-4-QA-GPT4o-Letters	0.87	0.94	0.45	5346
MMLU-Pro-4-QA-GPT4o-Explanations	0.87	0.93	0.60	5344

Version 1.2.0

(Versions 1.2.0 and earlier used the earlier formulation of the SDM estimator, with a different criterion for determining the admitted points.)

Data Split	Marginal Accuracy	Admitted Accuracy	Admitted Proportion	Dataset Size
Calibration (not held-out)	0.93	0.98	0.61	74684
MMLU Validation (binary verification)	0.93	0.98	0.47	3036
OpenVerification1 5k Test	0.93	0.99	0.63	5000
MMLU-Pro-4-QA-GPT4o-Letters	0.86	0.93	0.30	5346
MMLU-Pro-4-QA-GPT4o-Explanations	0.86	0.94	0.44	5344

Interestingly, with this version in particular, the models are sufficiently strong that a non-trivial number of annotation errors (i.e., irreducible/aleatoric error) is evident with the MMLU-Pro-4-QA datasets. These can be examined interactively using utils_graph_output.py. See the end of the training script for example use.

Version 1.1.0

Data Split	Marginal Accuracy	Admitted Accuracy	Admitted Proportion	Dataset Size
Calibration (not held-out)	0.92	0.98	0.62	28225
MMLU Validation (binary verification)	0.88	0.97	0.33	3036
OpenVerification1 5k Test	0.92	0.98	0.62	5000
MMLU-Pro-4-QA-GPT4o-Letters	0.74	0.96	0.09	5346
MMLU-Pro-4-QA-GPT4o-Explanations	0.84	0.95	0.31	5344

Legend

The "Admitted" instances are those for which the predicted probability from the SDM estimator for the verified class (i.e., label 1) is in the High-Reliability region. See the paper "Similarity-Distance-Magnitude Activations" for details. In the context of the MCP Server, these instances would have a confidence >= 90% in the main output.

As indicated in the table, the estimator correctly detects the instances that have a probability >= 0.9, even as the overall marginal accuracy varies substantially relative to the model's Calibration set.

Footnotes

1. These are instances that are in the High-Reliability region, but the predicted class does not match the ground-truth label. These are sorted descending by p(y | x). ↩

2. These are instances for which the confidence label of the main output from the MCP Server would be >= 90%. These are sorted descending by p(y | x). ↩