References

Where the borrowed parts come from.

This file exists because of one label. The threshold table marks three values convention, defined as "a value the surrounding literature uses, defensible by citation" — and for a while nothing here cited anything. An appeal to convention with no reference is an unfalsifiable claim, which is precisely what this tool exists to object to. Every method the battery borrows and every threshold that leans on prior art is named below, so a reader can disagree with the source rather than with an assertion.

Ordinary caveat: a citation licenses a method, never a setting. Where a number was chosen rather than derived, the threshold table says judgment, and that is 34 of 43 of them. Nothing here converts a judgment call into a citation.

Statistical methods the battery uses

method	used by	source
Wilson score interval	every reported accuracy	Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. JASA 22(158), 209–212.
KR-20 reliability	fleet-reliability (P1)	Kuder, G. F. & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika 2(3), 151–160.
Point-biserial item discrimination	item-discrimination (P2)	Standard classical test theory; see Crocker, L. & Algina, J. (1986). Introduction to Classical and Modern Test Theory, ch. 14, for the rest-score correction used here.
Fixed-margins null by swap randomisation	item-discrimination (P2)	Connor, E. F. & Simberloff, D. (1979). The assembly of species communities: chance or competition? Ecology 60(6), 1132–1140. The 2×2 checkerboard swap is the standard sampler for a binary matrix with both margins held.
McNemar's test	order-stability (P9), prompt-stability (P11)	McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157.
Paired bootstrap resampling	ordering-noise (P6), typed superiority claims	Efron, B. (1979). Bootstrap methods: another look at the jackknife. Annals of Statistics 7(1), 1–26.
Mutation testing	witness-coverage (W1)	DeMillo, R. A., Lipton, R. J. & Sayward, F. G. (1978). Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41. The mutation gauntlet applies this to scorers: a witness suite that no mutant survives is a suite that constrains the scorer.
Expected calibration error	overconfident (R23)	Naeini, M. P., Cooper, G. F. & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. AAAI 2015. Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017, for the equal-width binned form used here.
AUROC as the Mann-Whitney U	confidence-blind (R24)	Hanley, J. A. & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36.

The four convention thresholds, and what they lean on

• kr20_min = 0.5 — reliability coefficients are conventionally read against Nunnally's bands (Nunnally, J. C. (1978). Psychometric Theory, 2nd ed.), where 0.7 is the usual bar for applied work and lower values are "acceptable for early research". 0.5 is deliberately below the textbook bar, because a 0.7 gate would skip almost every fleet anyone actually runs. That choice is judgment; the scale it is read against is the citation.
• negative_discrimination = -0.2 — item analysis conventionally treats a negative point-biserial as a candidate key error, with practical screening bands around −0.2 to 0 (Crocker & Algina, above; Ebel, R. L. (1954). Procedures for the analysis of classroom tests. Educational and Psychological Measurement 14(2), 352–364).
• shortcut_z = 3.0 — three standard deviations against a null is the usual bar for calling a surface feature real rather than sampling noise. Convention, not derivation; the null it is measured against is analytic and described in METHODOLOGY.
• near_dup_bits = 5 — a Hamming distance of 5 in a 64-bit perceptual hash is the value the near-duplicate literature generally uses. It is the weakest of the four: it is a citation to common practice rather than to a specific result, S15 warns rather than gates because of that, and benchmarks/cifair/compare.py --sweep is the measurement that says what other values would buy and cost on real data.

Benchmarks this repository audits

Fetched unmodified by python benchmarks/fetch.py, which prints the SHA-256 of exactly what it downloaded. Nothing is vendored.

dataset	source	licence
GSM8K	Cobbe, K. et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168	MIT
MMLU	Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding. arXiv:2009.03300	MIT
TruthfulQA	Lin, S., Hilton, J. & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958	Apache-2.0
HellaSwag	Zellers, R. et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830	MIT
ARC (Easy and Challenge)	Clark, P. et al. (2018). Think you have Solved Question Answering? arXiv:1803.05457	CC-BY-SA-4.0
MMLU-Pro	Wang, Y. et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574	MIT
CommonsenseQA	Talmor, A. et al. (2019). CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. arXiv:1811.00937	MIT
OpenBookQA	Mihaylov, T. et al. (2018). Can a Suit of Armor Conduct Electricity? arXiv:1809.02789	Apache-2.0
BoolQ	Clark, C. et al. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv:1905.10044	CC-BY-SA-3.0
WinoGrande	Sakaguchi, K. et al. (2020). WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641	CC-BY
SciQ	Welbl, J., Liu, N. F. & Gardner, M. (2017). Crowdsourcing Multiple Choice Science Questions. arXiv:1707.06209	CC-BY-NC-3.0
MedMCQA	Pal, A., Umapathi, L. K. & Sankarasubbu, M. (2022). MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. arXiv:2203.14371	MIT
ARC-Challenge, as a third-party eval LOG	open-llm-leaderboard-old/details_Corianas__111m, file details_harness|arc:challenge|25_2023-07-19T13:48:53.093937.parquet. A real lm-evaluation-harness details file, fetched by benchmarks/lm-eval-import/fetch.py. Harness: Gao, L. et al. (2021). A framework for few-shot language model evaluation. Zenodo. Items: Clark et al. 2018, above.	CC-BY-SA-4.0 (items)
RACE	Lai, G. et al. (2017). RACE: Large-scale ReAding Comprehension Dataset From Examinations. arXiv:1704.04683	research use, per the authors
MuSR	Sprague, Z. et al. (2024). MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. arXiv:2310.16049	MIT
LogiQA	Liu, J. et al. (2020). LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. arXiv:2007.08124 (the lucasmccabe/logiqa copy)	CC-BY-NC-SA-4.0
MATH-500	Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv:2305.20050, a 500-problem subset of Hendrycks, D. et al. (2021), arXiv:2103.03874	MIT
DROP	Dua, D. et al. (2019). DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. arXiv:1903.00161	CC-BY-SA-4.0
MedQA (USMLE)	Jin, D. et al. (2021). What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv:2009.13081, via GBaker/MedQA-USMLE-4-options	MIT
AQuA-RAT	Ling, W. et al. (2017). Program Induction by Rationale Generation. arXiv:1705.04146	Apache-2.0
Iranian driving licence test	ckodser/Iran_Driving_licence_test. A statutory road-safety question bank, redistributed; no accompanying paper.	see the dataset card
NCLEX-style nursing items	InterCECTDev/nclex-nursing-questions. Nursing licensure practice items; no accompanying paper.	see the dataset card
2023 Chinese Pharmacist Licensure Examination	FreedomIntelligence/2023_Pharmacist_Licensure_Examination-Pharmacy_track, pharmacy track. A national professional licensing examination, redistributed.	see the dataset card
iris	Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics 7(2), 179–188. The UCI/scikit-learn lineage this repo pins differs from Fisher's table in two rows: Bezdek, J. C. et al. (1999). Will the real iris data please stand up? IEEE Trans. Fuzzy Systems 7(3), 368–369.	public domain

Findings against these datasets are in FINDINGS.md, series F. Each one is a defect in a specific artifact and none is a judgement of the work that produced it: these are the most-scrutinised benchmarks in the field, which is exactly why finding anything in them is worth publishing.

The human answer keys this repository is scored against

Two datasets here are not audit TARGETS but GRADERS: somebody else annotated them by hand, so they can say what this battery misses rather than what it finds. That is the only kind of evidence in this repo that the author did not also produce.

• MMLU-Redux 2.0 — Gema, A. P. et al. (2024). Are We Done with MMLU? arXiv:2406.04127. Manual re-annotation of MMLU items with error taxonomies. Scored in N-012; F-018 is the pair of items its annotators marked ok that the battery flagged.
• ciFAIR — Barz, B. and Denzler, J. (2020). Do We Train on Test Data? Purging CIFAR of Near-Duplicates. Journal of Imaging 6(6), 41. https://cvjena.github.io/cifair/. Hand-annotated near-duplicate pairs between the CIFAR-10/100 training and test sets, with judgment codes distinguishing exact duplicates from re-processed copies of one camera shot from merely similar images. Fetched by benchmarks/cifair/fetch.py, licensed CC-BY-SA, not vendored.
• XSTest v2 — Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F. & Hovy, D. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024. arXiv:2308.01263. Model completions each labelled by two annotators as full compliance, full refusal or partial refusal; the answer key for every refusal judge in audits/xstest-refusal and audits/xstest-refusal-guards (N-038). paul-rottger/xstest @ d7bb5bd7, CC-BY-4.0, fetched by the pod's build script, not vendored.

Artifacts audited outside the benchmark pods

The audits/ directory holds audits of things that are not eval pods: graders, refusal classifiers, agent harnesses, and one training corpus. Each is driven by audits/<name>/audit.py, which imports the subject's own code and runs it unmodified rather than reimplementing it, and nothing is vendored. Every subject is pinned by repository and commit so a reader can fetch the same bytes.

subject	source	pinned at
CUDA Agent	Dai, W., Wu, H., Yu, Q. et al. (2026). CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation. arXiv:2602.24286. ByteDance Seed / Institute for AI Industry Research, Tsinghua University.	repo BytedTsinghua-SIA/CUDA-Agent @ 473025c8; no licence file in the repo
CUDA-Agent-Ops-6K	The 6,000-sample synthesised training corpus released with the above, BytedTsinghua-SIA/CUDA-Agent-Ops-6K. A TRAINING set, not an eval: it carries no answer key, and the audit reads it as data at rest.	CC-BY-4.0, per the dataset card
KernelBench	Ouyang, A., Guo, S., Arora, S., Zhang, A. L., Hu, W., Ré, C. & Mirhoseini, A. (2025). KernelBench: Can LLMs Write Efficient GPU Kernels? arXiv:2502.10517	repo ScalingIntelligence/KernelBench @ 423217d9, MIT. Used here as the REFERENCE corpus for N-031, not as an audit target
SWE-bench	https://github.com/swe-bench/SWE-bench	commit c7fd5abffe0b2086a8bb9389d23c47d930ef571f
DeepSWE	https://github.com/datacurve-ai/deep-swe	commit 435ee89ec2f2e2289f33b0da4f992f0b7b7266b9
HarmBench	https://github.com/centerforaisafety/HarmBench	commit 8e1604d1171fe8a48d8febecd22f600e462bdcdd
JailbreakBench	https://github.com/JailbreakBench/jailbreakbench	commit 23dbdf6b19650521604456229bc1d9c4156c85c1
StrongREJECT	https://github.com/alexandrasouly/strongreject	commit f7cad6c17e624e21d8df2278e918ae1dddb4cb56
AISafetyLab	https://github.com/thu-coai/AISafetyLab	commit 3fd15737dee5c6858466162d6a68e6cb6f996b27
garak	https://github.com/NVIDIA/garak	commit b1e0101a177981cc423455448fd745be88b87916
WildGuard (judge, not target)	Han, S., Rao, K., Ettinger, A. et al. (2024). WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. arXiv:2406.18495. Allen Institute for AI. Run as a refusal judge in audits/xstest-refusal-guards with the model card's prompt.	model allenai/wildguard, Apache-2.0; outputs committed and pinned by sha256
StrongREJECT fine-tuned evaluator (judge, not target)	Souly, A., Lu, Q., Bowen, D. et al. (2024). A StrongREJECT for Empty Jailbreaks. arXiv:2402.10260. Run as a refusal judge in audits/xstest-refusal-guards with the reference template and scoring.	adapter qylu4156/strongreject-15k-v1 on google/gemma-2b (Gemma Terms of Use); template and scoring from dsbowen/strong_reject @ 7a551d5b
Jev 1.13 (judge and examinee, not target)	TypeSafe AI (2026). Introducing System One Models and Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev. A decisions model called through provider: typesafe and provider: jev.	jev-1.13.0 as reported by the endpoint; $0.042 per million input tokens per TypeSafe's docs (read 2026-09-19)

The lower seven rows record the artifact each audit actually ran against, which is what its findings are about. Several have accompanying papers that are not cited here yet; a commit is the stronger identifier for an audit of code, and the gap in the paper column is a gap rather than a claim that none exists.

The same scope note as the benchmark table applies, and applies harder here. CUDA Agent is a speed result and no audit in this repo checks a speedup: F-047 to F-049 and N-031 are about a released dataset and a released verification script. N-031 in particular records that their decontamination claim SURVIVED an independent check, which is the kind of result that only gets published if negative results are in the ledger by default.

Failure modes the battery was built against

The trials are not enumerated from the check registry. They are drawn from described eval-defect classes plus this project's own adversarial reviews, so that a check and the defect proving it come from different places.

failure mode	checks aimed at it	source
Answer leakage and train/test contamination	answer-leak (S2), canary-present (S8), canary-regurgitated (S10), corpus-overlap (S11)	Elangovan, A., He, J. & Verspoor, K. (2021). Memorization vs. generalization: quantifying data leakage in NLP task design. EACL.
Annotation artifacts a model can exploit without the task	surface-shortcut (S9), blind-solvable (R13), input-blind (R15)	Gururangan, S. et al. (2018). Annotation artifacts in natural language inference data. NAACL.
Shortcut learning generally ("Clever Hans")	the same family	Geirhos, R. et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 665–673.
Option-order sensitivity in multiple choice	order-stability (P9), position-bias (S3)	Pezeshkpour, P. & Hruschka, E. (2024). Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions. NAACL Findings.
Prompt-format sensitivity moving rankings	prompt-stability (P11), ranking-stability (P12)	Sclar, M. et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. ICLR.
Benchmark label errors	item-discrimination (P2), unanimous-wrong (P5), conflicting-keys (S7)	Northcutt, C. G., Athalye, A. & Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. NeurIPS Datasets and Benchmarks.
LLM-judge bias: verbosity, position, self-preference	judge-bias (J2), judge-self-preference (J4)	Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.
Reproducibility and under-reported eval conditions	the coverage line, the drift boundary, engine-drift (R19)	Reproducibility checklists in the NeurIPS/ML community, and Kapoor, S. & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4(9).

What this repository does not borrow

Stated because absence is easy to misread as oversight:

• No LLM-as-judge for the battery's own verdicts. Every core check is deterministic or a stated statistic. A judge is something dinostomp audits (J1–J4), never something it asks.
• No learned model anywhere in the checks. Nothing here needs a GPU and nothing needs a network; stomp is offline by construction.
• No claim about construct validity. Every report carries measures the intended construct: NOT ESTABLISHED BY DINOSTOMP, and the reason is in METHODOLOGY: construct validity is argued, not computed. Messick, S. (1995). Validity of psychological assessment. American Psychologist 50(9), 741–749, is the standard statement of what that argument requires, and it is not something a linter can supply.