dinocorpus leaderboard

Generated by python corpus/leaderboard.py from corpus/scorecards/. Do not edit by hand.

Read all four columns or none of them. Recall alone rewards a detector that flags everything, which is why the false-alarm rate sits beside it and why nothing here is sorted by score. blind is recall on defect classes the detector has no rule for, under STRICT scoring: it must name an item the defect was actually planted in, so an unrelated warning firing by chance is not a catch.

Recall figures are UPPER BOUNDS. The corpus items are synthetic and cleaner than real benchmark items; see basepool.py.

A score means nothing without its split. Splits and their label commitments: SPLITS.md.

detector	split	covered	blind (strict)	false alarms	answered
dinostomp 0.63.0	dev	100.0%	11.1%	15.7%	all
dinostomp 0.63.0	heldout-2026-08	100.0%	11.1%	5.0%	all
dinostomp 0.63.0	heldout-2026-08b	100.0%	11.1%	6.5%	all
dinostomp 0.63.0	heldout-assets-2026-08b	100.0%	11.1%	6.3%	all
dinostomp 0.63.0	heldout-shapes-2026-08	95.0%	6.4%	2.7%	all

covered is recall on defect classes the detector has a rule for; blind is recall on classes it does not, which is the column that says whether a detector generalises past its own rule list.

Submitting

Run your detector over a split's instances and send the JSON. Format, scoring rules and what counts as a catch: README.md. Submissions are scored with corpus/score.py, which refuses malformed files rather than publishing a 0% caused by a typo.