TMGoat Benchmark (held-out tier)

July 10, 2026 · View on GitHub

The difficult fixtures are the benchmark tier. Their inputs/ are public so anyone can run a tool over them — but their answer keys are not published. They live in a private vault, and scoring happens here, against a key your tool has never seen. This is what keeps a TMGoat score meaningful over time (see ../LIMITATIONS.md on contamination).

The easy and moderate fixtures are the open practice tier: full solutions ship in each solution/threat-model.yaml, so you can learn and self-score with harness/score.py directly.

What you submit

For each held-out fixture, run your tool over everything in inputs/ and export a findings.json — a JSON list of objects shaped like:

[
  { "title": "Cross-tenant data access via unscoped object lookup", "stride": ["I", "E"], "target": "tenant-api" },
  { "title": "Platform-wide token forgery from a shared signing key",  "stride": ["S", "E"], "target": "tenant-api" }
]

title is required; stride and target improve matching. If your tool emits a different native format, write a small adapter (see harness/adapters/ for a reference).

How to submit

  1. Fork the repo and add your findings under submissions/<your-tool>/<fixture-id>.json — see ../submissions/README.md for the exact format and the fixture ids.
  2. Open a pull request. A maintainer adds the score label, and the benchmark-score GitHub Action runs the fixed scorer against the private keys and comments your recall / precision back on the PR — the raw answer key is never exposed.
  3. Merged submissions are eligible for the public leaderboard.

The scorer, matcher, and metric definitions are the same ones in harness/ — nothing hidden except the answer keys themselves.

Maintainer setup (one-time)

The scoring workflow (.github/workflows/benchmark-score.yml) needs read access to the private key vault:

  • Secret KEYS_TOKEN — a fine-grained personal access token with Contents: Read-only on the virantisofficial/TMGoat-keys repo. Add it under Settings → Secrets and variables → Actions.
  • Label score — create a label named exactly score. Adding it to a submission PR triggers scoring (this gate is what keeps untrusted forks from running the workflow with the secret).

The workflow runs on pull_request_target but never checks out or executes PR code — it only reads the submitted JSON as data and clones the vault with KEYS_TOKEN.

Rules of the road

  • Feed your tool the inputs/ only. Don't try to source answer keys out-of-band; self-reported scores that can't be reproduced by the harness won't be listed.
  • Cite the corpus version with any published result — held-out fixtures may rotate over time.
  • The maintainers' own tool is scored by the identical harness, against keys it has not seen.