TMGoat

A benchmark for threat-modeling tools and a dojo for threat-modeling engineers: 30 realistic systems across 10 sectors, each with deliberately-planted design flaws, a hidden answer key, a graded challenge, a reference solution, and a scoring harness.

Like WebGoat / TerraGoat, these designs contain intentional weaknesses. Teaching and test material — do not copy them into production.

Why this exists

Most "sample architecture" sets show a tool running but can't tell you whether it was right. TMGoat makes the expected output first-class and ships a harness/ that scores any tool — or any engineer — on recall (did you find the real design threats?) and precision (did you avoid noise?). The reference solution in every result/ is expert-curated as the model answer for each fixture.

Two tracks, one corpus

• Engineers (dojo): open a challenge, threat-model from inputs/, use hints.md if stuck, self-score, then study result/threat-model.md. Earn Bronze → Silver → Gold. Start with GUIDE.md.
• Tools (benchmark): run the tool over inputs/, export findings, score with harness/. Publish recall/precision against the reference. (A reference adapter ships in harness/adapters/; bring your own for other tools.)

Two tiers: practice (open) + benchmark (held-out)

To keep scores meaningful, the answer keys aren't all public (see LIMITATIONS.md on contamination):

• Practice / dojo — the 20 easy + moderate fixtures. Full solutions ship in-repo (solution/, result/, hints.md). Learn, self-check, and score locally with harness/score.py.
• Benchmark — the 10 difficult fixtures. Inputs are public; answer keys are held out in a private vault and scored through a submission flow so no tool is graded on answers it has seen. See benchmark/README.md.

Repository layout

TMGoat/
├── GUIDE.md            how to threat-model a challenge (start here)
├── skills-matrix.md    what each fixture trains + the belt path
├── catalog.md          all 30 fixtures (ids, intake types, planted counts)
├── coverage-matrix.md  the 12 intake types × fixtures design
├── schema/             JSON Schemas for manifest.yaml and threat-model.yaml
├── harness/            score.py + a reference tool adapter + generators (render_diagram, md_to_pdf, rollout)
└── fixtures/<NN-sector>/<tier>/
    ├── README.md                  the challenge brief
    ├── manifest.yaml              tags: sector, axes, intake types, capabilities
    ├── inputs/                    what you analyze: diagrams/ docs/ code/ iac/
    ├── hints.md                   3 graduated hints          (practice tier only)
    ├── solution/threat-model.yaml the answer key             (practice tier only; held out for difficult)
    └── result/                    the reference model + score (practice tier only)

Difficult fixtures ship inputs/ + a neutral brief only; their solutions live in the held-out vault.

The corpus

10 sectors × 3 tiers = 30 fixtures · 239 planted threats · all 12 supported intake types, in many real file formats (drawio, png/jpg, pdf, terraform/cloudformation/bicep/kubernetes/docker-compose, openapi, otm, mtmt, lucidchart, confluence, virantis-project, …). The difficult tier raises the bar with chained attack paths and adversarial inputs — architecture docs that disagree with the code/IaC, so you learn to trust what's actually deployed. (Difficult solutions are held out; see below.)

How difficulty works (the important idea)

Difficulty is threat subtlety, not size — a 6-component app can hide a nastier flaw than a 30-component one. Three independent axes are tagged in each manifest.yaml:

Axis	Question it tests	Values
architectural_complexity	Can you recover a big design?	low / medium / high
threat_subtlety	Can you reason to non-obvious threats?	obvious / design / subtle
input_completeness	Are you robust to poor / contradictory input?	rich / partial / adversarial

A tool (or junior) that just runs STRIDE on each box misses the race condition, the fail-open control, the over-trusted channel, the doc that lies. Those design threats are where the points — and the risk — live.

Scoring

Practice tier (easy/moderate) — score locally against the published key:

python harness/score.py --expected fixtures/<sector>/<tier>/solution/threat-model.yaml --findings your-findings.json

Benchmark tier (difficult) — keys are held out; submit via benchmark/README.md.

Bronze = all planted found · Silver = + the chained/subtlest threat · Gold = + mitigations mapped to standards + residual risk.

Status

All 30 fixtures are built with expert-authored reference models. The 20 practice fixtures ship full solutions; the 10 difficult fixtures are the held-out benchmark tier. Scores from the bundled lexical matcher are indicative, not absolute — see LIMITATIONS.md for methodology, known gaps, and how ground truth is authored. Sectors are tags for sourcing diverse architectures, not the thing under test.

About

TMGoat is built and maintained by Virantis — autonomous, continuous threat modeling powered by agentic AI. It's open source (Apache-2.0) and tool-neutral: score any threat-modeling tool against it. Contributions welcome — see CONTRIBUTING.md.