Limitations & methodology

TMGoat is useful, not perfect. In the spirit of an honest benchmark, here is what it does not claim and where the sharp edges are. If you're evaluating TMGoat critically, start here — we'd rather tell you than have you find out.

Ground truth is expert-authored, not universal

Each fixture's answer key (solution/threat-model.yaml) is a curated reference model, not an absolute truth. Threat modeling is judgment work: reasonable experts disagree on what counts as a distinct threat, what's in scope, and what severity applies. We mark the planted design flaws we deliberately built in, and separate acceptable_threats (legitimate findings we won't penalize) from false_positive_guidance (claims the inputs actually refute). A tool can surface a real threat we didn't enumerate — that's a gap in the key, not necessarily a tool error. Treat recall against planted threats as the primary signal.

The corpus is synthetic

Fixtures are hand-built teaching artifacts, not scraped from real companies. They are deliberately smaller and more internally consistent than real systems, and they follow a regular file layout. This makes them learnable and scoreable — but a tool that does well here has proven it can reason over clean, bounded inputs, not that it survives a messy 200-service estate with contradictory tribal knowledge. The adversarial fixtures (docs that lie vs. code that tells the truth) are our attempt to inject real-world mess; most fixtures are cleaner than production.

The bundled scorer is deliberately simple

harness/score.py's default matcher is lexical (concept-overlap + STRIDE), tuned to credit clear matches and reject category-only coincidences. It is reproducible and free, but it is blunt: it can over-credit findings that share vocabulary with the reference and under-credit findings phrased differently. An optional LLM judge gives higher fidelity but is non-deterministic and depends on an external model. For any published comparison, state which matcher you used. Don't read three-significant-figure precision into either.

Difficulty labels are designed, not empirically calibrated

The three axes (architectural_complexity, threat_subtlety, input_completeness) are authored tags. We believe "difficult" fixtures are subtler, but we have not run a large human/tool study to prove the tiers are monotonic. Use the axes as descriptions of intent, not validated difficulty scores.

Coverage is intentionally bounded

The corpus centers on cloud/web/microservice architectures analyzed through a STRIDE lens, with compliance framings (PCI, HIPAA, PSD2, SOC 2, …) as mapping targets. It is thin on: privacy-specific modeling (LINDDUN), AI/ML and agentic-system threats, deep supply-chain scenarios, mobile, and true embedded/firmware. Sectors are tags for sourcing diverse architectures, not the variable under test — the real axes are intake-type × difficulty. Contributions that widen coverage are very welcome (see CONTRIBUTING.md).

Contamination & why some answers are held out

Public answer keys next to public inputs get ingested by tools and models over time, which erodes a benchmark's meaning (Goodhart's law). To keep scores honest, the corpus is split:

• Practice / dojo tier (easy + moderate): full solutions published, so you can learn and self-check.
• Held-out benchmark tier (the difficult fixtures): inputs are public, answer keys are not — they live in a private vault, and scoring happens through a submission flow (see benchmark/README.md). No tool — including the maintainers' — is scored against answers it could have seen.

Even so, once inputs are public they can be memorized. We version the corpus and may rotate held-out fixtures over time; cite the corpus version with any result.

Conflict of interest, stated plainly

TMGoat is built by Virantis, which also builds a commercial threat-modeling product. The benchmark is open (Apache-2.0), the ground truth is inspectable for the practice tier, and the held-out tier is scored by a fixed harness rather than by us. We would rather compete on an open yardstick than a private one — but you should weigh the source, and we encourage independent fixtures and independent scoring.