Precision benchmark: zero false positives on a clean app

ptai's pitch is precision: a finding earns a VERIFIED badge only when a machine oracle re-runs the exploit and reproduces it, and third-party scanner output is held back until an oracle re-proves it. This page is the reproducible counterpart to that claim. It measures how much noise ptai emits on an app that has nothing to find.

It is the paired half of the honeypot catch-rate story: the honeypot measures recall on real bugs, this measures precision on a safe surface. Together they answer the only question a skeptical operator asks of a scanner: does it catch the bug without crying wolf on the same shapes?

Method

tests/cleanapp is a fixture web app with zero planted vulnerabilities. Its routes are deliberately attackable-looking but correctly implemented: a search box that parameterizes its query, a redirect that allowlists its target, a login that uses a real password check, a file route that sandboxes its path, comment rendering that escapes its output. Anything a scanner reports here is, by construction, a false positive.

benchmarks/clean_fp_sweep.py starts that app and runs ptai's full deterministic web sweep (--no-llm, no API key) against it, then prints the false-positive count and lists each finding so the source probe is obvious.

python benchmarks/clean_fp_sweep.py

Result

ptai: 0 false positives. A green run is the launch-credibility gate; a non-zero run names exactly which probe to fix. The same surface is also asserted in CI by the eight tests/cleanapp/test_clean_fp.py cases (reflected XSS, open-redirect, SQLi login, SQLi search, path traversal, type confusion, trusted header, stored XSS), so a regression that starts emitting noise fails the build.

You can watch the same thing in two minutes against the bundled vulnerable and hardened twin:

ptai demo

The hardened run reports 0 findings on the exact routes the vulnerable run flagged. The only difference between the two is the fix.

Compare it yourself

We do not publish competitor numbers we did not measure. Instead, here is the harness to run any scanner against the same zero-vulnerability fixture, so the comparison is yours, not ours.

Start the fixture:

CLEANAPP_PORT=4111 python -m tests.cleanapp.server

Then point a scanner at http://127.0.0.1:4111 and count what it reports. Everything it reports is a false positive, because the app has no vulnerabilities.

# OWASP ZAP (baseline scan)
zap-baseline.py -t http://127.0.0.1:4111

# Nuclei (default templates)
nuclei -u http://127.0.0.1:4111

Scanner	False positives on the clean fixture
ptai (--no-llm)	0 (reproduce: python benchmarks/clean_fp_sweep.py)
OWASP ZAP	run it yourself with the command above
Nuclei	run it yourself with the command above

The point of leaving those cells for you to fill is the whole pitch: ptai surfaces nothing it cannot prove, and you do not have to take that on faith.

Recall, the other half

Precision alone is easy (report nothing). The companion benchmark measures recall: the private honeypot harness in tests/honeypot/ scores ptai against bugs we planted ourselves, asserted in CI at 10/10 on the public subset (tests/honeypot/test_mcp_honeypot_e2e.py). The private 20-bug run catches 19/20 at 100% precision. See the Juice Shop benchmark for ptai against a real, widely-studied target alongside ZAP / Nuclei / HexStrike.