Why verification

Most security tools ask you to trust their output. ptai does not. A finding earns a VERIFIED badge only when a machine oracle re-runs the exploit and reproduces it N out of N times. Everything else stays a candidate. This page is the evidence for why that design is the right one, especially now that "AI pentesting" is flooding the field with output nobody can trust.

The problem: a single LLM run is not evidence

Independent research shows that letting a language model "do the pentest" produces results that vary by the model, not by the ground-truth state of the target.

• In a 400-run study (four models, 100 runs each) against one fixed honeypot, full-exploitation success ranged from 25% to 85% depending only on which model ran it. The differences were statistically significant (p < 0.001) with large effect sizes. The target never changed; the verdict did. A result that swings that far on identical input is not a measurement, it is noise.
• LLM pentest agents actively fabricate evidence. Studies document models hallucinating commands and scanner features that do not exist, an "always answer" bias, and "soliloquizing" where the model narrates imagined command output instead of executing anything. These errors persist even when the agent is given a chance to correct itself.

If the tool can hallucinate the finding, the finding is worthless until something re-runs it against the real target. That something is a machine oracle, not another model.

The cost you already pay: the false-positive grind

A scanner that floods you with unverified findings forces a three-step manual grind: find, verify, assign. Worse, chronic false positives train a team to ignore their tools, so the real issue slips through with the noise. The thing that gets a tool adopted on a real engagement is the opposite: output you can act on without re-checking it yourself.

How ptai answers it

• A verdict comes from a machine oracle, never an assertion. The receipt contract enforces this in code: a verified verdict that cannot name the oracle that earned it is rejected. An LLM cannot assert a verdict into existence.
• The exploit is re-run, N out of N. Reflection, open-redirect, IDOR/BOLA, error-disclosure, MCP exposure, boolean and blind SQLi, error-based SSTI, and out-of-band SSRF/XXE each have an oracle that reproduces the effect, not a pattern match.
• Third-party scanner noise is gated. Output from nuclei, nikto, and the like is held back until an oracle re-proves it, so it never reaches your report unproven.
• The proof is portable. Every VERIFIED finding ships a proof capsule a developer or client can replay against the live target without trusting ptai. Multi-step chains bundle every proven hop into one replayable capsule.
• Impact is not assumed. A bare out-of-band callback proves a request left the target, not that it caused harm, so it is rated medium until an impact step reproduces real impact.

What ptai is, and is not

ptai is machine-verified proof for a human operator. It is not a fully-autonomous AI pentester, and it does not claim to be. The field's own security engineers are openly skeptical of unsupervised "AI security," and they are right to be: speed of generating checks is worthless if the output is false-positive or false-negative prone. The model coordinates and reasons; the oracles decide what is true.

See it and reproduce it

• ptai demo scans a bundled vulnerable app, oracle-verifies the planted flaws, replays one live from a proof capsule, then runs the same routes hardened and finds nothing. The only thing that changed is the fix.
• The honeypot harness (tests/honeypot/) measures catch rate on bugs we planted ourselves; the clean-app gate (tests/cleanapp/) measures false positives on a safe surface. Both are in the repo. See the precision benchmark and the Juice Shop benchmark.

The studies cited above were surfaced by an automated research pass; verify the primary sources before quoting specific figures in external material.