Agent Audit

Find security vulnerabilities in your AI agent code before they reach production.

Why Agent Security Fails in Production

AI agents are not just chatbots. They execute code, call tools, and touch real systems, so one unsafe input path can become a production incident.

• Prompt injection rewrites agent intent through user-controlled context
• Unsafe tool inputs can reach subprocess/eval and become command execution
• MCP configuration mistakes can leak credentials and expand access unintentionally

If your team ships agent features, owns CI security gates, or operates MCP servers and tool integrations, this is a high-probability risk surface rather than an edge case. You likely need this before every merge if agent code can trigger tools, commands, or external systems.

Agent Audit catches these issues before deployment with an analysis core designed for agent workflows today: tool-boundary taint tracking, MCP configuration auditing, and semantic secret detection, with room to extend into learning-assisted detection over time.

Think of it as security linting for AI agents, with 72 rules mapped to the OWASP Agentic Top 10 (2026).

Quick Start in 6 Lines

1. Install

pip install agent-audit

2. Scan your project

agent-audit scan ./your-agent-project

3. Interpret and gate in CI

# Show only high+ findings
agent-audit scan . --severity high

# Fail CI when high+ findings exist
agent-audit scan . --fail-on high

--severity controls what is reported. --fail-on controls when the command exits with code 1.

Sample report output:

╭──────────────────────────────────────────────────────────────────────────────╮
│ Agent Audit Security Report                                                  │
│ Scanned: ./your-agent-project                                                │
│ Files analyzed: 2                                                            │
│ Risk Score: 8.4/10 (HIGH)                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯

BLOCK -- Tier 1 (Confidence >= 90%) -- 16 findings

  AGENT-001: Command Injection via Unsanitized Input
    Location: agent.py:21
    Code: result = subprocess.run(command, shell=True, capture_output=True, text=True)

  AGENT-010: System Prompt Injection Vector in User Input Path
    Location: agent.py:13
    Code: system_prompt = f"You are a helpful {user_role} assistant..."

  AGENT-041: SQL Injection via String Interpolation
    Location: agent.py:31
    Code: cursor.execute(f"SELECT * FROM users WHERE name = '{query}'")

  AGENT-031: Mcp Sensitive Env Exposure
    Location: mcp_config.json:1
    Code: env: {"API_KEY": "sk-a***"}

  ... and 15 more

Summary:
  BLOCK: 16 | WARN: 2 | INFO: 1
  Risk Score: =========================----- 8.4/10 (HIGH)

Validation snapshot (as of 2026-05-30, v0.19.0, ground-truth dataset v2.2, 81 samples / 236 positive labels + 2 negative labels):

• Precision 73.58%, Recall 82.63%, F1 0.778 (raw, reproducible) — TP 195 / FP 70 / FN 41
• Footnote: an adjusted F1 of 0.84 is computable post-hoc by excluding FPs from rules added after v0.16 that are not yet labeled in GT, but it is not directly reproducible from precision_recall.py and is therefore not used as a headline figure. See docs/F1_REPRODUCTION.md. GT v2.3 refresh planned June 2026.
• OWASP Agentic Top 10 coverage: 10/10

Reproducible from a clean clone: pip install -e packages/audit/ && python tests/benchmark/precision_recall.py --output-json results/layer1.json. Result file checked in at results/layer1_v0.19.0.json. Ground-truth label refresh for the 18 new rules (AGENT-053+) lands June 2026.

Details: Benchmark Results | Competitive Comparison

What It Detects

Full coverage of all 10 OWASP Agentic Security categories. Framework-specific detection for LangChain, CrewAI, AutoGen, and AgentScope. See all rules ->

OpenClaw Support

Agent Audit is available as an OpenClaw skill on ClawHub:

npx clawhub@latest install agent-audit-scanner

Once installed, ask your OpenClaw agent:

• "Scan my installed skills for security issues"
• "Is this new skill safe?"
• "Audit my OpenClaw config"

The scanner covers all 10 OWASP Agentic AI threat categories and has been validated against 18,899 ClawHub skills at 80% precision.

Who Is This For

• Agent developers building with LangChain, CrewAI, AutoGen, OpenAI Agents SDK, or raw function-calling -- run it before every deploy
• Security engineers reviewing agent codebases -- get a structured report in SARIF for GitHub Security tab
• Teams shipping MCP servers -- validate your mcp.json / claude_desktop_config.json for secrets, auth gaps, and supply chain risks

Usage

# Scan a project
agent-audit scan ./my-agent

# JSON output for scripting
agent-audit scan ./my-agent --format json

# SARIF output for GitHub Code Scanning
agent-audit scan . --format sarif --output results.sarif

# Only fail CI on critical findings
agent-audit scan . --fail-on critical

# Inspect a live MCP server (read-only, never calls tools)
agent-audit inspect stdio -- npx -y @modelcontextprotocol/server-filesystem /tmp

Baseline Scanning

Track only new findings across commits:

# Save current state as baseline
agent-audit scan . --save-baseline baseline.json

# Only report new findings not in baseline
agent-audit scan . --baseline baseline.json --fail-on-new

GitHub Actions

name: Agent Security Scan
on: [push, pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: HeadyZhang/agent-audit@v1
        with:
          path: '.'
          fail-on: 'high'
          upload-sarif: 'true'

Evaluation Results

How It Works

Source Files (.py, .json, .yaml, .env, ...)
        |
        +-- PythonScanner ---- AST Analysis ---- Dangerous Patterns
        |        |                                Tool Metadata
        |        +-- TaintTracker --------------- Source->Sink Reachability
        |        +-- DangerousOperationAnalyzer - Tool Boundary Detection
        |
        +-- SecretScanner ---- Regex Candidates
        |        +-- SemanticAnalyzer ----------- 3-Stage Filtering
        |              (Known Formats -> Entropy/Placeholder -> Context)
        |
        +-- MCPConfigScanner -- Server Provenance / Path Permissions / Auth
        |
        +-- PrivilegeScanner -- Daemon / Sudoers / Sandbox / Credential Store
                 |
                 v
            RuleEngine -- 72 Rules x OWASP Agentic Top 10 -- Findings

Threat Coverage

72 detection rules covering all 10 categories of the OWASP Agentic Top 10 (2026):

Real-World Validation

Comparison with Existing Tools

agent-audit is complementary to general-purpose SAST tools. It targets the security gap specific to AI agent applications that existing tools cannot address.

Configuration

# .agent-audit.yaml
scan:
  exclude: ["tests/**", "venv/**"]
  min_severity: low
  fail_on: high

ignore:
  - rule_id: AGENT-003
    paths: ["auth/**"]
    reason: "Auth module legitimately communicates externally"

allowed_hosts:
  - "api.openai.com"

Current Scope

Documentation

• Technical Specification -- Detection methodology and analysis pipeline
• Benchmark Results -- Detailed Agent-Vuln-Bench evaluation
• Competitive Comparison -- Three-tool analysis vs Bandit and Semgrep
• Rule Reference -- Complete rule catalog with CWE mappings and remediation
• Architecture -- Internal design and extension points
• CI/CD Integration -- GitHub Actions, GitLab CI, Jenkins, Azure DevOps

Development

git clone https://github.com/HeadyZhang/agent-audit
cd agent-audit/packages/audit
poetry install
poetry run pytest ../../tests/ -v  # 1541 tests

See CONTRIBUTING.md for full development setup and PR guidelines.

Citation

If you use agent-audit in your research, please cite:

@software{agent_audit_2026,
  author = {Zhang, Haiyue},
  title = {Agent Audit: Static Security Analysis for AI Agent Applications},
  year = {2026},
  url = {https://github.com/HeadyZhang/agent-audit},
  note = {Based on OWASP Agentic Top 10 (2026) threat model}
}

Acknowledgments

• OWASP Agentic Top 10 for 2026
• CWE Top 25

License

MIT -- see LICENSE.