Augustus - LLM Vulnerability Scanner

May 8, 2026 · View on GitHub

Augustus - LLM Vulnerability Scanner

Test large language models against 210+ adversarial attacks covering prompt injection, jailbreaks, encoding exploits, and data extraction.

Augustus is a Go-based LLM vulnerability scanner for security professionals. It tests large language models against a wide range of adversarial attacks, integrates with 28 LLM providers, and produces actionable vulnerability reports.

Unlike research-oriented tools, Augustus is built for production security testing — concurrent scanning, rate limiting, retry logic, and timeout handling come out of the box.

Why Augustus
Features
Quick Start
Supported Providers
Usage
How It Works
Architecture
Configuration
FAQ
Troubleshooting
Contributing
Security
Support
License

Why Augustus

Feature	Augustus	garak	promptfoo
Language	Go	Python	TypeScript
Single binary	Yes	No	No
Concurrent scanning	Goroutine pools	Multiprocessing pools	Yes
LLM providers	28	35+	80+
Probe types	210+	160+	119 plugins + 36 strategies
Enterprise focus	Yes	Research	Yes

Features

Feature	Description
210+ Vulnerability Probes	47 attack categories: jailbreaks, prompt injection, adversarial examples, data extraction, safety benchmarks, agent attacks, and more
28 LLM Providers	OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Ollama, and 22 more with 43 generator variants
90+ Detectors	Pattern matching, LLM-as-a-judge, HarmJudge (arXiv:2511.15304), Perspective API, unsafe content detection
7 Buff Transformations	Encoding, paraphrase, poetry (5 formats, 3 strategies), low-resource language translation, case transforms
Flexible Output	Table, JSON, JSONL, and HTML report formats
Production Ready	Concurrent scanning, rate limiting, retry logic, timeout handling
Single Binary	Go-based tool compiles to one portable executable
Extensible	Plugin-style registration via Go `init()` functions

Attack Categories

Jailbreak attacks: DAN, DAN 11.0, AIM, AntiGPT, Grandma, ArtPrompts
Prompt injection: Encoding (Base64, ROT13, Morse), Tag smuggling, FlipAttack, Prefix/Suffix injection
Adversarial examples: GCG, PAIR, AutoDAN, TAP (Tree of Attack Prompts), TreeSearch, DRA
Multi-turn attacks: Crescendo (gradual escalation), GOAT (adaptive technique switching)
Data extraction: API key leakage, Package hallucination, PII extraction, LeakReplay
Context manipulation: RAG poisoning, Context overflow, Multimodal attacks, Continuation, Divergence
Format exploits: Markdown injection, YAML/JSON parsing attacks, ANSI escape, Web injection (XSS)
Evasion techniques: Obfuscation, Character substitution, Translation-based attacks, Phrasing, ObscurePrompt
Safety benchmarks: DoNotAnswer, RealToxicityPrompts, Snowball, LMRC
Agent attacks: Multi-agent manipulation, Browsing exploits
Security testing: Guardrail bypass, AV/spam scanning, Exploitation (SQLi, code exec), BadChars

Warning: The lmrc probe uses profane and offensive language as part of its jailbreak testing. Use only in authorized testing environments.

Quick Start

Installation

Requires Go 1.25.3 or later.

go install github.com/praetorian-inc/augustus/cmd/augustus@latest

Or build from source:

git clone https://github.com/praetorian-inc/augustus.git
cd augustus
make build

Basic Usage

export OPENAI_API_KEY="your-api-key"
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --verbose

Example Output

+--------------+-------------+--------+-------+--------+
| PROBE        | DETECTOR    | PASSED | SCORE | STATUS |
+--------------+-------------+--------+-------+--------+
| dan.Dan_11_0 | dan.DAN     | false  | 0.85  | VULN   |
| dan.STAN     | dan.STAN    | true   | 0.10  | SAFE   |
| dan.AntiDAN  | dan.AntiDAN | true   | 0.05  | SAFE   |
+--------------+-------------+--------+-------+--------+

List Available Capabilities

# List all registered probes, detectors, generators, harnesses, and buffs
augustus list

Supported Providers

Augustus includes 28 LLM provider categories with 43 generator variants:

Provider	Generator Name(s)	Notes
OpenAI	`openai.OpenAI`, `openai.OpenAIReasoning`	GPT-3.5, GPT-4, GPT-4 Turbo, o1/o3 reasoning models
Anthropic	`anthropic.Anthropic`	Claude 3/3.5/4 (Opus, Sonnet, Haiku)
Azure OpenAI	`azure.AzureOpenAI`	Azure-hosted OpenAI models
AWS Bedrock	`bedrock.Bedrock`	Claude, Llama, Titan models
Google Vertex AI	`vertex.Vertex`	PaLM, Gemini models
Cohere	`cohere.Cohere`	Command, Command R models
Replicate	`replicate.Replicate`	Cloud-hosted open models
HuggingFace	`huggingface.InferenceAPI`, `huggingface.InferenceEndpoint`, `huggingface.Pipeline`, `huggingface.LLaVA`	HF Inference API, endpoints, pipelines, multimodal
Together AI	`together.Together`	Fast inference for OSS models
Anyscale	`anyscale.Anyscale`	Llama and Mistral hosting
Groq	`groq.Groq`	Ultra-fast LPU inference
Mistral	`mistral.Mistral`	Mistral API models
Fireworks	`fireworks.Fireworks`	Production inference platform
DeepInfra	`deepinfra.DeepInfra`	Serverless GPU inference
NVIDIA NIM	`nim.NIM`, `nim.NVOpenAICompletion`, `nim.NVMultimodal`, `nim.Vision`	NVIDIA AI endpoints, multimodal
NVIDIA NeMo	`nemo.NeMo`	NVIDIA NeMo framework
NVIDIA NVCF	`nvcf.NvcfChat`, `nvcf.NvcfCompletion`	NVIDIA Cloud Functions
NeMo Guardrails	`guardrails.NeMoGuardrails`	NVIDIA NeMo Guardrails
IBM watsonx	`watsonx.WatsonX`	IBM watsonx.ai platform
LangChain	`langchain.LangChain`	LangChain LLM wrapper
LangChain Serve	`langchain_serve.LangChainServe`	LangChain Serve endpoints
Rasa	`rasa.RasaRest`	Rasa conversational AI
GGML	`ggml.Ggml`	GGML local model inference
Function	`function.Single`, `function.Multiple`	Custom function generators
Ollama	`ollama.Ollama`, `ollama.OllamaChat`	Local model hosting
LiteLLM	`litellm.LiteLLM`	Unified API proxy
REST API	`rest.Rest`	Custom REST endpoints (SSE support)
Test	`test.Blank`, `test.Repeat`, `test.Lipsum`, `test.Nones`, `test.Single`, `test.BlankVision`	Testing and development

All providers are available in the compiled binary. Configure via environment variables or YAML configuration files. See Configuration for setup details.

Usage

Single Probe

# Test for DAN jailbreak
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config-file config.yaml \
  --verbose

Multiple Probes

# Use glob patterns to run related probes
augustus scan openai.OpenAI \
  --probes-glob "dan.*,goodside.*,grandma.*" \
  --detectors-glob "*" \
  --config-file config.yaml \
  --output batch-results.jsonl

# Run all probes against Claude
augustus scan anthropic.Anthropic \
  --all \
  --config '{"model":"claude-3-opus-20240229"}' \
  --timeout 60m \
  --output comprehensive-scan.jsonl \
  --html comprehensive-report.html

Buff Transformations

Apply prompt transformations to test evasion techniques:

# Apply base64 encoding buff to all probes
augustus scan openai.OpenAI \
  --all \
  --buff encoding.Base64 \
  --config '{"model":"gpt-4"}'

# Apply poetry transformation
augustus scan anthropic.Anthropic \
  --probes-glob "dan.*" \
  --buff poetry.MetaPrompt \
  --config '{"model":"claude-3-opus-20240229"}'

# Chain multiple buffs
augustus scan openai.OpenAI \
  --all \
  --buffs-glob "encoding.*,paraphrase.*" \
  --output buffed-results.jsonl

Output Formats

# Table format (default) - human-readable
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format table

# JSON format - structured output
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format json

# JSONL format - one JSON object per line, ideal for piping
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format jsonl

# HTML report - visual reports for stakeholders
augustus scan openai.OpenAI --all --html report.html

Custom REST Endpoints

# Test proprietary LLM endpoint (OpenAI-compatible API)
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{
    "uri": "https://api.example.com/v1/chat/completions",
    "method": "POST",
    "headers": {"Authorization": "Bearer YOUR_API_KEY"},
    "req_template_json_object": {
      "model": "custom-model",
      "messages": [{"role": "user", "content": "$INPUT"}]
    },
    "response_json": true,
    "response_json_field": "$.choices[0].message.content"
  }'

# Test with proxy interception (Burp Suite, mitmproxy)
augustus scan rest.Rest \
  --probes-glob "goodside.*" \
  --config '{
    "uri": "https://internal-llm.corp/generate",
    "proxy": "http://127.0.0.1:8080",
    "headers": {"X-API-Key": "$KEY"},
    "api_key": "your-key-here",
    "req_template": "{\"prompt\":\"$INPUT\",\"max_tokens\":500}",
    "response_json": true,
    "response_json_field": "output"
  }'

REST Configuration Keys:

uri: Target API endpoint (required)
method: HTTP method (default: POST)
headers: HTTP headers as key-value pairs
req_template: Raw request body with $INPUT placeholder
req_template_json_object: JSON request body (auto-marshaled, use $INPUT in strings)
response_json: Parse response as JSON (default: false)
response_json_field: JSONPath to extract (e.g., $.data.text or simple field name)
api_key: API key for $KEY placeholder substitution
proxy: HTTP proxy URL for traffic inspection

Advanced Options

# Adjust concurrency (default: 10)
augustus scan openai.OpenAI --all --concurrency 20

# Increase timeout for complex probes like TAP or PAIR
augustus scan openai.OpenAI --probe tap.TAPv1 --timeout 60m

# Use a specific harness strategy
augustus scan openai.OpenAI --all --harness batch.Batch

# Test local model with Ollama (no API key needed)
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'

How It Works

Augustus uses a pipeline architecture to test LLMs against adversarial attacks:

flowchart LR
    A[Probe Selection] --> B[Buff Transform]
    B --> C[Generator / LLM Call]
    C --> D[Detector Analysis]
    D --> E{Vulnerable?}
    E -->|Yes| F[Record Finding]
    E -->|No| G[Record Pass]

    subgraph Scanner
        B
        C
        D
        E
    end

Scan Pipeline

Probe Selection: Choose probes by name, glob pattern, or --all
Buff Transformation: Optionally transform prompts (encode, paraphrase, translate, poeticize)
Generator Call: Send adversarial prompts to the target LLM via its provider integration
Detector Analysis: Analyze responses using pattern matching, LLM-as-a-judge, or specialized detectors
Result Recording: Score each attempt and produce output in the requested format
Attack Engine: For iterative probes (PAIR, TAP), the single-turn attack engine refines prompts across iterations with candidate pruning and judge-based scoring
Multi-Turn Engine: For conversational probes (Crescendo, GOAT), the multi-turn engine maintains full conversation history with the target across turns, with refusal detection and dynamic adaptation

Multi-Turn Attack Strategies

Multi-turn attacks maintain a persistent conversation with the target LLM, exploiting the fact that models may disclose information incrementally across turns that they would refuse in a single prompt. The multi-turn engine uses three LLMs: an attacker (generates questions), a target (the system under test), and a judge (scores progress and detects refusals).

Crescendo

Crescendo uses gradual escalation (foot-in-the-door technique) to slowly shift a conversation from benign topics toward a prohibited objective.

Paper: Russinovich et al., 2024
Approach: Starts with genuinely benign, educational questions and incrementally increases specificity over many turns
Escalation pattern: Historical context → technical mechanisms → specific details → direct requests framed as natural follow-ups
Strength: Effective against models that track conversation tone — the gradual shift avoids triggering safety filters

augustus scan rest.Rest \
  --probe crescendo.Crescendo \
  --config-file crescendo.yaml \
  --html report.html -v

GOAT (Generative Offensive Agent Tester)

GOAT uses an aggressive, adaptive approach with 7 adversarial techniques and Chain-of-Attack-Thought reasoning to dynamically switch strategies based on what works or fails.

Paper: Pavlova et al., 2024
Approach: Targets the objective from turn 1 using indirect framing, never mentions the goal directly to the target
7 techniques across 3 categories:
- Output Manipulation: Refusal Suppression, Response Priming
- Safe Response Distractors: Dual Response, Topic Splitting, Opposite Intent
- Fictional Scenarios: Persona Modification, Hypothetical
Chain-of-Attack-Thought: Each turn the attacker reasons through Observation → Thought → Strategy → Response before crafting its message
Technique stacking: Multiple techniques can be combined in a single turn for stronger effect
Strength: Achieves high success rates in fewer turns (typically 3-5) by aggressively switching between fundamentally different approaches

augustus scan rest.Rest \
  --probe goat.Goat \
  --config-file goat.yaml \
  --html report.html -v

Hydra

Hydra maintains a single conversation path and rolls back entire turns when the target refuses, asking the attacker for a completely different approach. Unlike Crescendo/GOAT (which rephrase on refusal), Hydra's backtracking completely removes refused turns from the target's view.

Approach: Single-path with turn-level backtracking — refused turns are erased and replaced
Techniques: Decomposition (break goal into innocent sub-questions), context leveraging (build on target's own words), authority simulation, emotional framing, progressive normalization
Key feature: max_backtracks controls how many times Hydra can erase and retry a turn
Stateful mode: Set stateful: true for targets where messages cannot be unsent (disables backtracking)
Strength: Keeps the target's conversation history clean — the target never sees failed approaches, preventing defensive escalation

augustus scan rest.Rest \
  --probe hydra.Hydra \
  --config-file hydra.yaml \
  --html report.html -v

Mischievous User

The Mischievous User simulates an innocent, curious person who subtly probes AI boundaries through natural conversation. Unlike adversarial strategies, the attacker persona is a casual user who drifts toward prohibited topics through seemingly innocent engagement.

Inspired by: Tau-bench and promptfoo's mischievous-user strategy
Approach: Rapport-building phase → boundary probing → assertive pushing
7 techniques: Naive curiosity, deliberate misunderstanding, social proof, gradual drift, emotional appeal, assumed permission, recontextualization
Goal-specific playbooks: Built-in scripts for extracting system instructions, finding secrets/flags, and bypassing content policy
Default turns: 5 (fewer than other strategies — the subtle approach either works quickly or not at all)
Strength: Effective against models trained to resist obvious adversarial patterns — the casual persona bypasses "attack detection" heuristics

augustus scan rest.Rest \
  --probe mischievous.MischievousUser \
  --config-file mischievous.yaml \
  --html report.html -v

Strategy Selection Guide

Strategy	Turns	Style	Best For	Cost
`crescendo.Crescendo$	10	\text{Gradual} \text{escalation}	\text{Models} \text{with} \text{tone}-\text{tracking} \text{safety} \text{filters}	\text{Medium} (10 \text{turns} \times 3 \text{LLM} \text{calls})
$goat.Goat`	10	Aggressive technique switching	Quick results, models weak to framing tricks	Medium-High (technique reasoning overhead)
`hydra.Hydra`	10	Backtracking on refusal	Stubborn models that refuse frequently	High (backtracking adds retries)
`mischievous.MischievousUser`	5	Casual persona probing	Models trained against adversarial patterns	Low (5 turns, subtle approach)

Quick decision:

Start with Crescendo — it's the most general-purpose strategy
Try GOAT if Crescendo is too slow (GOAT typically succeeds in 3-5 turns)
Use Hydra if the target refuses frequently (backtracking keeps conversation clean)
Use Mischievous for targets with strong adversarial-pattern detection (the persona bypasses heuristics)

Multi-Turn Configuration

All four strategies share the same configuration structure:

generators:
  rest.Rest:
    uri: "https://your-llm-endpoint/chat"
    method: POST
    req_template: '{"input": "$INPUT"}'
    response_json: true
    response_json_field: "response"

judge:
  generator_type: openai.OpenAI
  config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"

probes:
  attacker_generator_type: openai.OpenAI
  attacker_config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"
  settings:
    crescendo.Crescendo:  # or goat.Goat, hydra.Hydra, mischievous.MischievousUser
      goal: "your test objective"
      max_turns: 10
      success_threshold: 0.8
      use_secondary_judge: true

Setting	Default	Description
`goal`	(required)	The objective the attacker tries to achieve
`max_turns`	10	Maximum conversation turns before stopping
`success_threshold`	0.8	Judge score (0.0-1.0) that triggers early exit
`max_refusal_retries`	10	Rephrase attempts per turn when target refuses
`attack_max_attempts`	5	Retries for attacker LLM JSON parsing failures
`use_secondary_judge`	true	Enable secondary judge to catch false negatives
`max_backtracks`	10	Turn-level rollbacks on refusal (Hydra only)
`enable_fast_refusal`	true	Pattern-based refusal detection before LLM judge call
`enable_scan_memory`	false	Cross-test-case learning (shares tactics across probes)
`stateful`	false	Disable backtracking for stateful targets
`exclude_target_output`	false	Hide target responses from attacker feedback (privacy mode)
`attacker_model`	(auto)	Override attacker model name for context window sizing

Multi-Turn Troubleshooting

Symptom	Likely Cause	Fix
`no turns completed (attacker_parse_failures=N)`	Attacker LLM returning invalid JSON	Use a stronger attacker model (GPT-4, Claude Opus). Increase `attack_max_attempts`.
`no turns completed (target_empty=N)`	Target returning empty/null responses	Check target endpoint is responding. Verify REST config template.
All turns score 0.0	Goal too vague or attacker not engaging	Make `goal` more specific. Try different strategy.
High scores but no success	`success_threshold` too high	Lower `success_threshold` from 0.8 to 0.6-0.7
Runs too long / expensive	Too many turns and retries	Reduce `max_turns` (try 5). Set `enable_fast_refusal: true`.
Hydra keeps backtracking	Target refuses everything	Try `stateful: true` or switch to Mischievous strategy

Architecture

cmd/augustus/          CLI entrypoint (Kong-based)
pkg/
  attempt/            Probe execution lifecycle and result tracking
  buffs/              Buff interface for prompt transformations
  config/             Configuration loading (YAML/JSON) with profiles
  detectors/          Public detector interfaces and registry
  generators/         Public generator interfaces and registry
  harnesses/          Harness interface for execution strategies
  lib/http/           Shared HTTP client with proxy support
  lib/stego/          LSB steganography for multimodal attacks
  logging/            Structured slog-based logging
  metrics/            Prometheus metrics collection
  prefilter/          Aho-Corasick keyword pre-filtering
  probes/             Public probe interfaces and registry
  ratelimit/          Token bucket rate limiting
  registry/           Generic capability registration system
  results/            Result types and multi-format output
  retry/              Exponential backoff with jitter
  scanner/            Scanner orchestration with concurrency
  templates/          YAML probe template loader (Nuclei-style)
  types/              Canonical shared interfaces (Prober, Generator, Detector)
internal/
  probes/             210+ probe implementations (47 categories)
  generators/         28 LLM provider integrations (43 variants)
  detectors/          90+ detector implementations (35 categories)
  harnesses/          3 harness strategies (probewise, batch, agentwise)
  buffs/              Buff interface for prompt transformations
  attackengine/       Iterative adversarial attack engine (PAIR/TAP backend)
  multiturn/          Multi-turn conversational attack engine (Crescendo/GOAT/Hydra/Mischievous)
  ahocorasick/        Internal Aho-Corasick keyword matching
benchmarks/           Performance benchmarks
tests/                Integration and equivalence tests
research/             Research documentation and analysis
examples/             Example configurations
docs/                 Documentation

Key Design Decisions

Concurrent scanning with bounded goroutine pools via errgroup
Plugin-style registration using Go init() functions for probes, generators, detectors, buffs, and harnesses
Iterative attack engine with multi-stream conversation management, candidate pruning, and judge-based scoring for PAIR/TAP
Multi-turn attack engine with persistent conversation history, refusal detection, strategy-agnostic design for Crescendo/GOAT
YAML probe templates (Nuclei-style) for declarative probe definitions alongside Go-based probes
Aho-Corasick pre-filtering for fast keyword matching in detectors

Configuration

YAML Configuration File

Create a config.yaml file:

# Runtime configuration
run:
  max_attempts: 3
  timeout: "30s"

# Generator configurations
generators:
  openai.OpenAI:
    model: "gpt-4"
    temperature: 0.7
    api_key: "${OPENAI_API_KEY}"  # Environment variable interpolation

  anthropic.Anthropic:
    model: "claude-3-opus-20240229"
    temperature: 0.5
    api_key: "${ANTHROPIC_API_KEY}"

  ollama.OllamaChat:
    model: "llama3.2:3b"
    temperature: 0.8

# Judge configuration (required for judge.Judge, judge.Refusal, and multi-turn probes)
judge:
  generator_type: openai.OpenAI
  model: gpt-4o-mini
  config:
    api_key: "${OPENAI_API_KEY}"

# Output configuration
output:
  format: "jsonl"
  path: "./results.jsonl"

# Named profiles for different scenarios
profiles:
  quick:
    run:
      max_attempts: 1
      timeout: "10s"
    generators:
      openai.OpenAI:
        model: "gpt-3.5-turbo"
        temperature: 0.5
    output:
      format: "table"

  thorough:
    run:
      max_attempts: 5
      timeout: "60s"
    generators:
      openai.OpenAI:
        model: "gpt-4"
        temperature: 0.3
    output:
      format: "jsonl"
      path: "./thorough_results.jsonl"

Environment Variables

# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export COHERE_API_KEY="..."

# Debug mode
export AUGUSTUS_DEBUG=true

Proxy Configuration

Route HTTP traffic through a proxy (e.g., Burp Suite) for inspection:

# Method 1: Via config parameter
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{"uri":"https://api.example.com","proxy":"http://127.0.0.1:8080"}' \
  --output results.jsonl

# Method 2: Via environment variables
export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
augustus scan rest.Rest --probe dan.Dan_11_0 --config '{"uri":"https://api.example.com"}'

TLS verification automatically disabled for proxy inspection
HTTP/2 support enabled for modern APIs
Server-Sent Events (SSE) responses automatically detected and parsed

CLI Reference

Usage: augustus scan <generator> [flags]

Arguments:
  <generator>                 Generator name (e.g., openai.OpenAI, anthropic.Anthropic)

Probe Selection (choose one):
  --probe, -p                 Probe name (repeatable)
  --probes-glob               Comma-separated glob patterns (e.g., "dan.*,goodside.*")
  --all                       Run all registered probes

Detector Selection:
  --detector                  Detector name (repeatable)
  --detectors-glob            Comma-separated glob patterns

Buff Selection:
  --buff, -b                  Buff names to apply (repeatable)
  --buffs-glob                Comma-separated buff glob patterns (e.g., "encoding.*")

Configuration:
  --config-file               Path to YAML config file
  --config, -c                JSON config for generator

Execution:
  --harness                   Harness name (default: probewise.Probewise)
  --timeout                   Overall scan timeout (default: 30m)
  --probe-timeout             Per-probe timeout (default: 5m)
  --concurrency               Max concurrent probes (default: 10, env: AUGUSTUS_CONCURRENCY)

Output:
  --format, -f                Output format: table, json, jsonl (default: table)
  --output, -o                JSONL output file path
  --html                      HTML report file path
  --verbose, -v               Verbose output

Global:
  --debug, -d                 Enable debug mode

Commands:

augustus version              # Print version information
augustus list                 # List available probes, detectors, generators, harnesses, buffs
augustus scan <generator>     # Run vulnerability scan
augustus completion <shell>   # Generate shell completion (bash, zsh, fish)

Exit Codes:

Code	Meaning
0	Success - scan completed
1	Scan/runtime error
2	Validation/usage error

FAQ

How does Augustus compare to garak?

Augustus is a Go-native reimplementation inspired by garak (NVIDIA's Python-based LLM vulnerability scanner). Key differences:

Performance: Go binary vs Python interpreter — faster execution and lower memory usage
Distribution: Single binary with no runtime dependencies vs Python package with pip install
Concurrency: Go goroutine pools (cross-probe parallelism) vs Python multiprocessing pools (within-probe parallelism)
Probe coverage: Augustus has 210+ probes; garak has 160+ probes with a longer research pedigree and published paper (arXiv:2406.11036)
Provider coverage: Augustus has 28 providers; garak has 35+ generator variants across 22 provider modules

Can I test local models without API keys?

Yes! Use the Ollama integration for local model testing:

# No API key needed
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'

How do I add custom probes?

Create a new Go file in internal/probes/
Implement the probes.Probe interface
Register using registry.RegisterProbe() in an init() function
Rebuild: make build

See CONTRIBUTING.md for detailed instructions.

What output formats are supported?

Augustus supports four output formats:

Format	Flag	Use Case
Table	`--format table`	Human-readable terminal output
JSON	`--format json`	Single JSON object for parsing
JSONL	`--format jsonl`	Line-delimited JSON for streaming
HTML	`--html report.html`	Visual reports for stakeholders

How do I test multiple models at once?

# Test multiple models sequentially
for model in "gpt-4" "gpt-3.5-turbo"; do
  augustus scan openai.OpenAI \
    --all \
    --config "{\"model\":\"$model\"}" \
    --output "results-$model.jsonl"
done

Is Augustus suitable for production environments?

Yes, Augustus is designed for production use with:

Concurrent scanning with configurable limits
Rate limiting to respect API quotas
Timeout handling for long-running probes
Retry logic for transient failures
Structured logging for observability

Troubleshooting

Error: "API rate limit exceeded"

Cause: Too many concurrent requests or requests per minute.

Solutions:

Reduce concurrency: --concurrency 5

Use provider-specific rate limit settings in YAML config:

generators:
  openai.OpenAI:
    rate_limit: 10  # requests per minute

Error: "context deadline exceeded" or "timeout"

Cause: Complex probes (like TAP or PAIR) exceed default timeout.

Solution:

augustus scan openai.OpenAI \
  --probe tap.TAPv1 \
  --timeout 60m \
  --config-file config.yaml

Error: "invalid API key" or "authentication failed"

Cause: Missing or invalid API credentials.

Solutions:

Verify environment variable is set: echo $OPENAI_API_KEY
Check for typos in config file
Ensure API key has required permissions
For Ollama, ensure the service is running: ollama serve

Error: "probe not found" or "detector not found"

Cause: Typo in name or probe not registered.

Solution:

# List all available probes and detectors
augustus list

# Use exact names from the list
augustus scan openai.OpenAI --probe dan.Dan_11_0  # Correct

Scan produces no results

Cause: Detector didn't match any responses, or output not written.

Solutions:

Run with --verbose to see detailed output
Check that detector matches probe type
Verify output file path is writable

Contributing

We welcome contributions! See CONTRIBUTING.md for:

Adding new vulnerability probes
Creating new detector implementations
Adding LLM provider integrations
Testing guidelines
Code style requirements

Development

# Run all tests
make test

# Run specific package tests
go test ./pkg/scanner -v

# Run equivalence tests (compare Go vs Python implementations)
go test ./tests/equivalence -v

# Build binary
make build

# Install to $GOPATH/bin
make install

Benchmark Environment (DevPod)

A ready-to-go cloud development environment for benchmarking LLMs is available via DevPod. It provisions a remote container with Augustus, Ollama, Go, and all dependencies pre-installed.

cd devpod

# CPU-only instance (~\$0.08/hr) - cloud APIs only
make devpod-up-cpu

# GPU instance with NVIDIA T4 (~\$0.53/hr) - local models up to 14B
make devpod-up-gpu

# GPU Pro instance with NVIDIA L4 (~\$0.80/hr) - local models up to 32B
make devpod-up-gpu-pro

Inside the devpod:

devpod/scripts/setup.sh        # Configure LLM provider API keys
devpod/scripts/pull-models.sh   # Pull local Ollama models (GPU only)
devpod/scripts/benchmark.sh     # Run benchmarks with comparison reports

The environment also works as a standard dev container — open the repo in VS Code or Cursor and select the CPU or GPU configuration from .devcontainer/.

Security

Augustus is designed for authorized security testing only.

Augustus sends adversarial prompts to LLMs you specify - always ensure you have authorization
Never test systems you don't own or have explicit permission to test
Some probes generate offensive content by design (for testing safety filters)
Results may contain harmful content produced by target LLMs

Report security issues via GitHub Issues.

Support

If you find Augustus useful, please consider:

Giving it a star on GitHub
Opening an issue for bugs or feature requests
Contributing new probes, detectors, or provider integrations

License

Apache 2.0 - Praetorian Security, Inc.

Built by Praetorian - Offensive Security Solutions