RedAmon - Resource Enumeration Module

August 10, 2026 · View on GitHub

Complete Technical Documentation

Module: recon/resource_enum.py Purpose: Endpoint discovery, classification, and parameter extraction Author: RedAmon Security Suite

Overview
Features
Installation
Configuration Parameters
GAU Configuration
Kiterunner Configuration
Hakrawler Configuration
jsluice Configuration
Architecture & Flow
Output Data Structure
Endpoint Classification
Parameter Classification
Form Parsing
Usage Examples
Troubleshooting

Overview

The resource_enum.py module provides comprehensive endpoint discovery and classification for web applications. It combines active crawling (Katana + Hakrawler), passive historical URL discovery (GAU), passive parameter mining (ParamSpider), API bruteforcing (Kiterunner), JavaScript analysis (jsluice), directory fuzzing (FFuf), and hidden parameter discovery (Arjun) to maximize endpoint coverage, extracts parameters, parses HTML forms, discovers embedded secrets, and organizes everything into a structured format ready for vulnerability scanning.

Pipeline Position: GROUP 5 in the parallelized pipeline (GROUP 4: http_probe -> GROUP 5: resource_enum -> GROUP 6: vuln_scan). Katana, Hakrawler, GAU, ParamSpider, and Kiterunner run concurrently via internal ThreadPoolExecutor. jsluice runs sequentially after crawling to analyze discovered JavaScript files. FFuf runs after jsluice to brute-force directory paths using wordlists. Arjun runs after FFuf to discover hidden parameters on discovered endpoints, with multiple methods (GET/POST/JSON/XML) executing in parallel.

Why Resource Enumeration?

Feature	Without resource_enum	With resource_enum
Endpoint Discovery	Manual or basic	Automated crawling + passive + API bruteforce
Historical URLs	Missed	GAU finds old/deleted endpoints
Hidden APIs	Missed	Kiterunner finds undocumented APIs
POST Endpoints	Missed	Form parsing
Parameter Extraction	None	Full extraction
Endpoint Classification	None	Categorized
Parameter Types	Unknown	Inferred
Vulnerability Coverage	Limited	Comprehensive

How It Works

┌─────────────────┐     ┌────────────────────────────────────────────────┐     ┌─────────────────┐
│  http_probe     │────▶│              resource_enum                      │────▶│  js_recon (5b)  │────▶│  vuln_scan      │
│  (live URLs,    │     │                                                │     │  (targeted      │
│   responses)    │     │  ┌────────────┐ ┌────────────┐ ┌─────────────┐ │     │   scanning)     │
└─────────────────┘     │  │  Katana    │ │    GAU     │ │ Kiterunner  │ │     └─────────────────┘
                        │  │  (active)  │ │  (passive) │ │ (API brute) │ │
                        │  │            │ │            │ │             │ │
                        │  │ Crawl site │ │ Query:     │ │ Bruteforce: │ │
                        │  │ Parse JS   │ │ - Wayback  │ │ - 40k+ APIs │ │
                        │  │ Find forms │ │ - CommonCrl│ │ - Swagger   │ │
                        │  └─────┬──────┘ │ - OTX      │ │ - OpenAPI   │ │
                        │        │        │ - URLScan  │ └──────┬──────┘ │
                        │        │        └─────┬──────┘        │        │
                        │        └───────────┬──┴───────────────┘        │
                        │                    ▼                           │
                        │  ┌──────────────────────────────────────────┐  │
                        │  │  Merge & Deduplicate                     │  │
                        │  │  + Source tracking (sources array)       │  │
                        │  │  + Endpoint Classification               │  │
                        │  └──────────────────────────────────────────┘  │
                        └────────────────────────────────────────────────┘

Features

Feature	Description
Katana Crawling	Deep endpoint discovery using ProjectDiscovery's Katana (active)
GAU Discovery	Historical URL discovery from Wayback, CommonCrawl, OTX, URLScan (passive)
Kiterunner API Bruteforce	Hidden API discovery using 40k+ Swagger/OpenAPI specifications
Hakrawler Crawling	DOM-aware web crawling via Docker (active)
jsluice JS Analysis	JavaScript analysis to extract URLs, endpoints, and secrets (downloads JS files from target)
FFuf Directory Fuzzing	Brute-force directory/endpoint discovery using wordlists (built-in SecLists + custom uploads)
ParamSpider Parameter Mining	Passive Wayback Machine CDX query for parameterized URLs — returns only URLs with query parameters, values replaced by placeholder (FUZZ)
Arjun Parameter Discovery	Discovers hidden HTTP query/body parameters on discovered endpoints using ~25,000 parameter names. Multi-method parallel execution (GET/POST/JSON/XML)
Parallel Execution	Katana, Hakrawler, GAU, ParamSpider, and Kiterunner run simultaneously, then jsluice → FFuf → Arjun sequential (Arjun methods run in parallel internally)
URL Verification	Verifies GAU URLs are live before adding to results
Method Detection	OPTIONS probe detects allowed HTTP methods (GET, POST, PUT, DELETE)
Dead Endpoint Filtering	Filters out endpoints that don't respond (404, 500, timeout)
JavaScript Parsing	Discovers endpoints in JavaScript files
Form Extraction	Parses HTML forms for POST endpoints
Parameter Extraction	Extracts query and body parameters
Type Inference	Infers parameter data types (integer, email, URL, etc.)
Endpoint Classification	Categorizes endpoints (auth, api, admin, file_access, etc.)
Parameter Classification	Identifies sensitive params (id, file, auth, redirect, command)
Source Tracking	Each endpoint tracked with `sources` array: `["katana", "hakrawler", "gau", "paramspider", "kiterunner", "jsluice", "arjun"]`
Docker Execution	Runs via Docker for consistency
Incremental Output	Saves results as crawling progresses

Installation

Requirements

Docker installed and running
Previous pipeline steps completed (http_probe)

Setup

# Make sure Docker is running
sudo systemctl start docker

# Run the scan - image will be pulled automatically
python3 recon/main.py

Verify Docker is Ready

# Check Docker is running
docker info

# Optionally pre-pull the images
docker pull projectdiscovery/katana:latest
docker pull sxcurity/gau:latest
# Kiterunner binary is auto-downloaded from GitHub releases (no Docker needed)
docker pull projectdiscovery/httpx:latest  # For URL verification

Configuration Parameters

All parameters are configured via the webapp project settings (stored in PostgreSQL) or as defaults in project_settings.py.

1. Core Katana Configuration

Parameter	Type	Default	Description
`KATANA_DOCKER_IMAGE`	`str`	`"projectdiscovery/katana:latest"`	Docker image to use
`KATANA_DEPTH`	`int`	`3`	Maximum crawl depth (how many links deep to follow)
`KATANA_MAX_URLS`	`int`	`1000`	Maximum URLs to discover per target
`KATANA_RATE_LIMIT`	`int`	`150`	Requests per second
`KATANA_TIMEOUT`	`int`	`300`	Maximum crawl time in seconds (5 minutes)

Depth Tuning Guide:

Depth	Use Case	Coverage	Time
1	Quick scan, homepage only	Low	Fast
2	Standard reconnaissance	Medium	Moderate
3	Default - balanced coverage	Good	~5 min
5+	Deep analysis, large sites	Comprehensive	Long

2. Crawl Behavior

Parameter	Type	Default	Description
`KATANA_JS_CRAWL`	`bool`	`True`	Parse JavaScript files for endpoints
`KATANA_PARAMS_ONLY`	`bool`	`False`	Only keep URLs with query parameters
`KATANA_SCOPE`	`str`	`"rdn"`	Scope: `rdn` (root domain), `dn` (domain), `fqdn` (full)

Scope Options:

Scope	Description	Example
`rdn`	Root domain and all subdomains	`*.example.com`
`dn`	Exact domain only	`www.example.com`
`fqdn`	Exact FQDN only	`www.example.com` (no subdomains)

3. Filtering

Parameter	Type	Default	Description
`KATANA_EXCLUDE_PATTERNS`	`list`	See below	URL patterns to exclude
`KATANA_CUSTOM_HEADERS`	`list`	`[]`	Custom HTTP headers

Default Exclude Patterns:

KATANA_EXCLUDE_PATTERNS = [
    # Next.js / React
    "/_next/image",          # Image optimization
    "/_next/static",         # Static assets

    # WordPress
    "/wp-content/uploads",   # Media uploads
    "/wp-includes",          # Core files

    # Common static
    "/static/",              # Static directories
    "/assets/",              # Asset directories
    ".css", ".js",           # Stylesheets, scripts
    ".jpg", ".png", ".gif",  # Images
    ".woff", ".ttf",         # Fonts
]

4. Performance Profiles

Fast Mode (Quick Recon)

KATANA_DEPTH = 2
KATANA_MAX_URLS = 500
KATANA_RATE_LIMIT = 200
KATANA_TIMEOUT = 120
KATANA_JS_CRAWL = False
KATANA_PARAMS_ONLY = True

Expected: ~1-2 minutes per target

Balanced Mode (Default)

KATANA_DEPTH = 3
KATANA_MAX_URLS = 1000
KATANA_RATE_LIMIT = 150
KATANA_TIMEOUT = 300
KATANA_JS_CRAWL = True
KATANA_PARAMS_ONLY = False

Expected: ~3-5 minutes per target

Deep Analysis Mode

KATANA_DEPTH = 5
KATANA_MAX_URLS = 5000
KATANA_RATE_LIMIT = 100
KATANA_TIMEOUT = 600
KATANA_JS_CRAWL = True
KATANA_PARAMS_ONLY = False

Expected: ~10-15 minutes per target

GAU Configuration

GAU (GetAllUrls) provides passive URL discovery from historical archives. It runs in parallel with Katana.

1. Core GAU Settings

Parameter	Type	Default	Description
`GAU_ENABLED`	`bool`	`True`	Enable/disable GAU discovery
`GAU_DOCKER_IMAGE`	`str`	`"sxcurity/gau:latest"`	Docker image to use
`GAU_PROVIDERS`	`list`	`["wayback", "commoncrawl", "otx", "urlscan"]`	Data sources to query
`GAU_MAX_URLS`	`int`	`1000`	Maximum URLs per domain (0 = unlimited)
`GAU_TIMEOUT`	`int`	`60`	Timeout per provider in seconds
`GAU_THREADS`	`int`	`5`	Parallel threads for fetching

2. GAU Data Sources

Provider	Description	URL Type
wayback	Wayback Machine (web.archive.org)	Historical snapshots
commoncrawl	Common Crawl (index.commoncrawl.org)	Web crawl data
otx	AlienVault OTX	Threat intelligence
urlscan	URLScan.io	Security scan results

3. Filtering Options

Parameter	Type	Default	Description
`GAU_BLACKLIST_EXTENSIONS`	`list`	See below	File extensions to exclude
`GAU_YEAR_RANGE`	`list`	`[]`	Filter by year range, e.g., `["2020", "2024"]`

Default Blacklisted Extensions:

GAU_BLACKLIST_EXTENSIONS = [
    "png", "jpg", "jpeg", "gif", "svg", "ico", "webp", "avif",
    "css", "woff", "woff2", "ttf", "eot", "otf",
    "mp3", "mp4", "avi", "mov", "wmv", "flv", "webm",
    "pdf", "doc", "docx", "xls", "xlsx", "ppt", "pptx",
    "zip", "rar", "7z", "tar", "gz"
]

4. URL Verification

GAU URLs are verified to check if they're still live before adding to results.

Parameter	Type	Default	Description
`GAU_VERIFY_URLS`	`bool`	`True`	Enable HTTP verification
`GAU_VERIFY_DOCKER_IMAGE`	`str`	`"projectdiscovery/httpx:latest"`	httpx image for verification
`GAU_VERIFY_TIMEOUT`	`int`	`5`	Timeout per URL in seconds
`GAU_VERIFY_RATE_LIMIT`	`int`	`100`	Requests per second
`GAU_VERIFY_THREADS`	`int`	`50`	Concurrent verification threads
`GAU_VERIFY_ACCEPT_STATUS`	`list`	`[200, 201, 301, 302, 307, 308, 401, 403]`	HTTP status codes to accept

5. HTTP Method Detection (OPTIONS Probe)

GAU doesn't know which HTTP methods an endpoint supports. This feature uses the OPTIONS HTTP method to detect allowed methods from the server's Allow header.

Parameter	Type	Default	Description
`GAU_DETECT_METHODS`	`bool`	`True`	Enable OPTIONS probe for method detection
`GAU_METHOD_DETECT_TIMEOUT`	`int`	`5`	Timeout per URL in seconds
`GAU_METHOD_DETECT_RATE_LIMIT`	`int`	`50`	Requests per second
`GAU_METHOD_DETECT_THREADS`	`int`	`25`	Concurrent threads
`GAU_FILTER_DEAD_ENDPOINTS`	`bool`	`True`	Filter out endpoints that don't respond

How It Works:

Send OPTIONS request to each verified GAU URL
Parse Allow header from response (e.g., Allow: GET, POST, PUT, DELETE)
If no Allow header, fall back to GET check
Filter out dead endpoints (404, 500, timeout)
Store detected methods in endpoint data

Example Output:

{
  "/api/users": {
    "methods": ["GET", "POST", "PUT", "DELETE"],
    "source": "gau"
  },
  "/login": {
    "methods": ["GET", "POST"],
    "source": "gau"
  }
}

Why This Matters:

Katana can detect methods from form parsing (e.g., <form method="POST">)
GAU only returns URLs - no method info
OPTIONS probe discovers POST/PUT/DELETE endpoints that would otherwise be missed
Dead endpoints (404/500) are filtered out to reduce noise

6. GAU Configuration Profiles

Minimal (Fast)

GAU_ENABLED = True
GAU_PROVIDERS = ["wayback"]  # Single source
GAU_MAX_URLS = 500
GAU_VERIFY_URLS = False  # Skip verification
GAU_DETECT_METHODS = False  # Skip method detection

Expected: ~10-20 seconds per domain

Balanced (Default)

GAU_ENABLED = True
GAU_PROVIDERS = ["wayback", "commoncrawl", "otx", "urlscan"]
GAU_MAX_URLS = 1000
GAU_VERIFY_URLS = True
GAU_DETECT_METHODS = True
GAU_FILTER_DEAD_ENDPOINTS = True

Expected: ~30-60 seconds per domain

Comprehensive

GAU_ENABLED = True
GAU_PROVIDERS = ["wayback", "commoncrawl", "otx", "urlscan"]
GAU_MAX_URLS = 5000
GAU_YEAR_RANGE = []  # All years
GAU_VERIFY_URLS = True
GAU_DETECT_METHODS = True
GAU_FILTER_DEAD_ENDPOINTS = True

Expected: ~2-5 minutes per domain

7. What GAU Finds

Category	Examples	Why It Matters
Old Admin Panels	`/admin/`, `/wp-admin/`, `/administrator/`	May still be accessible
Debug Endpoints	`/phpinfo.php`, `/debug/`, `/test/`	Information disclosure
Backup Files	`/backup.sql`, `/db_dump.sql`, `/config.bak`	Sensitive data
Old API Versions	`/api/v1/`, `/api/beta/`	May have unpatched vulns
Hidden Parameters	`?debug=1`, `?admin=true`	Bypass security
Forgotten Uploads	`/uploads/temp/`, `/files/old/`	Sensitive files

Kiterunner Configuration

Kiterunner provides API endpoint bruteforcing using real Swagger/OpenAPI specifications. It runs in parallel with Katana and GAU.

1. Core Kiterunner Settings

Parameter	Type	Default	Description
`KITERUNNER_ENABLED`	`bool`	`True`	Enable/disable Kiterunner discovery
(Binary auto-download)	-	`~/.redamon/tools/kiterunner/kr`	Auto-downloaded from GitHub releases
`KITERUNNER_WORDLIST`	`str`	`"apiroutes-251227"`	Wordlist (354k+ API routes)
`KITERUNNER_RATE_LIMIT`	`int`	`100`	Requests per second
`KITERUNNER_CONNECTIONS`	`int`	`100`	Concurrent connections
`KITERUNNER_TIMEOUT`	`int`	`10`	Request timeout per endpoint (seconds)
`KITERUNNER_SCAN_TIMEOUT`	`int`	`300`	Overall scan timeout (seconds)
`KITERUNNER_THREADS`	`int`	`50`	Scanning threads

2. Kiterunner Wordlists

Run kr wordlist list to see all available wordlists.

Wordlist	Description	Routes
`apiroutes-251227`	Comprehensive API routes (default)	354,000+
`aspx-251227`	ASP.NET specific routes	~82,000
`jsp-251227`	JSP/Java specific routes	~21,000
`php-251227`	PHP specific routes	~178,000
`directories-251227`	Directory discovery	~703,000
Custom path	Your own wordlist	Variable

3. Filtering Options

Parameter	Type	Default	Description
`KITERUNNER_IGNORE_STATUS`	`list`	`[404, 400, 502, 503]`	Status codes to ignore
`KITERUNNER_MATCH_STATUS`	`list`	`[]`	Only match these status codes (empty = all)
`KITERUNNER_MIN_CONTENT_LENGTH`	`int`	`0`	Ignore responses smaller than this
`KITERUNNER_HEADERS`	`list`	`[]`	Custom headers for authenticated scanning

4. Kiterunner Configuration Profiles

Minimal (Fast)

KITERUNNER_ENABLED = True
KITERUNNER_WORDLIST = "apiroutes-251227"
KITERUNNER_RATE_LIMIT = 200
KITERUNNER_SCAN_TIMEOUT = 120

Expected: ~1-2 minutes per target

Balanced (Default)

KITERUNNER_ENABLED = True
KITERUNNER_WORDLIST = "apiroutes-251227"
KITERUNNER_RATE_LIMIT = 100
KITERUNNER_CONNECTIONS = 100
KITERUNNER_SCAN_TIMEOUT = 300

Expected: ~3-5 minutes per target

Comprehensive (Stealth)

KITERUNNER_ENABLED = True
KITERUNNER_WORDLIST = "apiroutes-251227"
KITERUNNER_RATE_LIMIT = 50
KITERUNNER_CONNECTIONS = 50
KITERUNNER_SCAN_TIMEOUT = 600

Expected: ~5-10 minutes per target

5. What Kiterunner Finds

Category	Examples	Why It Matters
Hidden REST APIs	`/api/v1/users`, `/api/admin/config`	Undocumented functionality
GraphQL Endpoints	`/graphql`, `/gql`, `/api/graphql`	Complex query surface
Internal APIs	`/internal/`, `/private/`, `/debug/`	Bypass access controls
Version Endpoints	`/version`, `/health`, `/status`	Information disclosure
CRUD Operations	`/users/create`, `/posts/delete`	Data manipulation
Swagger/OpenAPI	`/swagger.json`, `/api-docs`	API documentation exposure

6. Why Kiterunner Over Traditional Wordlists?

Feature	Traditional Wordlists	Kiterunner
Route coverage	Limited to common paths	40k+ real API routes
HTTP Methods	Usually GET only	Correct method per route
Parameters	None	Swagger-defined params
Headers	None	API-specific headers
False positives	Many 404s	Validated responses

Hakrawler Configuration

Hakrawler is a DOM-aware web crawler that runs as a Docker container (jauderho/hakrawler). It runs in parallel with Katana, GAU, and Kiterunner, providing an additional crawling perspective with scope-aware link following.

1. Core Hakrawler Settings

Parameter	Type	Default	Description
`HAKRAWLER_ENABLED`	`bool`	`True`	Enable/disable Hakrawler crawling
`HAKRAWLER_DOCKER_IMAGE`	`str`	`"jauderho/hakrawler:latest"`	Docker image to use
`HAKRAWLER_DEPTH`	`int`	`2`	Crawl depth (how many links deep to follow)
`HAKRAWLER_THREADS`	`int`	`5`	Concurrent threads
`HAKRAWLER_TIMEOUT`	`int`	`30`	Per-URL timeout in seconds
`HAKRAWLER_MAX_URLS`	`int`	`500`	Maximum URLs to discover
`HAKRAWLER_INCLUDE_SUBS`	`bool`	`True`	Include subdomains in crawl scope
`HAKRAWLER_INSECURE`	`bool`	`True`	Skip TLS certificate verification
`HAKRAWLER_CUSTOM_HEADERS`	`list`	`[]`	Custom HTTP headers

2. Scope Filtering

Hakrawler scope filtering works at two levels:

Level	Mechanism	Description
Crawl scope	`-subs` flag	If `INCLUDE_SUBS=True`, Hakrawler follows links to subdomains of the target
Output scope	Hostname exact match	After crawling, results are filtered against `target_domains` set (from http_probe). URLs with hostnames not in scope are captured as `ExternalDomain` nodes instead of `Endpoint` nodes

3. What Hakrawler Finds

Category	Examples	Why It Matters
DOM links	`<a href="...">`, `<link>`, `<script src>`	Links that JS-rendered crawlers may miss
Form actions	`<form action="/submit">`	POST endpoints for parameter fuzzing
Asset references	CSS imports, image URLs, font files	Reveals directory structure
Subdomain links	Links pointing to `api.example.com`	Discovers related subdomains
External domains	Links to third-party services	Maps external dependencies

4. Stealth Mode

When STEALTH_MODE is enabled, Hakrawler is automatically disabled to reduce the active crawling footprint. jsluice max files is reduced to 20.

jsluice Configuration

jsluice is a JavaScript analysis tool compiled into the recon container (no Docker image needed). It downloads JavaScript files already discovered by Katana/Hakrawler from the target and analyzes their contents locally to extract URLs, API endpoints, and embedded secrets.

1. Core jsluice Settings

Parameter	Type	Default	Description
`JSLUICE_ENABLED`	`bool`	`True`	Enable/disable jsluice analysis
`JSLUICE_MAX_FILES`	`int`	`50`	Maximum number of JS files to analyze
`JSLUICE_TIMEOUT`	`int`	`120`	Overall timeout in seconds
`JSLUICE_CONCURRENCY`	`int`	`5`	Files to process concurrently
`JSLUICE_EXTRACT_URLS`	`bool`	`True`	Run URL extraction mode
`JSLUICE_EXTRACT_SECRETS`	`bool`	`True`	Run secret detection mode

2. How jsluice Works

jsluice sends HTTP requests to download each JS file from the target, then analyzes them locally:

Filter: Selects .js and .mjs URLs from all URLs discovered by Katana/Hakrawler/GAU
Download: Fetches JS files to a temporary directory (/tmp/redamon/jsluice_<pid>/)
Analyze URLs: Runs jsluice urls to extract embedded URLs and API endpoints
Analyze Secrets: Runs jsluice secrets to detect API keys, tokens, credentials
Scope filter: Extracted URLs are filtered against allowed_hosts; out-of-scope URLs become ExternalDomain nodes
Cleanup: Temporary files are deleted in a finally block

3. What jsluice Finds

URL Extraction:

Category	Examples	Why It Matters
API endpoints	`/api/v2/users`, `/graphql`	Hidden backend routes
Internal URLs	`/admin/config`, `/debug/vars`	Undocumented functionality
External services	`https://api.stripe.com/v1/charges`	Third-party integrations
CDN/asset paths	`/static/js/`, `/assets/`	Directory structure

Secret Detection:

Secret Type	Examples	Severity
AWS Access Keys	`AKIA...`	High
GitHub Tokens	`ghp_...`, `gho_...`	High
GCP Credentials	`AIza...`	High
API Keys	Generic API key patterns	Medium
Private Keys	`-----BEGIN RSA PRIVATE KEY-----`	High
JWT Tokens	`eyJ...`	Medium

Discovered secrets are stored as Secret nodes in Neo4j, linked to their parent BaseURL via [:HAS_SECRET].

4. jsluice in the Pipeline

jsluice runs sequentially after the parallel crawling phase (Katana + Hakrawler + GAU + Kiterunner), because it needs their discovered URLs as input. It adds minimal time since JS file analysis is CPU-bound (no network scanning).

Architecture & Flow

Pipeline context: Resource enumeration runs in GROUP 5 of the parallelized recon pipeline, after HTTP probing (GROUP 4). The four discovery tools (Katana, Hakrawler, GAU, Kiterunner) run concurrently via an internal ThreadPoolExecutor. jsluice then runs sequentially on discovered JS files. Graph DB updates happen in a background thread.

Execution Flow

1. INITIALIZATION
   └── Check Docker availability
   └── Pull Katana + GAU + Kiterunner images in parallel

2. TARGET EXTRACTION
   └── Get live URLs from http_probe
   └── Extract domains for GAU
   └── Filter by status code (< 500)
   └── Fallback to DNS data if no http_probe

3. PARALLEL DISCOVERY (Katana + Hakrawler + GAU + Kiterunner)
   ┌───────────────────────────────────────────────────────────────────────────┐
   │  ThreadPoolExecutor (max_workers=4)                                       │
   │                                                                           │
   │  ┌───────────┐  ┌────────────┐  ┌───────────┐  ┌────────────────────┐    │
   │  │  KATANA   │  │ HAKRAWLER  │  │    GAU    │  │    KITERUNNER      │    │
   │  │  (active) │  │  (active)  │  │ (passive) │  │    (API brute)     │    │
   │  │           │  │            │  │           │  │                    │    │
   │  │ - Crawl   │  │ - DOM-walk │  │ - Wayback │  │ - Swagger specs    │    │
   │  │ - Parse JS│  │ - Docker   │  │ - CmnCrl  │  │ - 40k+ API routes  │    │
   │  │ - Find URL│  │ - Scope    │  │ - OTX     │  │ - Method detection │    │
   │  │           │  │            │  │ - URLScan │  │                    │    │
   │  └─────┬─────┘  └─────┬──────┘  └─────┬─────┘  └─────────┬──────────┘    │
   │        │               │               │                  │               │
   └────────┴───────────────┴───────────────┴──────────────────┴───────────────┘
                              │
                              ▼
3b. JSLUICE ANALYSIS (sequential, after parallel discovery)
   └── Filter discovered URLs to .js/.mjs files
   └── Download JS files to /tmp/redamon/jsluice_<pid>/
   └── Run jsluice urls (extract endpoints)
   └── Run jsluice secrets (detect API keys, tokens, credentials)
   └── Scope-filter extracted URLs
   └── Cleanup temp files

4. GAU URL VERIFICATION (if enabled)
   └── Write GAU URLs to temp file
   └── Run httpx Docker for verification
   └── Filter to live URLs only

5. METHOD DETECTION (if enabled)
   └── Send OPTIONS request to each verified URL
   └── Parse 'Allow' header for supported methods
   └── Fall back to GET check if OPTIONS fails
   └── Filter out dead endpoints (404/500/timeout)

6. MERGE & DEDUPLICATE
   └── Mark Katana endpoints with sources=['katana']
   └── Merge Hakrawler URLs, add 'hakrawler' to sources array
   └── Merge GAU URLs, add 'gau' to sources array
   └── Merge Kiterunner APIs, add 'kiterunner' to sources array
   └── Merge jsluice URLs, add 'jsluice' to sources array
   └── Apply detected methods to endpoints
   └── Track overlap statistics for each tool

7. FORM PARSING
   └── Extract HTML from http_probe responses
   └── Parse <form> elements
   └── Extract action URLs and methods
   └── Extract input fields

8. ENDPOINT ORGANIZATION
   └── Group by base URL
   └── Parse query parameters
   └── Merge form data
   └── Classify endpoints
   └── Classify parameters

9. OUTPUT GENERATION
   └── Build structured JSON
   └── Include GAU + Hakrawler + Kiterunner + jsluice stats
   └── Generate summary statistics
   └── Save to recon file

Data Flow Diagram

┌──────────────────────────────────────────────────────────────────┐
│                        http_probe data                           │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐     │
│  │ live URLs      │  │ response bodies │  │ status codes   │     │
│  │ (for crawling) │  │ (for forms)     │  │ (filtering)    │     │
│  └───────┬────────┘  └───────┬────────┘  └───────┬────────┘     │
└──────────┼───────────────────┼───────────────────┼──────────────┘
           │                   │                   │
           ▼                   ▼                   │
    ┌──────────────┐    ┌──────────────┐          │
    │ Katana       │    │ Form Parser  │          │
    │ Crawler      │    │              │          │
    └──────┬───────┘    └──────┬───────┘          │
           │                   │                   │
           ▼                   ▼                   │
    ┌──────────────────────────────────────┐      │
    │          organize_endpoints()         │◄─────┘
    │  - Parse URLs                         │
    │  - Extract parameters                 │
    │  - Merge form data                    │
    │  - Classify endpoints                 │
    │  - Classify parameters                │
    └──────────────────┬───────────────────┘
                       │
                       ▼
    ┌──────────────────────────────────────┐
    │          resource_enum result         │
    │  - by_base_url (organized endpoints)  │
    │  - forms (POST endpoints)             │
    │  - discovered_urls (raw list)         │
    │  - summary (statistics)               │
    └──────────────────────────────────────┘

Output Data Structure

Complete JSON Schema

{
  "resource_enum": {
    "scan_metadata": {
      "scan_timestamp": "2024-01-15T12:00:00.000000",
      "scan_duration_seconds": 145.5,

      "katana_docker_image": "projectdiscovery/katana:latest",
      "katana_crawl_depth": 3,
      "katana_max_urls": 1000,
      "katana_rate_limit": 150,
      "katana_js_crawl": true,
      "katana_params_only": false,
      "katana_urls_found": 234,

      "gau_enabled": true,
      "gau_docker_image": "sxcurity/gau:latest",
      "gau_providers": ["wayback", "commoncrawl", "otx", "urlscan"],
      "gau_urls_found": 156,
      "gau_verify_enabled": true,
      "gau_method_detection_enabled": true,
      "gau_filter_dead_endpoints": true,
      "gau_stats": {
        "gau_total": 156,
        "gau_parsed": 142,
        "gau_new": 87,
        "gau_overlap": 55,
        "gau_skipped_unverified": 14,
        "gau_skipped_dead": 8,
        "gau_with_post": 23,
        "gau_with_multiple_methods": 15
      },

      "kiterunner_enabled": true,
      "kiterunner_binary_path": "~/.redamon/tools/kiterunner/kr",
      "kiterunner_wordlist": "apiroutes-251227",
      "kiterunner_endpoints_found": 45,
      "kiterunner_stats": {
        "kr_total": 45,
        "kr_parsed": 45,
        "kr_new": 38,
        "kr_overlap": 7,
        "kr_methods": {"GET": 30, "POST": 12, "PUT": 3}
      },

      "hakrawler_enabled": true,
      "hakrawler_docker_image": "jauderho/hakrawler:latest",
      "hakrawler_depth": 2,
      "hakrawler_threads": 5,
      "hakrawler_urls_found": 78,

      "jsluice_enabled": true,
      "jsluice_max_files": 50,
      "jsluice_urls_found": 42,
      "jsluice_secrets_found": 3,

      "proxy_used": false,
      "target_urls_count": 5,
      "target_domains_count": 3,
      "total_discovered_urls": 321
    },

    "discovered_urls": [
      "https://example.com/",
      "https://example.com/login?redirect=/dashboard",
      "https://example.com/api/v1/users?id=1",
      "https://example.com/old-admin/config.php",
      "https://example.com/search?q=test"
    ],

    "by_base_url": {
      "https://example.com": {
        "base_url": "https://example.com",
        "endpoints": {
          "/login": {
            "path": "/login",
            "methods": ["GET", "POST"],
            "parameters": {
              "query": [
                {
                  "name": "redirect",
                  "type": "url",
                  "sample_values": ["/dashboard", "/home"],
                  "category": "redirect_params"
                }
              ],
              "body": [
                {
                  "name": "username",
                  "type": "string",
                  "input_type": "text",
                  "required": true,
                  "category": "auth_params"
                },
                {
                  "name": "password",
                  "type": "string",
                  "input_type": "password",
                  "required": true,
                  "category": "auth_params"
                }
              ],
              "path": []
            },
            "sample_urls": ["https://example.com/login?redirect=/dashboard"],
            "urls_found": 3,
            "category": "authentication",
            "sources": ["katana", "gau"],
            "parameter_count": {
              "query": 1,
              "body": 2,
              "path": 0,
              "total": 3
            }
          },
          "/api/v1/users": {
            "path": "/api/v1/users",
            "methods": ["GET"],
            "parameters": {
              "query": [
                {
                  "name": "id",
                  "type": "integer",
                  "sample_values": ["1", "2", "100"],
                  "category": "id_params"
                }
              ],
              "body": [],
              "path": []
            },
            "sample_urls": ["https://example.com/api/v1/users?id=1"],
            "urls_found": 5,
            "category": "api",
            "sources": ["katana"],
            "parameter_count": {
              "query": 1,
              "body": 0,
              "path": 0,
              "total": 1
            }
          },
          "/old-admin/config.php": {
            "path": "/old-admin/config.php",
            "methods": ["GET"],
            "parameters": {
              "query": [
                {
                  "name": "debug",
                  "category": "other"
                }
              ],
              "body": [],
              "path": []
            },
            "sample_urls": ["https://example.com/old-admin/config.php?debug=1"],
            "category": "admin",
            "sources": ["gau"],
            "parameter_count": {
              "query": 1,
              "body": 0,
              "path": 0,
              "total": 1
            }
          }
        },
        "summary": {
          "total_endpoints": 15,
          "total_parameters": 23,
          "methods": {
            "GET": 12,
            "POST": 3
          },
          "categories": {
            "api": 5,
            "authentication": 2,
            "dynamic": 4,
            "static": 3,
            "search": 1
          }
        }
      }
    },

    "forms": [
      {
        "action": "https://example.com/login",
        "method": "POST",
        "enctype": "application/x-www-form-urlencoded",
        "found_at": "https://example.com/login",
        "inputs": [
          {"name": "username", "type": "text", "value": "", "required": true},
          {"name": "password", "type": "password", "value": "", "required": true},
          {"name": "remember", "type": "checkbox", "value": "1", "required": false}
        ]
      },
      {
        "action": "https://example.com/upload",
        "method": "POST",
        "enctype": "multipart/form-data",
        "found_at": "https://example.com/dashboard",
        "inputs": [
          {"name": "file", "type": "file", "value": "", "required": true},
          {"name": "description", "type": "text", "value": "", "required": false}
        ]
      }
    ],

    "summary": {
      "total_base_urls": 3,
      "total_endpoints": 45,
      "total_parameters": 78,
      "total_forms": 5,
      "from_katana": 234,
      "from_gau": 156,
      "gau_new_endpoints": 87,
      "gau_overlap": 55,
      "methods": {
        "GET": 38,
        "POST": 7
      },
      "categories": {
        "api": 15,
        "dynamic": 12,
        "static": 8,
        "authentication": 4,
        "search": 3,
        "admin": 2,
        "file_access": 1
      }
    }
  }
}

Endpoint Sources Field

Each endpoint includes a sources array indicating where it was discovered:

Example	Meaning
`["katana"]`	Found only by Katana active crawling
`["hakrawler"]`	Found only by Hakrawler DOM crawling
`["gau"]`	Found only by GAU passive discovery
`["kiterunner"]`	Found only by Kiterunner API bruteforce
`["jsluice"]`	Found only by jsluice JavaScript analysis
`["katana", "hakrawler"]`	Found by both active crawlers
`["katana", "gau"]`	Found by both Katana and GAU
`["katana", "hakrawler", "gau", "kiterunner", "jsluice"]`	Found by all five tools

Why Array Format?

With 5 discovery tools, a simple string can't capture all combinations
Arrays allow precise tracking of which tools found each endpoint
Helps prioritize endpoints found by multiple tools (higher confidence)

Endpoint Classification

The module automatically classifies endpoints into categories based on URL patterns, HTTP methods, and parameters.

Category	Detection Patterns	Security Relevance
authentication	`/login`, `/signup`, `/auth`, `/token`, body params with `username`/`password`	Credential stuffing, brute force
admin	`/admin`, `/dashboard`, `/panel`, `/wp-admin`	Privilege escalation
api	`/api/`, `/v1/`, `/v2/`, `/rest/`, `/graphql`	API abuse, IDOR
file_access	`/download`, `/file`, `/image`, `/attachment`	LFI, path traversal
upload	`/upload`, `/import`	Malicious file upload
search	`/search`, `/find`, `/query`	SQL injection, XSS
dynamic	`.php`, `.asp`, `.jsp`, or URLs with params	Various injection attacks
static	`.html`, `.css`, `.js`, images	Low priority
other	Everything else	Manual review

Classification Logic

def classify_endpoint(path, methods, params):
    # Priority order:
    # 1. Check path patterns (auth, admin, api, file, search)
    # 2. Check body parameters for auth indicators
    # 3. Check file extension (static vs dynamic)
    # 4. Check for query parameters (dynamic)
    # 5. Default to "other"

Parameter Classification

Parameters are classified to identify potentially vulnerable inputs.

Parameter Categories

Category	Examples	Vulnerability Risk
id_params	`id`, `user_id`, `product_id`, `cat`	IDOR, SQL injection
file_params	`file`, `path`, `template`, `include`	LFI, RFI, path traversal
search_params	`q`, `query`, `search`, `keyword`	SQL injection, XSS
auth_params	`username`, `password`, `token`, `apikey`	Credential exposure
redirect_params	`url`, `redirect`, `next`, `callback`	Open redirect, SSRF
command_params	`cmd`, `exec`, `host`, `ip`	Command injection
other	Everything else	Context-dependent

Type Inference

The module infers parameter data types from names and sample values:

Type	Detection Method	Example
`integer`	Numeric values, names like `id`, `page`	`id=123`
`email`	Contains `@` and `.`	`email=user@example.com`
`url`	Starts with `http://` or `https://`	`redirect=https://...`
`path`	Contains `/`, `\`, or file extensions	`file=../etc/passwd`
`datetime`	Names like `date`, `time`, `timestamp`	`created_at=...`
`boolean`	Names like `enabled`, `active`, `is_*`	`active=true`
`string`	Default	Everything else

Form Parsing

The module parses HTML to extract form elements and their inputs.

Extracted Form Information

{
  "action": "https://example.com/login",
  "method": "POST",
  "enctype": "application/x-www-form-urlencoded",
  "found_at": "https://example.com/",
  "inputs": [
    {
      "name": "username",
      "type": "text",
      "value": "",
      "required": true,
      "placeholder": "Enter username"
    },
    {
      "name": "password",
      "type": "password",
      "value": "",
      "required": true
    }
  ]
}

Supported Input Types

HTML Element	Extracted Info
`<form>`	action, method, enctype
`<input>`	name, type, value, required, placeholder
`<textarea>`	name, required
`<select>`	name, required
`<button type="submit">`	name, value

Form Data in Endpoints

Forms are merged into the endpoint structure:

Form action URL becomes the endpoint path
Form method is added to endpoint methods
Form inputs become body parameters with input_type field

Usage Examples

Basic Usage (via main.py)

# Include "resource_enum" in SCAN_MODULES in project settings
SCAN_MODULES = ["domain_discovery", "port_scan", "http_probe", "resource_enum", "vuln_scan"]

# Run the full pipeline
python3 recon/main.py

Standalone Enrichment

from resource_enum import run_resource_enum
from pathlib import Path
import json

# Load existing recon data
with open("output/recon_example.com.json", "r") as f:
    recon_data = json.load(f)

# Run resource enumeration
enriched = run_resource_enum(recon_data, output_file=Path("output/recon_example.com.json"))

Command Line

# Enrich an existing recon file
python3 recon/resource_enum.py output/recon_example.com.json

Using Results in vuln_scan

The vuln_scan module automatically uses resource_enum data:

# vuln_scan.py - build_target_urls()

# Priority 1: Use resource_enum endpoints (most comprehensive)
resource_enum_data = recon_data.get("resource_enum")
if resource_enum_data:
    base_urls, endpoint_urls = build_target_urls_from_resource_enum(resource_enum_data)
    # Returns both base URLs and URLs with parameters for comprehensive scanning

Integration with Graph Database

Resource enumeration data is stored in Neo4j:

Node Types

Node	Properties
Endpoint	path, method, category, has_parameters, query_param_count, body_param_count
Parameter	name, position (query/body), type, category, sample_values
Secret	secret_type, severity, source, source_url, base_url, sample
ExternalDomain	domain, source (katana, hakrawler, jsluice, gau), url

Relationships

(BaseURL) -[:HAS_ENDPOINT]-> (Endpoint) -[:HAS_PARAMETER]-> (Parameter)
(BaseURL) -[:HAS_SECRET]-> (Secret)
(Domain) -[:HAS_EXTERNAL_DOMAIN]-> (ExternalDomain)

Example Cypher Queries

// Find all authentication endpoints
MATCH (e:Endpoint {category: 'authentication'})
RETURN e.path, e.method

// Find endpoints with file parameters (LFI risk)
MATCH (e:Endpoint)-[:HAS_PARAMETER]->(p:Parameter {category: 'file_params'})
RETURN e.path, p.name

// Find all POST forms
MATCH (e:Endpoint {method: 'POST', is_form: true})
RETURN e.path, e.form_found_at

// Find secrets discovered in JavaScript files
MATCH (b:BaseURL)-[:HAS_SECRET]->(s:Secret)
WHERE s.severity IN ['high', 'critical']
RETURN b.url, s.secret_type, s.source_url, s.sample

// Find endpoints discovered by Hakrawler but not Katana
MATCH (e:Endpoint)
WHERE 'hakrawler' IN e.sources AND NOT 'katana' IN e.sources
RETURN e.path, e.method

# Install Docker
sudo apt install docker.io

# Start Docker daemon
sudo systemctl start docker

"No URLs discovered"

Possible causes:

JavaScript-heavy site (SPAs)
WAF blocking crawler
Rate limiting

Solutions:

# Increase depth
KATANA_DEPTH = 5

# Enable JS crawling
KATANA_JS_CRAWL = True

# Reduce rate limit
KATANA_RATE_LIMIT = 50

"Too many URLs (noise)"

# Enable params-only mode
KATANA_PARAMS_ONLY = True

# Add exclude patterns
KATANA_EXCLUDE_PATTERNS = [
    "/static/",
    "/assets/",
    "/wp-content/",
    ".css", ".js", ".jpg", ".png"
]

# Reduce max URLs
KATANA_MAX_URLS = 500

"Crawl taking too long"

# Reduce depth
KATANA_DEPTH = 2

# Reduce timeout
KATANA_TIMEOUT = 120

# Increase rate limit
KATANA_RATE_LIMIT = 200

# Disable JS crawling
KATANA_JS_CRAWL = False

"GAU returning too many URLs"

# Limit URLs per domain
GAU_MAX_URLS = 500

# Use fewer providers
GAU_PROVIDERS = ["wayback"]  # Only Wayback Machine

# Filter by date range
GAU_YEAR_RANGE = ["2022", "2024"]

# Add more extensions to blacklist
GAU_BLACKLIST_EXTENSIONS.extend(["aspx", "jsp"])

"GAU URLs not being added to results"

Possible causes:

URL verification filtering them out
URLs from different domains (subdomains disabled)
Extension blacklist filtering

Solutions:

# Disable verification to see all URLs
GAU_VERIFY_URLS = False

# Check blacklist isn't too aggressive
GAU_BLACKLIST_EXTENSIONS = ["png", "jpg", "gif", "css"]  # Minimal

"GAU timeout errors"

# Increase timeout
GAU_TIMEOUT = 120

# Reduce providers
GAU_PROVIDERS = ["wayback", "commoncrawl"]  # Skip slower ones

# Reduce threads
GAU_THREADS = 2

"Kiterunner not finding endpoints"

Possible causes:

Target doesn't have REST APIs
WAF blocking bruteforce attempts
APIs use non-standard routes

Solutions:

# Try different wordlist
KITERUNNER_WORDLIST = "aspx-251227"  # For ASP.NET

# Reduce rate to avoid WAF
KITERUNNER_RATE_LIMIT = 50

# Add authentication headers
KITERUNNER_HEADERS = ["Authorization: Bearer <token>"]

"Kiterunner timeout errors"

# Increase scan timeout
KITERUNNER_SCAN_TIMEOUT = 600

# Reduce concurrent connections
KITERUNNER_CONNECTIONS = 50

# Increase per-request timeout
KITERUNNER_TIMEOUT = 15

"Kiterunner too aggressive (WAF blocks)"

# Stealth mode
KITERUNNER_RATE_LIMIT = 30
KITERUNNER_CONNECTIONS = 20
KITERUNNER_THREADS = 10

Debug Mode

Run Katana manually via Docker:

docker run --rm \
  projectdiscovery/katana:latest \
  -u https://example.com \
  -d 2 \
  -jc \
  -silent

Run GAU manually via Docker:

docker run --rm \
  sxcurity/gau:latest \
  --threads 5 \
  --timeout 60 \
  --providers wayback,commoncrawl \
  example.com

Run Kiterunner manually (binary auto-downloads to ~/.redamon/tools/kiterunner/):

# Binary location after first run
# Use -A flag for auto-downloaded wordlists
~/.redamon/tools/kiterunner/kr scan https://example.com \
  -A apiroutes-251227:20000 \
  -x 50 \
  -j 25 \
  -t 10s

Security Considerations

Risk	Mitigation
Rate limiting/bans	Reduce `KATANA_RATE_LIMIT` and `KITERUNNER_RATE_LIMIT`
WAF blocking	Use custom User-Agent, reduce rate
API bruteforce detection	Lower `KITERUNNER_CONNECTIONS` and `KITERUNNER_THREADS`
Legal issues	Only scan authorized targets

Safe Defaults

# Katana (Active Crawling)
KATANA_RATE_LIMIT = 50
KATANA_DEPTH = 2
KATANA_TIMEOUT = 120
KATANA_CUSTOM_HEADERS = [
    "User-Agent: Mozilla/5.0 (compatible; SecurityScanner/1.0)"
]

# Kiterunner (API Bruteforce)
KITERUNNER_RATE_LIMIT = 50
KITERUNNER_CONNECTIONS = 30
KITERUNNER_THREADS = 20

Dependencies

Package	Purpose
Docker	Container runtime for Katana, GAU, and httpx
`projectdiscovery/katana:latest`	Katana Docker image (auto-pulled)
`sxcurity/gau:latest`	GAU Docker image (auto-pulled)
Kiterunner binary	Auto-downloaded from GitHub releases to `~/.redamon/tools/kiterunner/`
`projectdiscovery/httpx:latest`	httpx Docker image for URL verification (auto-pulled)
Python 3.8+	Script runtime
`html.parser`	Built-in HTML form parsing
`urllib.parse`	Built-in URL parsing for GAU endpoint extraction

Downstream — AI Surface Recon (Phase 4.5): consumes the endpoints classified here (ai_interface_type = llm-chat/mcp/…) and actively confirms the AI/LLM/MCP/vector-DB surfaces with benign protocol probes.

References

Documentation generated for RedAmon v1.0 - Resource Enumeration Module