RedAmon - httpx HTTP Prober + Banner Grabbing

August 10, 2026 · View on GitHub

Complete Technical Documentation

Module: recon/httpx_scan.py
Purpose: HTTP probing, technology detection, and non-HTTP service detection
Author: RedAmon Security Suite

Overview
Features
Installation
Configuration Parameters
Banner Grabbing
Architecture & Flow
Output Data Structure
Usage Examples
Troubleshooting

Overview

The httpx_scan.py module integrates ProjectDiscovery's httpx toolkit into RedAmon's reconnaissance pipeline. httpx is a multi-purpose HTTP toolkit designed for probing, technology detection, and web service analysis.

This module also includes integrated banner grabbing for non-HTTP services (SSH, FTP, SMTP, MySQL, etc.) to provide complete service detection across all open ports.

⚠️ Important: httpx runs exclusively via Docker. No native installation is supported.

Feature	Traditional Tools	httpx + Banner Grab
HTTP Technology Detection	Limited	Wappalyzer-based
Non-HTTP Service Detection	None	SSH, FTP, SMTP, etc.
TLS Analysis	Basic	Comprehensive
Speed	Slow	Highly concurrent
Output Format	Variable	Rich JSON
Fingerprinting	None	Favicon, JARM, hash

How It Works

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Naabu Results  │────▶│  httpx_scan.py   │────▶│  Enriched JSON  │
│  (open ports,   │     │                  │     │  with live URLs,│
│   hostnames)    │     │  1. Build URLs   │     │  technologies,  │
└─────────────────┘     │  2. HTTP probe   │     │  TLS info,      │
        │               │  3. Tech detect  │     │  banners, etc.  │
        │               │  4. Banner grab  │     └─────────────────┘
        │               │  5. Parse JSON   │
        ▼               └──────────────────┘
┌─────────────────┐
│  DNS Data       │  (Fallback if no Naabu)
│  (hostnames)    │
└─────────────────┘

Features

Feature	Description
HTTP Probing	Identifies live HTTP/HTTPS endpoints
Technology Detection	Wappalyzer-based tech fingerprinting
TLS Analysis	Certificate info, cipher suites, JARM
CDN Detection	Identifies CDN providers
ASN Lookup	Autonomous System Number detection
Response Analysis	Status codes, headers, body hashing
Favicon Hashing	Fingerprint via favicon hash
Banner Grabbing	Service detection for non-HTTP ports (SSH, FTP, SMTP, etc.)
Docker Execution	No local installation required

Installation

Requirements

Docker installed and running

Setup

# Make sure Docker is running
sudo systemctl start docker

# Run the scan - image will be pulled automatically
python3 recon/main.py

Verify Docker is Ready

# Check Docker is running
docker info

# Optionally pre-pull the image
docker pull projectdiscovery/httpx:latest

Configuration Parameters

All parameters are configured via the webapp project settings (stored in PostgreSQL) or as defaults in project_settings.py. This section provides detailed explanations including performance impact for each option.

1. Core Configuration

Parameter	Type	Default	httpx Flag	Description
`HTTPX_DOCKER_IMAGE`	`str`	`"projectdiscovery/httpx:latest"`	-	Docker image to use. Only change for custom builds or specific versions.
`HTTPX_THREADS`	`int`	`50`	`-t`	Number of concurrent HTTP requests. Higher = faster but uses more CPU/memory.
`HTTPX_TIMEOUT`	`int`	`10`	`-timeout`	Seconds to wait for each HTTP response before giving up.
`HTTPX_RETRIES`	`int`	`2`	`-retries`	Number of retry attempts for failed/timeout requests.
`HTTPX_RATE_LIMIT`	`int`	`150`	`-rl`	Maximum requests per second. Set to `0` for unlimited. Lower = more stealthy.

Thread Tuning Guide:

Threads	Use Case	CPU Usage	Memory
10-25	Stealth scanning, limited bandwidth	Low	~100MB
50	Default - balanced performance	Medium	~200MB
100-200	Fast scanning, good hardware	High	~500MB
300+	Maximum speed, powerful servers only	Very High	1GB+

Timeout Considerations:

5s - Fast networks, modern servers
10s - Default - handles most scenarios
15-30s - Slow servers, high latency networks

2. Redirect Handling

Parameter	Type	Default	httpx Flag	Description
`HTTPX_FOLLOW_REDIRECTS`	`bool`	`True`	`-fr`	Follow HTTP 301/302/307/308 redirects. Essential for modern web apps.
`HTTPX_MAX_REDIRECTS`	`int`	`10`	`-maxr`	Maximum redirect chain length. Prevents infinite redirect loops.

Why Follow Redirects?

Many sites redirect http:// → https://
www → non-www redirects are common
Login pages often redirect after authentication
Without following, you might miss the actual content

3. HTTP Response Probing (Fast Options)

These options extract basic HTTP response information with minimal performance impact:

Parameter	Type	Default	httpx Flag	What It Extracts	Speed Impact
`HTTPX_PROBE_STATUS_CODE`	`bool`	`True`	`-sc`	HTTP status code (200, 301, 404, 500, etc.)	✅ None
`HTTPX_PROBE_CONTENT_LENGTH`	`bool`	`True`	`-cl`	Response body size in bytes	✅ None
`HTTPX_PROBE_CONTENT_TYPE`	`bool`	`True`	`-ct`	MIME type (`text/html`, `application/json`, etc.)	✅ None
`HTTPX_PROBE_TITLE`	`bool`	`True`	`-title`	HTML `<title>` tag content	✅ Minimal
`HTTPX_PROBE_SERVER`	`bool`	`True`	`-server`	Server header value (nginx, Apache, IIS, etc.)	✅ None
`HTTPX_PROBE_RESPONSE_TIME`	`bool`	`True`	`-rt`	Time to first byte in milliseconds	✅ None
`HTTPX_PROBE_WORD_COUNT`	`bool`	`True`	`-wc`	Number of words in response body	✅ Minimal
`HTTPX_PROBE_LINE_COUNT`	`bool`	`True`	`-lc`	Number of lines in response body	✅ Minimal

Example Output:

{
  "status_code": 200,
  "content_length": 15234,
  "content_type": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "server": "nginx/1.19.0",
  "response_time_ms": 145,
  "word_count": 892,
  "line_count": 156
}

4. Technology Detection

Parameter	Type	Default	httpx Flag	Description	Speed Impact
`HTTPX_PROBE_TECH_DETECT`	`bool`	`True`	`-td`	Detects technologies using Wappalyzer-like fingerprinting	⚡ Medium

How It Works:

Analyzes response headers, body content, and JavaScript
Matches against signature database
Identifies frameworks, CMS, libraries

Detected Technologies Include:

Category	Examples
Web Servers	nginx, Apache, IIS, LiteSpeed, Caddy
Languages	PHP, Python, Java, Node.js, Ruby, Go
Frameworks	Laravel, Django, Spring, Express, Rails
CMS	WordPress, Drupal, Joomla, Magento
Frontend	React, Angular, Vue.js, jQuery
CDN/WAF	Cloudflare, Akamai, AWS CloudFront, Sucuri
Analytics	Google Analytics, Matomo, Hotjar
Caching	Varnish, Redis, Memcached

5. Network Information

Parameter	Type	Default	httpx Flag	What It Does	Speed Impact
`HTTPX_PROBE_IP`	`bool`	`True`	`-ip`	Resolves and returns the target's IP address	✅ Minimal (DNS lookup)
`HTTPX_PROBE_CNAME`	`bool`	`True`	`-cname`	Extracts CNAME DNS records - reveals CDN/hosting	✅ Minimal (DNS lookup)
`HTTPX_PROBE_ASN`	`bool`	`True`	`-asn`	Autonomous System Number - identifies network provider	⚡ Medium (external lookup)
`HTTPX_PROBE_CDN`	`bool`	`True`	`-cdn`	Detects CDN provider (Cloudflare, Akamai, etc.)	✅ Minimal

ASN Information Reveals:

Hosting provider (AWS, Google Cloud, Azure, DigitalOcean)
ISP information
Geographic region hints
Useful for identifying shared hosting

6. SSL/TLS Information

Parameter	Type	Default	httpx Flag	What It Does	Speed Impact
`HTTPX_PROBE_TLS_INFO`	`bool`	`True`	`-tls-probe`	Basic TLS info: version, cipher suite	⚡ Medium
`HTTPX_PROBE_TLS_GRAB`	`bool`	`True`	`-tls-grab`	Full certificate extraction	⚠️ SLOW

TLS_INFO extracts:

TLS version (TLS 1.0, 1.1, 1.2, 1.3)
Cipher suite in use

TLS_GRAB extracts (additional):

Certificate subject (CN, O, OU)
Certificate issuer (CA)
Validity period (not_before, not_after)
Subject Alternative Names (all domains on cert)
Serial number

Example TLS Output:

{
  "tls": {
    "version": "TLS 1.3",
    "cipher": "TLS_AES_256_GCM_SHA384",
    "certificate": {
      "subject_cn": "*.example.com",
      "issuer": "Let's Encrypt Authority X3",
      "not_before": "2024-01-01T00:00:00Z",
      "not_after": "2024-04-01T00:00:00Z",
      "san": ["example.com", "*.example.com", "www.example.com"]
    }
  }
}

⚠️ Performance Note: TLS_GRAB requires a full TLS handshake and certificate parsing. Disable for faster scans.

7. Fingerprinting (Advanced)

Parameter	Type	Default	httpx Flag	What It Does	Speed Impact
`HTTPX_PROBE_FAVICON`	`bool`	`True`	`-favicon`	Calculates MMH3 hash of favicon.ico	⚡ Medium (extra HTTP request)
`HTTPX_PROBE_JARM`	`bool`	`True`	`-jarm`	JARM TLS fingerprint	⚠️ VERY SLOW
`HTTPX_PROBE_HASH`	`str`	`"sha256"`	`-hash`	Hash of response body	⚡ Medium

Favicon Hashing

What It Is:

Downloads /favicon.ico and calculates MMH3 hash
Same favicon = same application/framework

Use Cases:

Identify Cobalt Strike C2 servers (known hashes)
Find admin panels (phpMyAdmin, cPanel)
Detect specific applications
Cross-reference with Shodan favicon database

Example:

{
  "favicon_hash": "-1396576824"
}

JARM Fingerprinting

What It Is:

JARM = JA3 + RDP fingerprint for servers
Creates unique fingerprint based on TLS server configuration
Requires 10 separate TLS handshakes per target

Why It's Slow:

Each JARM fingerprint requires:
- 10 different TLS ClientHello messages
- 10 separate connections
- Parsing 10 ServerHello responses
- Total: ~5-10 seconds per target

Use Cases:

Identify C2 (Command & Control) servers
Detect malware infrastructure
Fingerprint load balancers
Track threat actors

⚠️ Recommendation: Disable JARM (HTTPX_PROBE_JARM = False) for regular scans. Enable only for targeted threat hunting.

Body Hash

Options: "md5", "sha256", "sha512", "" (disabled)

Use Cases:

Detect identical pages across different URLs
Find mirror sites
Identify default/template pages

8. Response Content Inclusion

Parameter	Type	Default	httpx Flag	What It Does	Speed Impact
`HTTPX_INCLUDE_RESPONSE`	`bool`	`True`	`-irr`	Include full HTTP response (headers + body)	⚠️ SLOW + Large Output
`HTTPX_INCLUDE_RESPONSE_HEADERS`	`bool`	`True`	`-irh`	Include only response headers	⚡ Medium

Output Size Comparison:

Setting	Typical Size per URL
Headers only	1-5 KB
Headers + Body	50-500 KB
Full response	100+ KB

When to Enable Full Response:

✅ Detailed analysis of response content
✅ Looking for sensitive data exposure
✅ Custom pattern matching needed
❌ Large-scale scanning (creates huge files)
❌ Speed is priority

9. Advanced Options

Parameter	Type	Default	Description
`HTTPX_PATHS`	`list`	`[]`	Additional paths to probe beyond root
`HTTPX_CUSTOM_HEADERS`	`list`	`[]`	Custom HTTP headers to send
`HTTPX_MATCH_CODES`	`list`	`[]`	Only keep responses with these status codes
`HTTPX_FILTER_CODES`	`list`	`[]`	Exclude responses with these status codes

Path Probing

# Probe common sensitive paths
HTTPX_PATHS = [
    "/robots.txt",           # Discover hidden paths
    "/.well-known/security.txt",  # Security contact info
    "/sitemap.xml",          # Site structure
    "/.git/config",          # Git exposure
    "/wp-admin/",            # WordPress admin
    "/phpmyadmin/",          # Database admin
]

⚠️ Each path multiplies your scan time!

Custom Headers

HTTPX_CUSTOM_HEADERS = [
    "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0",
    "Accept-Language: en-US,en;q=0.9",
    "X-Forwarded-For: 127.0.0.1",  # Bypass some WAFs
]

Status Code Filtering

# Only successful responses
HTTPX_MATCH_CODES = ["200", "301", "302"]

# Exclude noise
HTTPX_FILTER_CODES = ["404", "403", "500", "502", "503"]

Speed Profiles

🏎️ Fast Mode (Reconnaissance)

# Minimal features, maximum speed
HTTPX_THREADS = 100
HTTPX_TIMEOUT = 5
HTTPX_RETRIES = 0
HTTPX_RATE_LIMIT = 0

HTTPX_PROBE_JARM = False           # DISABLE - very slow
HTTPX_PROBE_TLS_GRAB = False       # DISABLE - slow
HTTPX_PROBE_FAVICON = False        # DISABLE - extra request
HTTPX_PROBE_HASH = ""              # DISABLE
HTTPX_INCLUDE_RESPONSE = False     # DISABLE - large output

Expected: ~5-10 seconds for 10 URLs

⚖️ Balanced Mode (Default)

# Good balance of speed and information
HTTPX_THREADS = 50
HTTPX_TIMEOUT = 10
HTTPX_RETRIES = 1
HTTPX_RATE_LIMIT = 150

HTTPX_PROBE_JARM = False           # DISABLE for balance
HTTPX_PROBE_TLS_GRAB = True
HTTPX_PROBE_FAVICON = True
HTTPX_INCLUDE_RESPONSE = False

Expected: ~20-40 seconds for 10 URLs

🔬 Deep Analysis Mode

# Maximum information gathering
HTTPX_THREADS = 25
HTTPX_TIMEOUT = 15
HTTPX_RETRIES = 2
HTTPX_RATE_LIMIT = 50

HTTPX_PROBE_JARM = True            # Enable for threat hunting
HTTPX_PROBE_TLS_GRAB = True
HTTPX_PROBE_FAVICON = True
HTTPX_PROBE_HASH = "sha256"
HTTPX_INCLUDE_RESPONSE = True      # Full response capture

Expected: ~2-5 minutes for 10 URLs

Banner grabbing is integrated into httpx_scan.py and runs automatically after HTTP probing. It detects service versions on non-HTTP ports that httpx cannot probe.

Configuration

# Banner Grabbing Configuration (project_settings.py)

# Enable/disable banner grabbing for non-HTTP ports
BANNER_GRAB_ENABLED = True

# Connection timeout per port (seconds)
BANNER_GRAB_TIMEOUT = 5

# Number of concurrent threads
BANNER_GRAB_THREADS = 20

# Maximum banner length to store (characters)
BANNER_GRAB_MAX_LENGTH = 500

Supported Services

Service	Port(s)	Detection Method
SSH	22	Banner: `SSH-2.0-OpenSSH_8.2p1`
FTP	21	Banner: `220 vsFTPd 3.0.3`
SMTP	25, 465, 587	Banner: `220 mail.example.com ESMTP Postfix`
POP3	110, 995	Banner: `+OK Dovecot ready`
IMAP	143, 993	Banner: `* OK Dovecot IMAP`
MySQL	3306	Connection handshake
PostgreSQL	5432	Cancel request response
Redis	6379	`INFO` command response
VNC	5900	RFB protocol banner
Memcached	11211	`VERSION` command
Telnet	23	Login prompt

Skipped Ports (Handled by httpx)

HTTP ports are automatically skipped since httpx provides better detection:

80, 443, 8080, 8443, 8000, 8888, 8008, 3000, 5000, 9000, 9090, 8800

Output Structure

{
  "banner_grab": {
    "scan_metadata": {
      "scan_timestamp": "2024-01-15T10:30:00",
      "scan_duration_seconds": 5.2,
      "total_ports_scanned": 3,
      "banners_retrieved": 2
    },
    "by_host": {
      "example.com": {
        "host": "example.com",
        "ports": {
          "22": {
            "port": 22,
            "banner": "SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5",
            "service": "ssh",
            "version": "OpenSSH 8.2p1",
            "confidence": "high"
          },
          "21": {
            "port": 21,
            "banner": "220 (vsFTPd 3.0.3)",
            "service": "ftp",
            "version": "vsFTPd 3.0.3",
            "confidence": "high"
          }
        }
      }
    },
    "services_found": {
      "ssh": [{"host": "example.com", "port": 22, "version": "OpenSSH 8.2p1"}],
      "ftp": [{"host": "example.com", "port": 21, "version": "vsFTPd 3.0.3"}]
    }
  }
}

Limitations

Limitation	Description
Binary protocols	RDP, LDAP, DNS don't send text banners
Authentication required	Some services won't respond without auth
Custom services	Unknown port/protocol combinations
Firewall blocking	May timeout on filtered ports

For comprehensive service detection (1000+ services), consider using nmap with -sV flag separately.

Wappalyzer Technology Enhancement

Wappalyzer enhancement is integrated into httpx_scan.py and runs automatically after httpx parsing. It uses existing HTML from httpx (no additional HTTP requests) to detect 1000+ technologies, including CMS plugins, analytics tools, security tools, and frameworks that httpx might miss.

How It Works

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  httpx Output    │────▶│  Wappalyzer      │────▶│  Enhanced JSON   │
│  (HTML bodies,   │     │  Enhancement     │     │  with additional │
│   technologies)   │     │                  │     │  technologies    │
└──────────────────┘     │  1. Parse HTML   │     └──────────────────┘
                          │  2. Match 1000+  │
                          │     patterns     │
                          │  3. Detect       │
                          │     plugins,     │
                          │     analytics    │
                          │  4. Merge with   │
                          │     httpx techs  │
                          └──────────────────┘

Configuration

# Wappalyzer Configuration (project_settings.py)

# Enable/disable Wappalyzer technology enhancement
WAPPALYZER_ENABLED = True

# Minimum confidence level (0-100) to include a technology
# Lower = more detections but potentially more false positives
WAPPALYZER_MIN_CONFIDENCE = 50

# Analyze only URLs with HTML body (recommended: True)
# When False, attempts to analyze URLs without body (limited detection)
WAPPALYZER_REQUIRE_HTML = True

Installation

Wappalyzer requires the python-Wappalyzer library:

pip install python-Wappalyzer

Note: If Wappalyzer is not installed, the enhancement will be skipped with a warning message.

What Wappalyzer Detects

Category	Examples	Why It Matters
CMS Plugins	Yoast SEO, WooCommerce, Contact Form 7, Advanced Custom Fields	Plugin vulnerabilities, outdated plugins
Analytics Tools	Google Analytics, Facebook Pixel, Hotjar, Mixpanel	Privacy concerns, tracking detection
Security Tools	Cloudflare, Sucuri, reCAPTCHA, Wordfence	WAF detection, security posture
JavaScript Frameworks	React, Vue.js, Angular, jQuery	Framework-specific vulnerabilities
E-commerce Platforms	Shopify, WooCommerce, Magento	Payment processing, PCI compliance
CDN Providers	Cloudflare, Fastly, Akamai	CDN-specific features, bypass techniques
Database Tools	phpMyAdmin, Adminer	Database exposure risks
Version Information	WordPress 6.0, jQuery 3.6.0	Version-based CVE lookup

Comparison: httpx vs Wappalyzer

Feature	httpx Built-in	Wappalyzer Enhancement
Pattern Database	~50-100 patterns	1000+ patterns
CMS Plugins	❌ Limited	✅ Excellent
Analytics Tools	❌ Few	✅ Comprehensive
Security Tools	⚠️ Basic	✅ Detailed
Version Detection	⚠️ Basic	✅ Enhanced with confidence
Category Classification	❌ None	✅ Full categorization
Confidence Scores	❌ None	✅ Per-technology

Output Structure

Wappalyzer results are added under the wappalyzer key within the httpx section:

{
  "httpx": {
    "by_url": {
      "https://example.com": {
        "technologies": ["Nginx:1.19.0", "PHP:5.6.40", "WordPress:6.0", "jQuery:3.6.0", "Yoast SEO"]
      }
    },
    "wappalyzer": {
      "scan_metadata": {
        "scan_timestamp": "2025-12-31T12:34:56.789012",
        "urls_analyzed": 1,
        "new_technologies_found": 3,
        "plugins_detected": 1,
        "analytics_detected": 1,
        "security_tools_detected": 1
      },
      "by_url": {
        "https://example.com": [
          {
            "name": "jQuery",
            "version": "3.6.0",
            "categories": ["JavaScript frameworks"],
            "confidence": 100
          },
          {
            "name": "Yoast SEO",
            "version": "20.0",
            "categories": ["CMS"],
            "confidence": 95
          },
          {
            "name": "Google Analytics",
            "version": null,
            "categories": ["Analytics"],
            "confidence": 100
          }
        ]
      },
      "new_technologies": {
        "jQuery:3.6.0": ["https://example.com"],
        "Yoast SEO": ["https://example.com"],
        "Google Analytics": ["https://example.com"]
      },
      "all_technologies": {
        "WordPress": {
          "name": "WordPress",
          "versions_found": ["6.0"],
          "categories": ["CMS"],
          "urls": ["https://example.com"]
        },
        "jQuery": {
          "name": "jQuery",
          "versions_found": ["3.6.0"],
          "categories": ["JavaScript frameworks"],
          "urls": ["https://example.com"]
        }
      },
      "plugins": [
        {
          "name": "Yoast SEO",
          "version": "20.0",
          "url": "https://example.com"
        }
      ],
      "analytics": [
        {
          "name": "Google Analytics",
          "url": "https://example.com"
        }
      ],
      "security_tools": [
        {
          "name": "Cloudflare",
          "url": "https://example.com"
        }
      ],
      "frameworks": [
        {
          "name": "jQuery",
          "version": "3.6.0",
          "url": "https://example.com"
        }
      ],
      "summary": {
        "urls_analyzed": 1,
        "total_technologies": 5,
        "new_technologies": 3,
        "httpx_missed": ["jQuery:3.6.0", "Yoast SEO", "Google Analytics"],
        "plugins_count": 1,
        "analytics_count": 1,
        "security_tools_count": 1,
        "frameworks_count": 1
      }
    },
    "summary": {
      "technology_count": 5,
      "unique_technologies": ["Nginx:1.19.0", "PHP:5.6.40", "WordPress:6.0", "jQuery:3.6.0", "Yoast SEO"],
      "wappalyzer_additions": 3
    }
  }
}

Benefits for CVE Lookup

Wappalyzer enhancement significantly improves CVE lookup results:

More Technologies Detected: More technologies = more CVEs found
Version Information: Better version detection enables accurate CVE matching
Plugin Vulnerabilities: CMS plugins often have their own CVEs
Framework CVEs: JavaScript frameworks (jQuery, React) have known vulnerabilities

Example:

httpx only: Detects WordPress:6.0 → 5 CVEs found
httpx + Wappalyzer: Detects WordPress:6.0, jQuery:3.6.0, Yoast SEO:20.0 → 15 CVEs found

Performance Impact

URLs	Wappalyzer Time	Total Scan Time
1 URL	~1-2 seconds	+1-2 seconds
10 URLs	~5-10 seconds	+5-10 seconds
100 URLs	~30-60 seconds	+30-60 seconds

Note: Wappalyzer uses existing HTML from httpx, so there are no additional HTTP requests. The time is purely for HTML parsing and pattern matching.

Troubleshooting

"Wappalyzer not installed"

pip install python-Wappalyzer

"No new technologies found"

Possible causes:

httpx already detected all technologies
HTML body not included (HTTPX_INCLUDE_RESPONSE = False)
Technologies don't match Wappalyzer patterns
Confidence threshold too high (WAPPALYZER_MIN_CONFIDENCE)

Solutions:

# Ensure HTML is included
HTTPX_INCLUDE_RESPONSE = True

# Lower confidence threshold
WAPPALYZER_MIN_CONFIDENCE = 30

"Wappalyzer errors for some URLs"

Wappalyzer may fail on:

Malformed HTML
Non-HTML responses (JSON, XML)
Empty responses

This is normal - the enhancement continues for other URLs.

Integration with CVE Lookup

Wappalyzer-detected technologies are automatically included in CVE lookup:

{
  "technology_cves": {
    "by_technology": {
      "WordPress:6.0": {"cve_count": 5, "cves": [...]},
      "jQuery:3.6.0": {"cve_count": 3, "cves": [...]},  // From Wappalyzer
      "Yoast SEO:20.0": {"cve_count": 2, "cves": [...]}  // From Wappalyzer
    }
  }
}

Architecture & Flow

Pipeline context: HTTP probing runs in GROUP 4 of the parallelized recon pipeline, sequentially after GROUP 3 (Shodan + Port Scan). It depends on port scan results to build probe targets. While the module itself runs sequentially in the pipeline, httpx internally uses parallel threads for probing multiple targets.

Execution Flow

1. INITIALIZATION
   └── Check Docker availability
   └── Pull httpx image if needed

2. TARGET BUILDING
   └── Priority 1: Build URLs from Naabu port data
   └── Priority 2: Build URLs from DNS data (fallback)
   └── Determine HTTP/HTTPS based on port numbers

3. PROBE EXECUTION
   └── Create targets file with URLs
   └── Build httpx Docker command
   └── Execute with JSON Lines output
   └── Monitor progress

4. RESULT PROCESSING
   └── Parse JSONL output line by line
   └── Extract technologies per URL
   └── Aggregate data by host
   └── Track unique technologies and servers

5. DATA ENRICHMENT
   └── Add "httpx" section to recon_data
   └── Save incrementally to JSON file
   └── Generate summary statistics

URL Building from Naabu

# Port-based protocol detection
https_ports = {443, 8443, 4443, 9443}
http_ports = {80, 8080, 8000, 8888}

# Example transformations:
# Port 443 -> https://example.com
# Port 80 -> http://example.com
# Port 8443 -> https://example.com:8443
# Port 8080 -> http://example.com:8080
# Unknown port -> try both http:// and https://

Docker Command Structure

docker run --rm -i \
  -v /targets:/targets:ro \
  -v /output:/output \
  projectdiscovery/httpx:latest \
  -l /targets/targets.txt \
  -o /output/httpx_output.json \
  -json \
  -silent \
  -nc \
  -t 50 \                    # Threads
  -timeout 10 \
  -retries 2 \
  -rl 150 \                  # Rate limit
  -fr \                      # Follow redirects
  -maxr 10 \                 # Max redirects
  -sc \                      # Status code
  -cl \                      # Content length
  -ct \                      # Content type
  -title \                   # Page title
  -server \                  # Server header
  -rt \                      # Response time
  -wc \                      # Word count
  -lc \                      # Line count
  -td \                      # Tech detect
  -ip \                      # IP address
  -cname \                   # CNAME records
  -tls-probe \               # TLS info
  -tls-grab \                # TLS certificate
  -favicon \                 # Favicon hash
  -jarm \                    # JARM fingerprint
  -hash sha256 \             # Body hash
  -irr \                     # Include response (headers + body)
  -irh \                     # Include headers only
  -asn \                     # ASN info
  -cdn                       # CDN detection

Output Data Structure

Complete JSON Schema

{
  "httpx": {
    "scan_metadata": {
      "scan_timestamp": "2024-01-15T12:00:00.000000",
      "scan_duration_seconds": 120.5,
      "docker_image": "projectdiscovery/httpx:latest",
      "threads": 50,
      "timeout": 10,
      "rate_limit": 150,
      "follow_redirects": true,
      "tech_detection": true,
      "tls_probing": true,
      "response_included": true,
      "proxy_used": false,
      "total_urls_probed": 150
    },
    
    "by_url": {
      "https://example.com": {
        "url": "https://example.com",
        "host": "example.com",
        "status_code": 200,
        "content_length": 45678,
        "content_type": "text/html; charset=UTF-8",
        "title": "Example Domain",
        "server": "nginx/1.18.0",
        "response_time_ms": 245,
        "word_count": 1234,
        "line_count": 89,
        "technologies": ["nginx", "PHP", "WordPress"],
        "ip": "93.184.216.34",
        "cname": null,
        "cdn": null,
        "is_cdn": false,
        "asn": {
          "as_number": "AS15133",
          "as_name": "Edgecast Inc.",
          "as_country": "US"
        },
        "tls": {
          "version": "TLS 1.3",
          "cipher": "TLS_AES_256_GCM_SHA384",
          "certificate": {
            "subject_cn": "example.com",
            "issuer": "Let's Encrypt Authority X3",
            "not_before": "2024-01-01T00:00:00Z",
            "not_after": "2024-04-01T00:00:00Z",
            "san": ["example.com", "www.example.com"]
          }
        },
        "favicon_hash": "116323821",
        "jarm": "29d29d15d29d29d00029d29d29d29d...",
        "body_hash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        "headers": {
          "Content-Type": "text/html; charset=UTF-8",
          "X-Powered-By": "PHP/7.4.3"
        },
        "body": "<!DOCTYPE html>..."
      }
    },
    
    "by_host": {
      "example.com": {
        "hostname": "example.com",
        "urls": ["http://example.com", "https://example.com"],
        "live_urls": ["https://example.com"],
        "technologies": ["nginx", "PHP", "WordPress"],
        "servers": ["nginx/1.18.0"],
        "status_codes": [200, 301]
      }
    },
    
    "technologies_found": {
      "nginx": ["https://example.com", "https://api.example.com"],
      "PHP": ["https://example.com"],
      "WordPress": ["https://example.com"],
      "React": ["https://app.example.com"]
    },
    
    "servers_found": {
      "nginx/1.18.0": ["https://example.com"],
      "Apache/2.4.41": ["https://legacy.example.com"]
    },
    
    "summary": {
      "total_urls_probed": 150,
      "live_urls": 45,
      "total_hosts": 25,
      "by_status_code": {
        "200": 35,
        "301": 8,
        "404": 2
      },
      "unique_technologies": ["nginx", "PHP", "WordPress", "React"],
      "technology_count": 4,
      "unique_servers": ["nginx/1.18.0", "Apache/2.4.41"],
      "server_count": 2,
      "cdn_hosts": 3
    }
  }
}

Usage Examples

Basic Usage (via main.py)

# Include "http_probe" in SCAN_MODULES in project settings
SCAN_MODULES = ["initial_recon", "naabu", "httpx", "nuclei"]

# Run the full pipeline
python3 recon/main.py

Standalone Enrichment

from httpx_scan import enrich_recon_file
from pathlib import Path

enriched = enrich_recon_file(Path("output/recon_example.com.json"))

Command Line

# Enrich an existing recon file
python3 recon/httpx_scan.py output/recon_example.com.json

Configuration Profiles

Quick Probe (Fast):

HTTPX_THREADS = 100
HTTPX_RATE_LIMIT = 300
HTTPX_TIMEOUT = 5
HTTPX_PROBE_TECH_DETECT = True
HTTPX_INCLUDE_RESPONSE = False  # Smaller output

Comprehensive Analysis:

HTTPX_THREADS = 50
HTTPX_RATE_LIMIT = 100
HTTPX_TIMEOUT = 15
HTTPX_PROBE_TECH_DETECT = True
HTTPX_PROBE_TLS_INFO = True
HTTPX_PROBE_TLS_GRAB = True
HTTPX_PROBE_FAVICON = True
HTTPX_PROBE_JARM = True
HTTPX_INCLUDE_RESPONSE = True
HTTPX_INCLUDE_RESPONSE_HEADERS = True

Stealth Probe:

HTTPX_THREADS = 10
HTTPX_RATE_LIMIT = 20
HTTPX_TIMEOUT = 20
HTTPX_CUSTOM_HEADERS = [
    "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
]

Technology-Focused:

HTTPX_PROBE_TECH_DETECT = True
HTTPX_PROBE_SERVER = True
HTTPX_PROBE_FAVICON = True
HTTPX_PROBE_HASH = "sha256"
HTTPX_INCLUDE_RESPONSE_HEADERS = True

# Install Docker
sudo apt install docker.io

# Start Docker daemon
sudo systemctl start docker

"No live URLs found"

Possible causes:

Hosts are down
Firewall blocking requests
Wrong ports probed

Solutions:

# Increase timeout
HTTPX_TIMEOUT = 20

# Increase retries
HTTPX_RETRIES = 3

# Try without Naabu (probe default ports)
SCAN_MODULES = ["initial_recon", "httpx"]

Scan too slow

# Increase performance
HTTPX_THREADS = 100
HTTPX_RATE_LIMIT = 300
HTTPX_TIMEOUT = 5
HTTPX_RETRIES = 1

# Disable expensive probes
HTTPX_PROBE_JARM = False
HTTPX_INCLUDE_RESPONSE = False

Too much output data

# Disable response body inclusion
HTTPX_INCLUDE_RESPONSE = False
HTTPX_INCLUDE_RESPONSE_HEADERS = False

# Filter status codes
HTTPX_MATCH_CODES = ["200"]

Debug Mode

Run httpx manually via Docker:

docker run --rm \
  projectdiscovery/httpx:latest \
  -u https://example.com \
  -sc -td -server -title \
  -v -debug

Security Considerations

⚠️ Legal Warning: Only scan systems you have explicit permission to test.

Risk	Mitigation
Rate limiting/bans	Reduce `HTTPX_RATE_LIMIT`
WAF blocking	Use custom User-Agent, reduce rate
Detection	Reduce threads
Fingerprinting	Your requests are identifiable

Safe Defaults

HTTPX_THREADS = 25
HTTPX_RATE_LIMIT = 50
HTTPX_TIMEOUT = 15
HTTPX_CUSTOM_HEADERS = [
    "User-Agent: Mozilla/5.0 (compatible; SecurityScanner/1.0)"
]

Dependencies

Package	Purpose
Docker	Container runtime for httpx
`projectdiscovery/httpx:latest`	httpx Docker image (auto-pulled)
Python 3.8+	Script runtime

References

Documentation generated for RedAmon v1.0 - httpx HTTP Prober Module