Krawl Architecture

June 13, 2026 · View on GitHub

Overview

Krawl is a cloud-native deception honeypot server built on FastAPI. It creates realistic fake web applications (admin panels, login pages, fake credentials) to attract, detect, and analyze malicious crawlers and attackers while wasting their resources with infinite spider-trap pages.

Tech Stack

Layer	Technology
Backend	FastAPI, Uvicorn, Python 3.11
ORM / DB	SQLAlchemy 2.0, SQLite (WAL mode) or PostgreSQL
Templating	Jinja2 (server-side rendering)
Reactivity	Alpine.js 3.14
Partial Updates	HTMX 2.0
Charts	Chart.js 3.9 (doughnut), custom SVG radar
Maps	Leaflet 1.9 + CartoDB dark tiles
Scheduling	APScheduler
Container	Docker (python:3.11-slim), Helm/K8s ready

Directory Structure

Krawl/
├── src/
│   ├── app.py                    # FastAPI app factory + lifespan
│   ├── config.py                 # YAML + env config loader
│   ├── dependencies.py           # DI providers (templates, DB, client IP)
│   ├── database.py               # DatabaseManager singleton
│   ├── models.py                 # SQLAlchemy ORM models
│   ├── tracker.py                # In-memory + DB access tracking
│   ├── logger.py                 # Rotating file log handlers
│   ├── deception_responses.py    # Attack detection + fake responses
│   ├── sanitizer.py              # Input sanitization
│   ├── generators.py             # Random content generators
│   ├── wordlists.py              # JSON wordlist loader
│   ├── geo_utils.py              # IP geolocation API
│   ├── ip_utils.py               # IP validation
│   │
│   ├── routes/
│   │   ├── honeypot.py           # Trap pages, credential capture, catch-all
│   │   ├── dashboard.py          # Dashboard page (Jinja2 SSR)
│   │   ├── api.py                # JSON API endpoints
│   │   └── htmx.py               # HTMX HTML fragment endpoints
│   │
│   ├── middleware/
│   │   ├── deception.py          # Path traversal / XXE / cmd injection detection
│   │   └── ban_check.py          # Banned IP enforcement
│   │
│   ├── tasks/                    # APScheduler background jobs
│   │   ├── analyze_ips.py        # IP categorization scoring
│   │   ├── fetch_ip_rep.py       # Geolocation + blocklist enrichment
│   │   ├── db_dump.py            # Database export
│   │   ├── memory_cleanup.py     # In-memory list trimming
│   │   └── db_retention.py       # Data retention cleanup
│   │
│   ├── tasks_master.py           # Task discovery + APScheduler orchestrator
│   ├── firewall/                 # Banlist export (iptables, raw)
│   ├── migrations/               # Schema migrations (auto-run)
│   │
│   └── templates/
│       ├── jinja2/
│       │   ├── base.html                     # Layout + CDN scripts
│       │   └── dashboard/
│       │       ├── index.html                # Main dashboard page
│       │       └── partials/                 # 13 HTMX fragment templates
│       ├── html/                             # Deceptive trap page templates
│       └── static/
│           ├── css/dashboard.css
│           └── js/
│               ├── dashboard.js              # Alpine.js app controller
│               ├── map.js                    # Leaflet map
│               ├── charts.js                 # Chart.js doughnut
│               └── radar.js                  # SVG radar chart
│
├── config.yaml               # Application configuration
├── wordlists.json             # Attack patterns + fake credentials
├── Dockerfile                 # Container build
├── docker-compose.yaml        # Local orchestration
├── entrypoint.sh              # Container startup (gosu privilege drop)
├── kubernetes/                # K8s manifests
└── helm/                      # Helm chart

Application Entry Point

src/app.py uses the FastAPI application factory pattern with an async lifespan manager:

Startup                              Shutdown
  │                                    │
  ├─ Initialize logging                └─ Log shutdown
  ├─ Initialize database (SQLite or PostgreSQL)
  ├─ Initialize cache (in-memory or Redis)
  ├─ Resolve server public IP (once)
  ├─ Create AccessTracker
  ├─ Store config + tracker in app.state
  ├─ Start APScheduler background tasks
  └─ Log dashboard URL

Request Pipeline

        Request
          │
          ▼
┌──────────────────────┐
│  BanCheckMiddleware  │──→ IP banned? → Return 500
└──────────┬───────────┘
           ▼
┌──────────────────────┐
│ DeceptionMiddleware  │──→ Attack detected? → Fake error response
└──────────┬───────────┘
           ▼
┌───────────────────────┐
│ ServerHeaderMiddleware│──→ Add random Server header
└──────────┬────────────┘
           ▼
┌───────────────────────┐
│     Route Matching    │
│  (ordered by priority)│
│                       │
│  1. Static files      │  /{secret}/static/*
│  2. Dashboard router  │  /{secret}/          (prefix-based)
│  3. API router        │  /{secret}/api/*     (prefix-based)
│  4. HTMX router       │  /{secret}/htmx/*   (prefix-based)
│  5. Honeypot router   │  /* (catch-all)
└───────────────────────┘

Prefix-Based Routing

Dashboard, API, and HTMX routers are mounted with prefix=f"/{secret}" in app.py. This means:

Route handlers define paths without the secret (e.g., @router.get("/api/all-ips"))
FastAPI prepends the secret automatically (e.g., GET /a1b2c3/api/all-ips)
The honeypot catch-all /{path:path} only matches paths that don't start with the secret
No _is_dashboard_path() checks needed — the prefix handles access scoping

Route Architecture

Honeypot Routes (`routes/honeypot.py`)

Method	Path	Response
`GET`	`/{path:path}`	Trap page with random links (catch-all)
`HEAD`	`/{path:path}`	200 OK
`POST`	`/{path:path}`	Credential capture
`GET`	`/admin`, `/login`	Fake login form
`GET`	`/wp-admin`, `/wp-login.php`	Fake WordPress login
`GET`	`/phpmyadmin`	Fake phpMyAdmin
`GET`	`/robots.txt`	Honeypot paths advertised
`GET/POST`	`/api/search`, `/api/sql`	SQL injection honeypot
`POST`	`/api/contact`	XSS detection endpoint
`GET`	`/.env`, `/credentials.txt`	Fake sensitive files

Dashboard Routes (`routes/dashboard.py`)

Method	Path	Response
`GET`	`/`	Server-rendered dashboard (Jinja2)

API Routes (`routes/api.py`)

Method	Path	Response
`GET`	`/api/all-ips`	Paginated IP list with stats
`GET`	`/api/attackers`	Paginated attacker IPs
`GET`	`/api/ip-stats/{ip}`	Single IP detail
`GET`	`/api/credentials`	Captured credentials
`GET`	`/api/honeypot`	Honeypot trigger counts
`GET`	`/api/top-ips`	Top requesting IPs
`GET`	`/api/top-paths`	Most requested paths
`GET`	`/api/top-user-agents`	Top user agents
`GET`	`/api/attack-types-stats`	Attack type distribution
`GET`	`/api/attack-types`	Paginated attack log
`GET`	`/api/raw-request/{id}`	Full HTTP request
`GET`	`/api/export-ips`	Export IPs for firewall integration

HTMX Fragment Routes (`routes/htmx.py`)

Each returns a server-rendered Jinja2 partial (hx-swap="innerHTML"):

Path	Template
`/htmx/honeypot`	`honeypot_table.html`
`/htmx/top-ips`	`top_ips_table.html`
`/htmx/top-paths`	`top_paths_table.html`
`/htmx/top-ua`	`top_ua_table.html`
`/htmx/attackers`	`attackers_table.html`
`/htmx/credentials`	`credentials_table.html`
`/htmx/attacks`	`attack_types_table.html`
`/htmx/patterns`	`patterns_table.html`
`/htmx/ip-detail/{ip}`	`ip_detail.html`

Database Schema

┌─────────────────┐     ┌──────────────────┐
│   AccessLog     │     │ AttackDetection   │
├─────────────────┤     ├──────────────────┤
│ id (PK)         │◄────│ access_log_id(FK)│
│ ip (indexed)    │     │ attack_type      │
│ path            │     │ matched_pattern  │
│ user_agent      │     └──────────────────┘
│ method          │
│ is_suspicious   │     ┌──────────────────┐
│ is_honeypot     │     │CredentialAttempt │
│ timestamp       │     ├──────────────────┤
│ raw_request     │     │ id (PK)          │
└─────────────────┘     │ ip (indexed)     │
                        │ path, username   │
┌─────────────────┐     │ password         │
│    IpStats      │     │ timestamp        │
├─────────────────┤     └──────────────────┘
│ ip (PK)         │
│ total_requests  │     ┌──────────────────┐
│ first/last_seen │     │ CategoryHistory  │
│ country_code    │     ├──────────────────┤
│ city, lat, lon  │     │ id (PK)          │
│ asn, asn_org    │     │ ip (indexed)     │
│ isp, reverse    │     │ old_category     │
│ is_proxy        │     │ new_category     │
│ is_hosting      │     │ timestamp        │
│ list_on (JSON)  │     └──────────────────┘
│ category        │
│ category_scores │
│ analyzed_metrics│
│ manual_category │
└─────────────────┘

SQLite config (standalone): WAL mode, 30s busy timeout, file permissions 600.

PostgreSQL config (scalable): Connection pool (10 + 20 overflow), pre-ping enabled, connections recycled every 30 minutes.

Frontend Architecture

base.html
  ├── CDN: Leaflet, Chart.js, HTMX, Alpine.js (deferred)
  ├── Static: dashboard.css
  │
  └── dashboard/index.html (extends base)
      │
      ├── Stats cards ──────────── Server-rendered on page load
      ├── Suspicious table ─────── Server-rendered on page load
      │
      ├── Overview tab (Alpine.js x-show)
      │   ├── Honeypot table ───── HTMX hx-get on load
      │   ├── Top IPs table ────── HTMX hx-get on load
      │   ├── Top Paths table ──── HTMX hx-get on load
      │   ├── Top UA table ─────── HTMX hx-get on load
      │   └── Credentials table ── HTMX hx-get on load
      │
      └── Attacks tab (Alpine.js x-show, lazy init)
          ├── Attackers table ──── HTMX hx-get on load
          ├── Map ──────────────── Leaflet (init on tab switch)
          ├── Chart ────────────── Chart.js (init on tab switch)
          ├── Attack types table ─ HTMX hx-get on load
          └── Patterns table ───── HTMX hx-get on load

Responsibility split:

Alpine.js — Tab state, modals, dropdowns, lazy initialization
HTMX — Table pagination, sorting, IP detail expansion
Leaflet — Interactive map with category-colored markers
Chart.js — Doughnut chart for attack type distribution
Custom SVG — Radar charts for IP category scores

Background Tasks

Managed by TasksMaster (APScheduler). Tasks are auto-discovered from src/tasks/.

Task	Schedule	Purpose
`analyze_ips`	Every 1 min	Score IPs into categories (attacker, crawler, user)
`fetch_ip_rep`	Every 5 min	Enrich IPs with geolocation + blocklist data
`dashboard_warmup`	Every 5 min	Pre-compute dashboard overview data (optional, disable via `cache_warmup: false`)
`db_dump`	Configurable	Export database backups
`memory_cleanup`	Periodic	Trim in-memory lists
`db_retention`	Daily (3 AM)	Clean up old records based on retention policy

IP Categorization Model

Each IP is scored across 4 categories based on:

HTTP method distribution (risky methods ratio)
Robots.txt violations
Request timing anomalies (coefficient of variation)
User-Agent diversity
Attack URL detection

Categories: attacker, bad_crawler, good_crawler, regular_user, unknown

Configuration

config.yaml with environment variable overrides (KRAWL_{FIELD}):

mode: standalone                # "standalone" or "scalable"

server:
  port: 5000
  delay: 100                    # Response delay (ms)

dashboard:
  secret_path: "test"           # Auto-generates if null
  cache_warmup: true            # Background cache refresh (optional in scalable mode)

database:
  path: "data/krawl.db"
  retention_days: 30

# Scalable mode only
postgres:
  host: "localhost"
  port: 5432

redis:
  host: "localhost"
  port: 6379
  cache_ttl: 600                # Dashboard warmup data TTL
  hot_ttl: 30                   # Ban info / IP category TTL
  table_ttl: 120                # Paginated table TTL

crawl:
  infinite_pages_for_malicious: true
  max_pages_limit: 250
  ban_duration_seconds: 600

behavior:
  probability_error_codes: 0    # 0-100%

canary:
  token_url: null               # External canary alert URL

Logging

Three rotating log files (1MB max, 5 backups each):

Logger	File	Content
`krawl.app`	`logs/krawl.log`	Application events, errors
`krawl.access`	`logs/access.log`	HTTP access, attack detections
`krawl.credentials`	`logs/credentials.log`	Captured login attempts

Docker

FROM python:3.11-slim
# Non-root user: krawl:1000
# Volumes: /app/logs, /app/data, /app/exports
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "5000", "--app-dir", "src", "--no-server-header"]

Key Data Flows

Honeypot Request

Client → BanCheck → DeceptionMiddleware → HoneypotRouter
                                              │
                                    ┌─────────┴──────────┐
                                    │ tracker.record()    │
                                    │   ├─ in-memory ++   │
                                    │   ├─ detect attacks │
                                    │   └─ DB persist     │
                                    └────────────────────┘

Dashboard Load

Browser → GET /{secret}/ → SSR initial stats + Jinja2 render
       → Alpine.js init → HTMX fires hx-get for each table
       → User clicks Attacks tab → setTimeout → init Leaflet + Chart.js
       → Leaflet fetches /api/all-ips → plots markers
       → Chart.js fetches /api/attack-types-stats → renders doughnut

IP Enrichment Pipeline

APScheduler (every 5 min)
  └─ fetch_ip_rep.main()
       ├─ DB: get unenriched IPs (limit 50)
       ├─ ip-api.com → geolocation (country, city, ASN, coords)
       ├─ iprep.lcrawl.com → blocklist memberships
       └─ DB: update IpStats with enriched data