Crawler

The crawler fetches Wallet Relying Party data from external sources, normalizes it, and stores it in the database. It supports crash recovery via checkpoints, content-hash-based change detection, and soft-delete tracking for entities that disappear from sources.

Architecture

Data Flow

Source → Orchestrator → DataRepository → PostgreSQL

Components

Orchestrator (internal/crawler/orchestrator.go) coordinates the entire crawl:

• Manages a crawl run lifecycle (create run → process sources → complete/interrupt)
• Iterates sources, calling FetchBatch in a loop until io.EOF
• Saves checkpoint state every N batches (configurable)
• Handles graceful shutdown on context cancellation
• Runs sources with configurable concurrency

SourceCrawler interface (internal/crawler/repositories/interfaces.go):

type SourceCrawler interface {
    Name() string
    Initialize(ctx context.Context, state *CrawlerState) error
    FetchBatch(ctx context.Context, batchSize int) (*FetchResult, error)
    GetState() CrawlerState
    Close() error
}

Each source handles its own pagination and rate limiting internally.

CrawlerFactory (internal/crawler/factory.go) creates source instances from config:

switch cfg.Type {
case "generic-json":
    return sources.NewGenericJSONCrawler(name, cfg, httpClient), nil
}

Adding a new source type means implementing SourceCrawler and adding a case to the factory.

DataRepository (internal/crawler/repositories/data_repository.go) handles upserts with change detection:

• Computes content hashes to detect actual changes vs. unchanged data
• Returns ChangeType (none, inserted, updated) for each upsert
• Supports soft-delete tracking via MarkDeletedSince

Technology Choices

Technology	Why
Same pgx/DB stack as API	Shared persistence layer, single migration set
Content hashing	Avoid unnecessary writes — only update when data actually changes
Checkpoint-based state	Resume after crash without re-processing already-handled batches
Context-based cancellation	Clean shutdown via Go's context.Context and signal handling

Running the Crawler

Docker (Recommended)

# 1. Generate mock data
go run ./cmd/generate-mock -count 100

# 2. Start database and mock HTTP server
docker compose up -d db mock-server

# 3. Run the crawler (one-shot)
docker compose run --rm crawler

# Or start all services including crawler
docker compose --profile crawler up

Local Development

# 1. Generate mock data
go run ./cmd/generate-mock -count 100

# 2. Start the database
docker compose up -d db

# 3. Serve mock data locally
cd data && python3 -m http.server 8081

# 4. Update config.yaml source URL to localhost:
#    url: "http://localhost:8081/mock-crawler-data.json"

# 5. Run the crawler
WIMAPI_DATABASE_HOST=localhost go run ./cmd/crawler

See tools.md for details on generating mock data.

Configuration

Crawler settings in config.yaml:

crawler:
  orchestrator:
    batch_size: 100              # Records processed per batch
    checkpoint_interval: 10      # Save state every N batches
    debug_sample_rate: 0.01      # Fraction of responses stored for debugging (1%)
    max_retries: 3               # Retries per failed batch
    retry_backoff_base: 1s       # Base delay between retries (exponential)
    concurrency: 2               # Parallel source crawlers
    run_timeout: 4h              # Maximum total run duration

  http_client:
    timeout: 30s                 # HTTP request timeout
    max_idle_conns: 10           # Connection pool size

  sources:
    mock-api:
      enabled: true
      type: "generic-json"
      url: "http://mock-server/data/mock-crawler-data.json"
      rate_limit: 1000.0         # Requests per second (simulated)

All settings can be overridden via environment variables with the WIMAPI_ prefix (e.g., WIMAPI_CRAWLER_ORCHESTRATOR_BATCH_SIZE=50).

Source Types

generic-json

Fetches a single JSON file containing all relying party data. Useful for static data sources or testing.

Rate limiting simulates realistic crawl timing: each record in a batch counts as one "request", so with rate_limit: 10 and batch_size: 100, the crawler waits 10 seconds per batch.

Adding a New Source Type

1. Create a new file in internal/crawler/sources/ implementing SourceCrawler
2. Handle pagination, rate limiting, and error recovery internally
3. Register the type in internal/crawler/factory.go → Create() switch statement
4. Add configuration under crawler.sources in config.yaml

State Management

The crawler maintains three tables for crash recovery and change tracking:

Table	Purpose
crawler_run	Tracks each execution: start time, status (running/completed/interrupted/failed), total processed
crawler_state	Per-source checkpoint: cursor, offset, counts. Linked to a run
crawler_content_hash	Content hashes per entity for change detection and soft-delete tracking

Crash Recovery

When the crawler starts, it checks for incomplete runs (status = running or interrupted). If found, it resumes from the last saved checkpoint for each source rather than starting from scratch.

The checkpoint interval controls how much work can be lost on crash: with checkpoint_interval: 10 and batch_size: 100, at most 1000 records may need re-processing.

To force a fresh crawl, either:

• Let the previous run complete normally
• Manually update the run status in the database

Graceful Shutdown

Sending SIGINT (Ctrl+C) or SIGTERM triggers graceful shutdown:

1. The context is cancelled
2. The current batch finishes processing
3. State is saved for all active sources (using a fresh context.Background() with timeout to avoid the cancelled context)
4. The run is marked as interrupted (allows resume on next start)

Important: go run wraps the binary in a subprocess. Signals sent to the go run process kill the wrapper, not the actual binary. For reliable signal testing, build first:

go build -o /tmp/crawler ./cmd/crawler
/tmp/crawler
# Now Ctrl+C or kill -TERM works correctly