Dealbot Architecture

System Architecture

graph TD
    User["Operator / SP / Reviewer"] --> WebUI["Web UI"]
    WebUI --> Caddy["Caddy Wrapper (static hosting + reverse proxy)"]
    Caddy --> ApiControllers["Backend API Controllers (NestJS)"]
    ApiControllers --> ApiServices["Backend Services"]
    ApiServices --> Addons["Deal/Retrieval Add-ons"]
    ApiServices --> Db["Postgres"]
    ApiServices --> Wallet["Filecoin RPC / Contracts"]
    Jobs["Jobs Module (pg-boss scheduler + workers)"] --> ApiServices
    Jobs --> Db

Component Responsibilities

• Web UI: React/Vite dashboard served by a Caddy wrapper (static hosting plus /api reverse proxy).
• Backend API: controllers and services for endpoints and business logic.
• Deal add-ons and Retrieval add-ons: deal/retrieval check integrations.
• Job execution: pg-boss scheduler + workers for deal/retrieval jobs.
• Wallet + chain integration: provider discovery and on-chain operations.
• Persistence: deal/retrieval state plus pg-boss queue/schedule state in Postgres.
• Metrics: Prometheus instrumentation. Metric semantics live in docs/checks/events-and-metrics.md; observability wiring lives in infra.md.

Data Stores and Ownership

Postgres is the system-of-record for Dealbot state:

• SPs under test and provider metadata in storage_providers.
• Deal lifecycle records in deals, including managed dataset/piece identifiers (data_set_id, piece_id, piece_cid).
• Retrieval lifecycle records in retrievals.
• Scheduler state in job_schedule_state and queue execution state in pgboss.job.

ClickHouse is an optional append-only sink for long-term check result analysis. Each network with a <NET>_CLICKHOUSE_URL has its own database, client, and buffer; the selected database identifies the network of its rows. The global batching settings apply independently to every network buffer, so a failed destination does not block another network. Failed flushes are retried, and rows are dropped only when that network reaches CLICKHOUSE_MAX_BUFFER_SIZE. The service starts and runs without ClickHouse. Table layout is documented in docs/checks/events-and-metrics.md; operator wiring is described in infra.md.

Prometheus metrics are runtime observability, not durable state. Job health and per-check timing metrics are emitted at runtime; metric semantics live in docs/checks/events-and-metrics.md. External monitoring (Grafana, BetterStack, or other) is an observability surface, not canonical state.