Deal Bot Backend

July 2, 2026 · View on GitHub

NestJS-based API server for automated Filecoin deal creation and monitoring

This is the backend service for Deal Bot, built with NestJS and TypeScript. It handles automated deal creation, retrieval testing, metrics collection, and provides a REST API for the web dashboard.

Tech Stack

Framework: NestJS 11.x
Language: TypeScript 5.x
Database: PostgreSQL + TypeORM
Blockchain: Filecoin (via Synapse SDK)
Scheduling: NestJS Schedule (cron jobs)
API Docs: Swagger/OpenAPI

Prerequisites

Node.js 20+
pnpm
PostgreSQL database (running and accessible)
Filecoin wallet with private key

Quick Start

1. Install Dependencies

pnpm install

2. Configure Environment

cp .env.example .env

Edit .env with your configuration (see Configuration below).

3. Set Up Database

Ensure PostgreSQL is running and create a database:

CREATE DATABASE filecoin_dealbot;
CREATE USER dealbot WITH PASSWORD 'dealbot_password';
GRANT ALL PRIVILEGES ON DATABASE filecoin_dealbot TO dealbot;

The application will automatically run migrations on startup.

4. Run the Server

Development Mode (with hot-reload)

pnpm start:dev

Server runs at: http://localhost:8080 API Documentation: http://localhost:8080/api

Production Mode

# Build the application
pnpm build

# Run the built application
pnpm start:prod

Configuration

All configuration is done via environment variables in .env.

📖 For detailed documentation on all environment variables, see docs/environment-variables.md

Database Configuration

Variable	Description	Example
`DATABASE_HOST`	PostgreSQL host	`localhost`
`DATABASE_PORT`	PostgreSQL port	`5432`
`DATABASE_USER`	Database user	`dealbot`
`DATABASE_PASSWORD`	Database password	`dealbot_password`
`DATABASE_NAME`	Database name	`filecoin_dealbot`

Server Configuration

Variable	Description	Default
`NODE_ENV`	Environment mode	`development`
`DEALBOT_PORT`	Server port	`8080`
`DEALBOT_HOST`	Server host	`localhost`
`DEALBOT_ALLOWED_ORIGINS`	CORS allowed origins (comma-separated)	`http://localhost:5173,http://127.0.0.1:5173`

Blockchain Configuration

Variable	Description	Example
`NETWORK`	Filecoin network	`calibration` or `mainnet`
`WALLET_ADDRESS`	Your Filecoin wallet address	`0x...`
`WALLET_PRIVATE_KEY`	Your wallet private key (keep secure!)	`0x...`
`CHECK_DATASET_CREATION_FEES`	Check fees before dataset creation	`true`
`ENABLE_IPNI_TESTING`	IPNI testing mode (`disabled`/`random`/`always`)	`always`
`USE_ONLY_APPROVED_PROVIDERS`	Only use approved storage providers	`true`
`PDP_SUBGRAPH_ENDPOINT`	PDP subgraph API endpoint for PDP proof-set/data-retention	`https://api.thegraph.com/subgraphs/filecoin/pdp`
`SUBGRAPH_ENDPOINT`	Subgraph GraphQL endpoint for sampled-retrieval queries	`https://api.goldsky.com/api/public/<project>/subgraphs/dealbot-subgraph/<version>/gn`

Scheduling Configuration (pg-boss)

Dealbot uses pg-boss for all job scheduling. See docs/jobs.md for scheduling behavior and docs/environment-variables.md for defaults and full definitions.

Variable	Description	Recommended
`PROVIDERS_REFRESH_INTERVAL_SECONDS`	Providers refresh interval (seconds)	`14400` (4 hours)
`DATA_RETENTION_POLL_INTERVAL_SECONDS`	Data retention polling interval (seconds)	`3600` (1 hour)
`DEALS_PER_SP_PER_HOUR`	Deal checks per SP per hour	`4`
`RETRIEVALS_PER_SP_PER_HOUR`	Retrieval checks per SP per hour	`2`
`DATASET_CREATIONS_PER_SP_PER_HOUR`	Dataset creation checks per SP per hour	`1`
`PG_BOSS_LOCAL_CONCURRENCY`	Per-process `sp.work` concurrency	`20`
`JOB_SCHEDULER_POLL_SECONDS`	Scheduler poll interval	`300`
`JOB_WORKER_POLL_SECONDS`	Worker poll interval	`60`
`JOB_CATCHUP_MAX_ENQUEUE`	Max catch-up enqueues per schedule per tick	`10`
`JOB_SCHEDULE_PHASE_SECONDS`	Phase offset for multi-deploy staggering	`0`
`DEALBOT_PGBOSS_POOL_MAX`	Max pg-boss DB connections per instance	`1`
`DEALBOT_PGBOSS_SCHEDULER_ENABLED`	Enable the enqueue loop	`true` (api/both), `false` (worker)
`DEALBOT_RUN_MODE`	Run mode for the application	`both` (or split api/worker)

Note: If you run multiple deployments in the same environment, use a non-zero JOB_SCHEDULE_PHASE_SECONDS to stagger schedules.

Dataset Configuration

Variable	Description	Default
`DEALBOT_LOCAL_DATASETS_PATH`	Local path for random dataset storage	`./datasets`
`RANDOM_PIECE_SIZES`	Comma-separated byte sizes for uploaded content	`10485760`

Project Structure

backend/
├── src/
│   ├── main.ts                 # Application entry point
│   ├── app.module.ts           # Root module
│   ├── app.controller.ts       # Root controller
│   ├── config/                 # Configuration modules
│   ├── database/               # Database entities and migrations
│   ├── deal/                   # Deal creation logic
│   ├── deal-addons/            # Deal add-ons and extensions
│   ├── retrieval/              # Retrieval testing logic
│   ├── retrieval-addons/       # Retrieval add-ons (IPNI)
│   ├── metrics/                # Metrics collection and analytics
│   ├── scheduler/              # Cron job scheduling
│   ├── wallet-sdk/             # Wallet and smart contract operations
│   ├── http-client/            # HTTP client utilities
│   └── common/                 # Shared utilities and decorators
├── test/                       # E2E tests
├── dist/                       # Compiled output (after build)
└── README.md                   # This file

API Documentation

Interactive API documentation is available via Swagger UI:

Local: http://localhost:8080/api
Production: https://dealbot.filoz.org/api

Development

Available Scripts

pnpm start          # Start in normal mode
pnpm start:dev      # Start with hot-reload (recommended for development)
pnpm start:debug    # Start with debugger
pnpm build          # Build for production
pnpm start:prod     # Run production build

Code Quality

pnpm format         # Format code with Biome
pnpm format:check   # Check formatting
pnpm lint           # Lint and auto-fix
pnpm lint:check     # Check linting
pnpm check          # Run both format and lint
pnpm check:ci       # CI checks (no auto-fix)

Testing

pnpm test           # Run unit tests
pnpm test:watch     # Run tests in watch mode
pnpm test:cov       # Run tests with coverage
pnpm test:e2e       # Run end-to-end tests

Database Schema

The application uses TypeORM with PostgreSQL. Key entities:

Deal - Storage deal records
Retrieval - Retrieval test records
Provider - Storage provider information
DailyMetric - Daily performance metrics

Migrations run automatically on application startup.

Troubleshooting

Database Connection Issues

# Check PostgreSQL is running
psql -U dealbot -d filecoin_dealbot

# Verify DATABASE_* environment variables in .env

Port Already in Use

# Change DEALBOT_PORT in .env
# Also update VITE_API_BASE_URL in web/.env

Wallet/Blockchain Issues

# Verify WALLET_ADDRESS and WALLET_PRIVATE_KEY are correct
# Ensure wallet has sufficient USDFC tokens on the specified NETWORK

CORS Errors from Frontend

# Add frontend URL to DEALBOT_ALLOWED_ORIGINS in .env
# Example: DEALBOT_ALLOWED_ORIGINS=http://localhost:5173,http://127.0.0.1:5173

Contributing

See the main README for contribution guidelines.

Resources

License

Dual-licensed: MIT, Apache Software License v2 by way of the Permissive License Stack.