Development Guide

March 19, 2026 · View on GitHub

Dev Environment Setup

API Server

Three options, from most isolated to simplest:

Option 1: Docker with Air (Recommended)

Consistent environment, no local Go required for running:

# Start database + API with hot reloading
docker compose up --build app

Code changes are detected automatically and the app rebuilds inside the container.

Option 2: Local with Air

Run natively with hot reloading:

# Install Air
go install github.com/air-verse/air@latest

# Start the database
docker compose up -d db

# Run with Air
export WIMAPI_DATABASE_HOST=localhost
air

Option 3: Direct Execution

No hot reloading, simplest setup:

docker compose up -d db
export WIMAPI_DATABASE_HOST=localhost
go run cmd/api/main.go

Crawler

The crawler requires mock data and a mock HTTP server to serve it:

Docker (recommended):

# Generate mock data, then run everything
go run ./cmd/generate-mock -count 100
docker compose up -d db mock-server
docker compose run --rm crawler

Local:

# Generate mock data
go run ./cmd/generate-mock -count 100

# Start DB, serve mock data locally
docker compose up -d db
cd data && python3 -m http.server 8081 &

# Update config.yaml source URL to: http://localhost:8081/mock-crawler-data.json
WIMAPI_DATABASE_HOST=localhost go run ./cmd/crawler

See tools.md for details on the mock data generator and seed tool.

IPFS Publisher

The publisher requires Storacha credentials (ipfs.key, proof.ucan) and a configured space DID.

Docker:

# Run as a one-shot command
docker compose run --rm ipfs-publisher

# Or start with a profile
docker compose --profile ipfs-publisher up

Local:

docker compose up -d db
WIMAPI_DATABASE_HOST=localhost go run ./cmd/ipfs-publisher

See ipfs-publisher.md for credential setup, configuration, and architecture details.

Stopping

# Stop containers
docker compose down

# Stop and delete all data (full reset)
docker compose down -v

Docker Compose Services

Service	Description	Port	Profile
`db`	PostgreSQL 15	5432	default
`app`	API server with Air hot reloading	8080	default
`mock-server`	Nginx serving mock JSON files	8081	default
`crawler`	Crawler (one-shot)	—	`crawler`
`ipfs-publisher`	IPFS publisher (one-shot)	—	`ipfs-publisher`

The crawler uses the crawler profile, so it only starts when explicitly requested:

# Start everything including crawler
docker compose --profile crawler up

# Or run crawler as a one-shot command
docker compose run --rm crawler

Volumes:

postgres_data — Persistent database storage
app_tmp — Temporary build artifacts for Air

Configuration

Precedence (highest to lowest)

CLI flag -config (config file location only)
Environment variable WIMAPI_CONFIG_FILE (config file location)
Individual environment variables (e.g., WIMAPI_DATABASE_HOST)
Config file values (config.yaml)
Built-in defaults

Environment Variable Naming

Prefix with WIMAPI_, use underscores for nesting:

database.host → WIMAPI_DATABASE_HOST
server.port → WIMAPI_SERVER_PORT
crawler.orchestrator.batch_size → WIMAPI_CRAWLER_ORCHESTRATOR_BATCH_SIZE

All Settings

Server

Setting	Env Var	Default
`server.port`	`WIMAPI_SERVER_PORT`	`8080`
`server.host`	`WIMAPI_SERVER_HOST`	`localhost`
`server.read_timeout`	`WIMAPI_SERVER_READ_TIMEOUT`	`30` (seconds)
`server.write_timeout`	`WIMAPI_SERVER_WRITE_TIMEOUT`	`30` (seconds)

Database

Setting	Env Var	Default
`database.host`	`WIMAPI_DATABASE_HOST`	`db` (Docker) / `localhost` (local)
`database.port`	`WIMAPI_DATABASE_PORT`	`5432`
`database.user`	`WIMAPI_DATABASE_USER`	`wimapi`
`database.password`	`WIMAPI_DATABASE_PASSWORD`	`wimapi_password`
`database.name`	`WIMAPI_DATABASE_NAME`	`wimapi_db`
`database.ssl_mode`	`WIMAPI_DATABASE_SSL_MODE`	`disable`

Logging

Setting	Env Var	Default
`log.level`	`WIMAPI_LOG_LEVEL`	`info`
`log.file`	`WIMAPI_LOG_FILE`	`wimapi.log`

Crawler Orchestrator

Setting	Env Var	Default
`crawler.orchestrator.batch_size`	`WIMAPI_CRAWLER_ORCHESTRATOR_BATCH_SIZE`	`100`
`crawler.orchestrator.checkpoint_interval`	`WIMAPI_CRAWLER_ORCHESTRATOR_CHECKPOINT_INTERVAL`	`10` (batches)
`crawler.orchestrator.debug_sample_rate`	`WIMAPI_CRAWLER_ORCHESTRATOR_DEBUG_SAMPLE_RATE`	`0.01` (1%)
`crawler.orchestrator.max_retries`	`WIMAPI_CRAWLER_ORCHESTRATOR_MAX_RETRIES`	`3`
`crawler.orchestrator.retry_backoff_base`	`WIMAPI_CRAWLER_ORCHESTRATOR_RETRY_BACKOFF_BASE`	`1s`
`crawler.orchestrator.concurrency`	`WIMAPI_CRAWLER_ORCHESTRATOR_CONCURRENCY`	`2`
`crawler.orchestrator.run_timeout`	`WIMAPI_CRAWLER_ORCHESTRATOR_RUN_TIMEOUT`	`4h`

Crawler HTTP Client

Setting	Env Var	Default
`crawler.http_client.timeout`	`WIMAPI_CRAWLER_HTTP_CLIENT_TIMEOUT`	`30s`
`crawler.http_client.max_idle_conns`	`WIMAPI_CRAWLER_HTTP_CLIENT_MAX_IDLE_CONNS`	`10`

IPFS Publisher

Setting	Env Var	Default	Description
`ipfs.space_did`	`WIMAPI_IPFS_SPACE_DID`	`""`	Storacha space DID (required)
`ipfs.private_key_path`	`WIMAPI_IPFS_PRIVATE_KEY_PATH`	`./ipfs.key`	Path to Ed25519 private key
`ipfs.proof_path`	`WIMAPI_IPFS_PROOF_PATH`	`./proof.ucan`	Path to UCAN delegation proof
`ipfs.gateway_url`	`WIMAPI_IPFS_GATEWAY_URL`	`https://w3s.link`	IPFS gateway base URL
`ipfs.max_retries`	`WIMAPI_IPFS_MAX_RETRIES`	`3`	Upload retry attempts
`ipfs.retry_backoff_base`	`WIMAPI_IPFS_RETRY_BACKOFF_BASE`	`1s`	Base backoff between retries
`ipfs.concurrency`	`WIMAPI_IPFS_CONCURRENCY`	`2`	Parallel WRP processing
`ipfs.batch_size`	`WIMAPI_IPFS_BATCH_SIZE`	`100`	WRPs claimed per batch
`ipfs.run_timeout`	`WIMAPI_IPFS_RUN_TIMEOUT`	`1h`	Maximum run duration

See ipfs-publisher.md for full documentation.

Crawler Sources

Sources are configured as a map under crawler.sources. Each source has:

Setting	Description
`enabled`	Whether this source is active
`type`	Source type (currently: `generic-json`)
`url`	Data URL to fetch
`rate_limit`	Requests per second (simulated for generic-json)

See crawler.md for details on source types and adding new sources.

Example config.yaml

server:
  port: "8080"
  host: "localhost"
  read_timeout: 30
  write_timeout: 30

database:
  host: "db"
  port: "5432"
  user: "wimapi"
  password: "wimapi_password"
  name: "wimapi_db"
  ssl_mode: "disable"

log:
  level: "info"
  file: "wimapi.log"

crawler:
  orchestrator:
    batch_size: 100
    checkpoint_interval: 10
    concurrency: 2
    run_timeout: 4h
  http_client:
    timeout: 30s
    max_idle_conns: 10
  sources:
    mock-api:
      enabled: true
      type: "generic-json"
      url: "http://mock-server/data/mock-crawler-data.json"
      rate_limit: 1000.0

Database Migrations

Migrations run automatically on startup using golang-migrate with embedded SQL files.

How It Works

Migration files live in internal/database/migrations/
Naming convention: NNNNNN_description.up.sql / NNNNNN_description.down.sql
On startup, the app:
- Creates schema_migrations table if needed
- Checks which migrations have been applied
- Runs pending migrations in order
- Reports the current schema version

Migration SQL is compiled into the binary — no need to ship files separately.

Adding a Migration

Create numbered files in internal/database/migrations/:

000003_add_new_feature.up.sql    # Forward migration
000003_add_new_feature.down.sql  # Rollback migration

Best practices:

Use sequential numbering (000001, 000002, ...)
Always provide both .up.sql and .down.sql
Wrap in transactions: BEGIN; ... COMMIT;
Never modify already-applied migration files
Test on a clean database before committing
For production: add indexes CONCURRENTLY (outside transactions) to avoid table locks

Checking Migration Version

docker exec wimapi-db psql -U wimapi -d wimapi_db -c "SELECT * FROM schema_migrations;"

Resetting the Database

docker compose down -v

This removes all volumes, including the PostgreSQL data directory.

Testing

Unit Tests

GOCACHE=$(pwd)/.cache go test -cover ./...

The explicit GOCACHE avoids sandbox permission issues on some systems (e.g., Docker-mounted volumes).

Run a single test or package:

# Single test
go test ./internal/services -run TestRelyingPartyService_Search

# Single package
go test ./internal/services -cover

Integration Tests (Smoke Tests)

./scripts/smoke-test.sh

This script:

Starts the PostgreSQL container if not running
Waits for the database to be ready
Runs integration-tagged tests (//go:build integration) in: ./cmd/seed, ./internal/database, ./internal/database/seed, ./internal/server
Stops the container after tests complete

Crawler E2E Tests

./scripts/crawler-e2e-test.sh

Tests multi-source crawling, resume after SIGTERM, idempotency, change detection, and hard crash (SIGKILL) recovery. Individual tests can be run selectively:

./scripts/crawler-e2e-test.sh test1   # Full multi-source crawl
./scripts/crawler-e2e-test.sh test2   # Resume after SIGTERM
./scripts/crawler-e2e-test.sh test3   # Idempotency
./scripts/crawler-e2e-test.sh test4   # Change detection
./scripts/crawler-e2e-test.sh test5   # Resume after SIGKILL

IPFS Publisher E2E Tests

./scripts/ipfs-e2e-test.sh              # Start DB if needed, run test
./scripts/ipfs-e2e-test.sh --skip-db    # DB already running

Performs real uploads to Storacha and verifies the full publishing pipeline. Requires ipfs.key and proof.ucan in the project root.

Performance Tests

./scripts/perf-test.sh

Seeds a large dataset, then tests pagination, retrieval, and concurrent load. Results are exported to tmp/perf-results-<timestamp>.json.

Env Var	Default	Description
`WIMAPI_PERF_RP_COUNT`	`2000`	Relying parties to seed
`WIMAPI_PERF_PAGE_LIMIT`	`100`	Page size for pagination tests
`WIMAPI_PERF_MAX_PAGES`	`50`	Maximum pages to retrieve
`WIMAPI_PERF_HTTP_CONCURRENCY`	`16`	HTTP request concurrency
`WIMAPI_PERF_STRESS_CONCURRENCY`	`32`	Stress test concurrency
`WIMAPI_PERF_STRESS_DURATION_SEC`	`15`	Stress test duration (seconds)

Test Patterns by Layer

Layer	Approach	Example
Repositories	Lightweight pgx stubs (no live DB) — assert query args, scanning, errors	`internal/repositories/relying_party_repository_test.go`
Services	Testify mocks for repositories — assert business rules, pagination, filters	`internal/services/relying_party_service_test.go`
Handlers	Gin test context, call handler methods directly — assert status codes, JSON shape	`internal/server/handlers_helper_test.go`
Router integration	`NewServer()` + `httptest.Server` — verify routing, middleware, error mapping	`internal/server/router_integration_test.go`
Smoke tests	Integration-tagged, require live DB — exercise endpoints end-to-end	`internal/server/api_smoke_test.go`

Tests are colocated with implementation files and follow the Test<Function> naming convention.

Common Troubleshooting

"connection refused" to database:

Local development requires WIMAPI_DATABASE_HOST=localhost (the default db is the Docker hostname)

Stale data or schema issues:

Reset with docker compose down -v to drop all data and re-run migrations on next startup

Duplicate key errors during seeding:

Use the -fresh flag: go run ./cmd/seed -fresh to truncate all tables before inserting