Development & Contribution Guide

usulnet - Docker Management Platform Guide for setting up the development environment and contributing to the project.

---

Table of Contents

1. Prerequisites
2. Development Environment Setup
3. Project Structure
4. Makefile Reference
5. Development Workflow
6. Code Style Guide
7. Commit Conventions
8. Testing
9. Pull Request Process
10. Common Development Tasks
11. Profiling

---

Prerequisites

Required Tools

Tool	Version	Installation
Go	1.25+	go.dev/dl
Docker	24.0+	docs.docker.com
Docker Compose	v2.20+	Included with Docker Desktop or docker-compose-plugin
templ	0.3.977+	go install github.com/a-h/templ/cmd/templ@latest
Git	2.40+	System package manager

Optional Tools

Tool	Purpose	Installation
golangci-lint	Code linting	go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
k6	Load testing	k6.io/docs/get-started
psql	Database debugging	apt install postgresql-client
redis-cli	Cache debugging	apt install redis-tools

Note: Tailwind CSS standalone CLI is downloaded automatically by make css. No Node.js or npm is required.

---

Development Environment Setup

Step 1: Clone the Repository

git clone https://github.com/fr4nsys/usulnet.git
cd usulnet

Step 2: Start Infrastructure Services

The development compose file starts PostgreSQL, Redis, and NATS with ports exposed for debugging:

make dev-up

This runs docker-compose.dev.yml which starts:

Service	Host Port	Purpose
PostgreSQL	5432	Database
Redis	6379	Cache/sessions
NATS	4222 (client), 8222 (monitoring)	Messaging

Step 3: Install Go Dependencies

make deps

Step 4: Generate Templates and CSS

make frontend

This runs templ generate to compile .templ files to Go code, and downloads + runs the Tailwind CSS standalone CLI to compile web/static/src/input.css to web/static/css/style.css.

Step 5: Run Database Migrations

make migrate

Step 6: Run the Application

make run

The application starts on http://localhost:8080. Default credentials: admin / usulnet.

Development with Hot Reload

For active development, run template and CSS watchers in separate terminals:

# Terminal 1: Watch templates
make templ-watch

# Terminal 2: Watch CSS
make css-watch

# Terminal 3: Run the app (restart manually on Go changes)
make run

Development with Agent

To also start an agent for multi-host development:

make dev-up-agent

Stopping the Environment

make dev-down

---

Project Structure

usulnet/
+-- cmd/                      # Application entry points
|   +-- usulnet/              # Main server (cobra CLI: serve, migrate)
|   +-- usulnet-agent/        # Remote agent binary
+-- internal/                 # Private application code
|   +-- api/                  # REST API (handlers, middleware, DTOs, router)
|   +-- web/                  # Web UI (page handlers, adapters, templates)
|   +-- app/                  # Bootstrap, config, scheduler setup
|   +-- services/             # Business logic (37 packages)
|   +-- repository/           # Data access (PostgreSQL repos, Redis, migrations)
|   +-- models/               # Domain models and types
|   +-- docker/               # Docker Engine client wrapper
|   +-- gateway/              # NATS gateway (master side)
|   +-- agent/                # Agent implementation
|   +-- nats/                 # NATS client wrapper
|   +-- scheduler/            # Cron job scheduler
|   +-- license/              # License validation
|   +-- integrations/         # External integrations (Git providers)
|   +-- observability/        # Logging, tracing
|   +-- pkg/                  # Shared utilities (crypto, errors, logger, totp, validator)
+-- web/static/               # Frontend assets (CSS, JS)
+-- deploy/                   # Production deployment files
+-- tests/                    # Test suites (e2e, benchmarks, load)
+-- scripts/                  # Build scripts
+-- docs/                     # Documentation
+-- .github/workflows/        # CI/CD pipelines

Key Patterns

• Handlers (internal/web/handler_*.go): Each handler serves one or more related web pages. They use adapters to fetch data from services.
• Adapters (internal/web/adapter_*.go): Bridge between web handlers and services. Translate between web-layer DTOs and service-layer models.
• Services (internal/services/*/): Contain business logic. Created via constructor injection (NewService(deps)). Depend on interfaces for testability.
• Repositories (internal/repository/postgres/): Data access objects using pgx/v5. Each repository implements a specific interface.
• Templates (internal/web/templates/): Templ files (.templ) that compile to type-safe Go functions.

---

Makefile Reference

Target	Description
make all	Full build: templ + css + lint + test + build
make build	Build the main binary (includes frontend generation)
make build-agent	Build the agent binary
make build-all	Build both binaries
make run	Run the application with go run
make templ	Generate Go code from .templ files
make templ-watch	Watch mode for template generation
make css	Compile Tailwind CSS
make css-watch	Watch mode for CSS compilation
make frontend	Run both templ and css
make test	Run all tests with race detector and coverage
make test-coverage	Generate HTML coverage report (coverage.html)
make test-check-coverage	Check coverage (interim 15%, target 40%; auto-generated _templ.go excluded)
make test-benchmark	Run performance benchmarks
make test-e2e	Run end-to-end tests
make lint	Run golangci-lint
make lint-fix	Run linter with auto-fix
make fmt	Format code with gofmt
make vet	Run go vet
make quality	Run all quality checks (lint + vet + coverage)
make migrate	Apply pending database migrations
make migrate-down	Rollback database migrations
make migrate-status	Show migration status
make dev-up	Start development infrastructure (PostgreSQL, Redis, NATS)
make dev-down	Stop development infrastructure
make dev-logs	View development service logs
make dev-up-agent	Start development with agent profile
make docker-build	Build main Docker image
make docker-build-agent	Build agent Docker image
make deps	Download and tidy Go modules
make generate	Run go generate
make clean	Remove build artifacts
make install-hooks	Install git pre-commit hook

---

Development Workflow

1. Create a Feature Branch

git checkout -b feat/my-feature

2. Implement Changes

• Write code following the Code Style Guide
• Add or update tests for your changes
• Run make templ if you modified .templ files
• Run make css if you added new Tailwind classes

3. Validate Locally

# Run tests
make test

# Run linter
make lint

# Full quality gate
make quality

4. Commit Changes

Follow Commit Conventions:

git add -A
git commit -m "feat(containers): add bulk restart operation"

5. Push and Create PR

git push -u origin feat/my-feature

Create a pull request on GitHub following the PR Process.

---

Code Style Guide

General Principles

• Follow standard Go idioms (gofmt, govet)
• Exported identifiers must have documentation comments (godoc format)
• Keep functions short and focused (< 50 lines preferred)
• Return early on errors (guard clauses)
• Use context propagation (ctx context.Context as first parameter)

Error Handling

// Use the internal errors package
import "github.com/fr4nsys/usulnet/internal/pkg/errors"

// Wrap errors with context
if err != nil {
    return errors.Wrap(err, "failed to list containers")
}

// Return domain errors for expected failures
return errors.NotFound("container %s not found", containerID)

Logging

// Use structured logging with context
logger := logger.FromContext(ctx)
logger.Info("Container started",
    "container_id", containerID,
    "host_id", hostID,
)

// Never log sensitive data
// BAD: logger.Info("User login", "password", password)
// GOOD: logger.Info("User login", "username", username)

Service Pattern

// Constructor injection
type Service struct {
    containerRepo repository.ContainerRepository
    dockerClient  docker.Client
    logger        *logger.Logger
}

func NewService(repo repository.ContainerRepository, client docker.Client, log *logger.Logger) *Service {
    return &Service{
        containerRepo: repo,
        dockerClient:  client,
        logger:        log,
    }
}

// Interface-based dependencies for testability
type ContainerRepository interface {
    List(ctx context.Context, filters ListFilters) ([]models.Container, error)
    Get(ctx context.Context, id string) (*models.Container, error)
    // ...
}

Repository Pattern

// Use parameterized queries (NEVER string concatenation)
func (r *containerRepo) List(ctx context.Context, hostID string) ([]models.Container, error) {
    query := `SELECT id, name, status FROM containers WHERE host_id = \$1 ORDER BY created_at DESC`
    rows, err := r.pool.Query(ctx, query, hostID)
    // ...
}

Template Pattern

// Templ templates in internal/web/templates/pages/
// Use typed props, not interface{}
templ ContainerList(containers []ContainerView, pagination PaginationView) {
    @layouts.Base("Containers") {
        <div class="p-6">
            for _, c := range containers {
                @ContainerCard(c)
            }
        </div>
    }
}

Naming Conventions

Item	Convention	Example
Package	lowercase, short	container, auth, backup
Interface	noun or -er suffix	ContainerRepository, Authenticator
Struct	PascalCase	ContainerService, BackupHandler
Method	PascalCase (exported), camelCase (unexported)	ListContainers, parseFilters
Constant	PascalCase or ALL_CAPS	MaxRetries, DefaultTimeout
File	snake_case	container_handler.go, auth_service.go

---

Commit Conventions

This project uses Conventional Commits:

<type>(<scope>): <description>

[optional body]

[optional footer(s)]

Types

Type	Description
feat	New feature
fix	Bug fix
docs	Documentation changes
style	Code style (formatting, missing semicolons, etc.)
refactor	Code refactoring (no feature or bug fix)
perf	Performance improvement
test	Adding or updating tests
build	Build system or external dependencies
ci	CI/CD configuration
chore	Maintenance tasks

Scopes (Optional)

Scope	Area
api	REST API handlers/middleware
web	Web UI handlers/templates
containers	Container management
images	Image management
stacks	Stack management
hosts	Host management
agent	Agent system
security	Security scanning
backup	Backup/restore
proxy	Reverse proxy
auth	Authentication/authorization
db	Database/migrations
config	Configuration
ci	CI/CD
docs	Documentation

Examples

feat(containers): add bulk restart operation
fix(auth): prevent timing attack on login
docs(api): add curl examples for container endpoints
refactor(security): extract scanner interface
test(backup): add integration tests for S3 storage
ci: add coverage threshold check to pipeline
chore: update Go to 1.25.7

---

Testing

Running Tests

# All tests with race detector
make test

# Generate coverage report
make test-coverage
# Open coverage.html in browser

# Check coverage threshold (interim 15%, target 40%; auto-generated _templ.go excluded)
make test-check-coverage

# Run benchmarks
make test-benchmark

# Run E2E tests (requires infrastructure)
make test-e2e

Test Structure

Tests follow Go conventions:

internal/
  services/
    container/
      service.go
      service_test.go      # Unit tests
  api/
    handlers/
      containers.go
      containers_test.go   # Integration tests with httptest
      testutil_test.go     # Test helpers and fixtures

Writing Tests

func TestContainerService_List(t *testing.T) {
    // Arrange
    repo := &mockContainerRepo{
        containers: []models.Container{{ID: "abc123", Name: "test"}},
    }
    svc := container.NewService(repo, nil, logger.Nop())

    // Act
    result, err := svc.List(context.Background(), container.ListFilters{})

    // Assert
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if len(result) != 1 {
        t.Errorf("expected 1 container, got %d", len(result))
    }
}

Test Infrastructure

For integration tests, use docker-compose.test.yml:

# Start test infrastructure (isolated ports)
docker compose -f docker-compose.test.yml up -d

# Run E2E tests
make test-e2e

# Stop test infrastructure
docker compose -f docker-compose.test.yml down -v

Test infrastructure uses isolated ports:

• PostgreSQL: 15432
• Redis: 16379
• NATS: 14222

---

Pull Request Process

Before Creating a PR

1. Ensure all tests pass: make test
2. Ensure linter passes: make lint
3. Run the full quality gate: make quality
4. Verify templates compile: make templ
5. Verify CSS compiles: make css

PR Template

## Summary
Brief description of the changes.

## Changes
- Added X
- Fixed Y
- Updated Z

## Testing
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Manual testing performed

## Screenshots
(if UI changes)

Review Process

1. Create a PR from your feature branch to main
2. CI pipeline runs automatically (lint, test, build, security scan)
3. At least one reviewer must approve
4. All CI checks must pass
5. Squash merge or regular merge (maintainer's choice)

---

Common Development Tasks

Adding a New API Endpoint

1. Create/update the handler in internal/api/handlers/
2. Define request/response DTOs in internal/api/dto/
3. Register routes in the handler's Routes() method
4. Mount the handler in internal/api/router.go
5. Add tests in *_test.go

Adding a New Web Page

1. Create the handler method in the appropriate internal/web/handler_*.go
2. Create the Templ template in internal/web/templates/pages/
3. Create an adapter in internal/web/adapter_*.go if needed
4. Register the route in internal/web/routes_frontend.go
5. Run make templ to compile

Adding a Database Migration

1. Create new migration files:

   internal/repository/postgres/migrations/
     031_my_feature.up.sql
     031_my_feature.down.sql
   

2. Write the UP migration (create tables, add columns, etc.)
3. Write the DOWN migration (reverse the UP changes)
4. Apply: make migrate
5. Verify rollback works: make migrate-down then make migrate

Adding a New Service

1. Create the package: internal/services/myservice/
2. Define the service struct with constructor injection
3. Define the interface for testability
4. Wire the service in internal/app/app.go
5. Add tests

Debugging

# View application logs
make run 2>&1 | jq .  # If JSON logging

# Connect to database
docker exec -it usulnet-postgres psql -U usulnet

# Connect to Redis
docker exec -it usulnet-redis redis-cli

# Check NATS monitoring
curl http://localhost:8222/varz

# Check Docker socket
curl --unix-socket /var/run/docker.sock http://localhost/version

---

Profiling

The Go runtime ships with first-class CPU and heap profilers; this section is the repeatable procedure for using them against usulnet.

The goal is not to chase micro-optimizations — it's to find the real hot paths and prove any change pays for itself with a benchmark before/after. Two rules govern this loop:

1. Numbers come from the same machine on the same load. Capture the baseline (git stash), pop the patch, capture the after run. Never compare a stash-pop after to a number you wrote down hours earlier — machine load drift is huge and will fool you.
2. <5% improvement is noise — drop the change. Use -benchtime=5s -count=8 so per-bench variance is small enough for a 5% delta to actually mean something. If the delta is smaller than that, your change is decoration, not optimization.

Tools

Tool	Install	Purpose
go test -bench	bundled with Go	Repeatable micro-benchmarks (the source of truth for "is this faster?").
go tool pprof	bundled with Go	Analyze CPU and heap profiles.
hey	go install github.com/rakyll/hey@latest	Hammer a running server when you want a profile that reflects real HTTP traffic.
k6	https://k6.io	Optional — tests/load/k6_api_test.js is shipped; use it for scripted scenarios that hit auth + multiple endpoints.

Baseline benchmarks (per-PR before/after)

tests/benchmarks/benchmark_test.go covers the API hot paths (router, JWT, JSON, health, paginated responses). The repeatable A/B loop:

# 1. Baseline (current code).
git stash
go test -bench=. -benchmem -benchtime=5s -count=8 \
    ./tests/benchmarks/ | tee /tmp/profile/bench-before.txt

# 2. Apply the change.
git stash pop

# 3. After.
go test -bench=. -benchmem -benchtime=5s -count=8 \
    ./tests/benchmarks/ | tee /tmp/profile/bench-after.txt

# 4. Compare (benchstat is the canonical diff tool).
go install golang.org/x/perf/cmd/benchstat@latest
benchstat /tmp/profile/bench-before.txt /tmp/profile/bench-after.txt

Paste the benchstat output into the PR body. The Liveness fast-path in #46 is the worked example — ns/op -9.0%, B/op -5.6%, allocs/op -14.3%, all comfortably above the 5% noise floor.

CPU and heap profiles from the benchmarks

When the question is which line dominates, capture profiles from the benchmark binary itself — they're reproducible across machines and don't need any server up:

mkdir -p /tmp/profile
go test -bench=. -benchmem -benchtime=5s -count=1 \
    -cpuprofile=/tmp/profile/cpu.prof \
    -memprofile=/tmp/profile/mem.prof \
    ./tests/benchmarks/

# Top-N flat (where time is *spent*, not just attributed):
go tool pprof -top -flat -nodecount=20 \
    ./tests/benchmarks/benchmarks.test /tmp/profile/cpu.prof

# Top-N cumulative (which call tree owns the cost):
go tool pprof -top -cum -nodecount=15 \
    ./tests/benchmarks/benchmarks.test /tmp/profile/cpu.prof

# Drill into a specific function (use the symbol shown by -top):
go tool pprof -list 'SystemHandler.*Health' \
    ./tests/benchmarks/benchmarks.test /tmp/profile/cpu.prof

# Heap (objects allocated, not just bytes — alloc count usually
# dominates GC pressure):
go tool pprof -top -alloc_objects -nodecount=15 \
    ./tests/benchmarks/benchmarks.test /tmp/profile/mem.prof

Profiles from a running server (real traffic shape)

The benchmarks miss anything outside the API hot path (DB queries, templated HTML, websocket terminal, background workers). For those, profile against a running usulnet serve while a load generator hammers it:

# Terminal 1 — run the server.
make dev-up
USULNET_RECON_ENABLED=true make run

# Terminal 2 — sustained load. Pick whichever shape matches what
# you're investigating. JWT auth path:
TOKEN=$(curl -sS -X POST http://127.0.0.1:8080/api/v1/auth/login \
    -H 'Content-Type: application/json' \
    -d '{"username":"admin","password":"usulnet"}' | jq -r .access_token)
hey -z 60s -c 50 \
    -H "Authorization: Bearer $TOKEN" \
    http://127.0.0.1:8080/api/v1/containers

# Terminal 3 — capture CPU + heap concurrently.
go tool pprof -seconds=30 \
    http://127.0.0.1:8080/debug/pprof/profile > /tmp/profile/cpu-live.prof
go tool pprof http://127.0.0.1:8080/debug/pprof/heap > /tmp/profile/heap-live.prof

Note (v26.5.0): the /debug/pprof endpoints are not wired into the API router yet; profiling a running server today still goes through the benchmark binary, or requires a debug build that imports net/http/pprof. Adding a config-gated /debug/pprof mount on a separate listener is a small, isolated follow-up that belongs in its own PR.

Baseline snapshot — May 2026

Reference numbers from tests/benchmarks/ on an Intel Xeon @ 2.10 GHz, 4 cores, Go 1.25.7, -benchtime=5s -count=1. Use them to spot regressions, not as a fixed target — machine drift makes absolute numbers noisy across hardware.

Benchmark	ns/op	B/op	allocs/op
LivenessEndpoint (post-#46)	4 638	7 053	30
HealthEndpoint (2 checkers)	7 464	8 273	50
HealthWithMultipleCheckers (4 checkers)	9 696	9 158	68
AuthenticatedRequest	123 168	13 187	117
JWTTokenGeneration	4 186	2 905	34
JWTTokenValidation	6 794	3 264	53
JSONSerialization	1 294	512	7
JSONDeserialization	1 201	1 008	10
PaginatedResponse (100 items, []map[string]any)	62 421	27 859	703

Top cumulative CPU contributors (from the May-2026 baseline run):

1. runtime.mallocgc — ~27 % cum. Allocation-heavy: any reduction in per-request allocs feeds back here.
2. encoding/json (mapEncoder.encode, structEncoder.encode, appendString) — ~19 % cum. Map-shaped payloads are the expensive case (key sort, reflection, per-element encoder lookup); typed structs are several times cheaper. The PaginatedResponse benchmark's 703 allocs/op is the smoking gun — it's encoding []map[string]any.
3. chi/v5.(*Mux).ServeHTTP — ~19 % cum. Routing trie walk plus middleware orchestration. Hard to move without forking chi; middleware ordering is the lever we control.
4. JWT validation (jwt.ParseWithClaims + HMAC SHA-256) — ~1 % cum flat, but the dominant per-call cost on AuthenticatedRequest. CPU-bound HMAC; the practical lever is caching validated tokens for short windows when the token store already supports it.

Known wins shipped so far

• #46 — Liveness fast-path. Pre-encoded response body. -9.0 % ns/op, -14.3 % allocs/op.

Catalogued non-wins (and why)

These were tried and rejected because they didn't clear the 5% threshold or because the win was masked by an unrelated bottleneck. Recorded here so future profiling rounds don't redo the work:

• Health-handler single-checker fast-path. A synchronous code path for the n == 1 case avoids a goroutine spawn, the WaitGroup, the per-component mutex and a context.WithTimeout child. Real win — but the existing BenchmarkHealthEndpoint setup uses 2 mock checkers and BenchmarkHealthWithMultipleCheckers uses 4, so the optimised branch is never executed. The win would only show up against a new single-checker benchmark; not in this series.
• Hoisting the per-request secrets slice in middleware.Auth. ~40 B and one append per authenticated request. Total impact on BenchmarkAuthenticatedRequest (~13 kB/req) is well below 1 % — drowned in JWT verification cost. Skipped.

Profiling fix workflow

1. Pick one hot path. Don't stack optimisations.
2. Capture before with -count=8 so per-run jitter washes out.
3. Make the change.
4. Capture after the same way. Run benchstat.
5. ≥ 5 % on at least one of ns/op, B/op, allocs/op? Ship it (one PR per fix), paste the benchstat table in the PR body.
6. < 5 %? Drop the change. Add a brief entry to the "Catalogued non-wins" list above so the next profiler doesn't chase the same dead end.

---

For more information, see the Architecture Guide and API Documentation.