Multi-Node Deployment Requirements

May 5, 2026 · View on GitHub

Overview

This document outlines the requirements and recommendations for deploying the workflow v2 engine across multiple server nodes. Multi-node deployments enable horizontal scaling, high availability, and geographic distribution.

The v2 engine separates correctness substrate from acceleration layer. The shared workflow database is the correctness substrate; every multi-node guarantee in this document depends on it. Shared cache and the wake-notification layer are acceleration: they shorten discovery latency without affecting which tasks are eligible or who claims them. A deployment whose acceleration backend is degraded continues to make correct progress, bounded by the durable poll and repair cadence. See docs/architecture/scheduler-correctness.md for the full contract, including the reversible migration path that lets a deployment move between cache-coordinated wake notification and a stronger acceleration primitive (Redis pub/sub, PostgreSQL LISTEN/NOTIFY, NATS) without a cutover.

The engine-side contract for high availability and failover behavior inside a single region — managed-database failover, managed-Redis failover, API-node loss, worker loss, scheduler-runner restart, and the load-balancer / readiness rules that govern traffic shift during each event — is frozen in docs/deployment/ha-failover.md. The cross-region active/passive recovery contract lives separately in docs/deployment/multi-region.md.

Prerequisites

Shared Database

All nodes must connect to the same database instance:

  • ✅ MySQL 8.0+
  • ✅ PostgreSQL 13+
  • ✅ MariaDB 10.5+

Connection Configuration:

All nodes share identical database credentials:

DB_CONNECTION=mysql
DB_HOST=db.internal.example.com
DB_PORT=3306
DB_DATABASE=workflows
DB_USERNAME=workflow_user
DB_PASSWORD=<secure-password>

Connection Pooling:

For high-throughput deployments, use connection pooling:

  • ProxySQL (MySQL)
  • PgBouncer (PostgreSQL)
  • RDS Proxy (AWS)

Shared Cache Backend (Acceleration Layer)

A multi-node deployment runs correctly without a shared cache backend — discovery latency simply rises to the durable long-poll and task-repair cadence. Configuring a shared cache backend recovers the sub-second discovery latency that wake signals provide and is strongly recommended for production.

Acceleration backends supported for multi-node:

  • ✅ Redis 6.0+ (recommended)
  • ✅ Database cache (MySQL/PostgreSQL)
  • ✅ Memcached 1.6+
  • ❌ File cache — wake signals do not propagate across nodes
  • ❌ Array cache — process-local; signals are lost on restart

The file and array cache stores cannot carry the multi-node acceleration layer because they cannot propagate wake signals between nodes. Pollers fall back to long-poll timeout when the acceleration layer is absent or unable to propagate; the durable substrate continues to govern correctness.

Redis Configuration:

All nodes share identical Redis credentials:

CACHE_DRIVER=redis
REDIS_HOST=redis.internal.example.com
REDIS_PASSWORD=<secure-password>
REDIS_PORT=6379
REDIS_DB=1

Database Cache Configuration:

Uses shared database for cache storage:

CACHE_DRIVER=database
# DB_* credentials same as above

Why a Shared Cache Backend Is Recommended:

When Node A creates a task, Node B's pollers can observe the wake signal and re-probe immediately rather than waiting for the long-poll timeout. File cache is per-node, so wake signals never reach other nodes — discovery still occurs on the next poll, just with higher latency. The system never silently loses work in this state; the backend_capabilities health check surfaces a warning so operators can detect and repair the acceleration layer.

See Long-Poll Coordination for the wake-signal protocol details and Scheduler Correctness for the bounded discovery latency contract that holds even with the acceleration layer absent.

Boot-time validation:

Set DW_V2_MULTI_NODE=true (env) or workflows.v2.long_poll.multi_node=true (config) to enable boot-time validation that the configured cache backend can carry the acceleration layer. Tune behaviour with DW_V2_VALIDATE_CACHE_BACKEND (default true) and DW_V2_CACHE_VALIDATION_MODE (fail, warn, or silent; default warn). The cache admission is warning-only by contract: silent suppresses the diagnostic, and warn and fail both log a warning without blocking boot. Validation failures surface through the backend_capabilities health check; they do not affect correctness.

Node Configuration

Environment Variables

All nodes should have identical configuration for workflow behavior:

# Namespace (consistent across nodes)
DW_V2_NAMESPACE=production

# Compatibility (deploy same build to all nodes)
DW_V2_CURRENT_COMPATIBILITY=build-20260415-1a2b3c4
DW_V2_SUPPORTED_COMPATIBILITIES=build-20260415-1a2b3c4

# Task dispatch
DW_V2_TASK_DISPATCH_MODE=queue

# Limits (consistent across nodes)
DW_V2_LIMIT_PENDING_ACTIVITIES=2000
DW_V2_LIMIT_PENDING_CHILDREN=1000
# ... etc

The legacy WORKFLOW_V2_* names are honored as fallbacks during the deprecation window, but new deployments should use the DW_V2_* primary names.

Node-Specific Variables

Each node can have unique values for:

# Worker identity (unique per node)
APP_NAME=workflow-node-1

# Local storage
FILESYSTEM_DISK=local

# Logging
LOG_CHANNEL=stack
LOG_LEVEL=info

Deployment Process

Zero-Downtime Deployment

  1. Pre-deployment validation:

    # Verify database connectivity
php artisan migrate:status

# Verify cache connectivity
php artisan tinker
>>> cache()->put('deploy-test', time());
>>> cache()->get('deploy-test');

  2. Deploy to first node (canary):

    git pull origin main
composer install --no-dev --optimize-autoloader
php artisan migrate --force
php artisan config:cache
php artisan route:cache
php artisan view:cache

  3. Verify canary health:

    curl https://node-1.example.com/api/health

  4. Deploy to remaining nodes (rolling):

    • Deploy one node at a time
    • Wait for health check to pass
    • Move to next node

  5. Post-deployment verification:

    # Verify all nodes responding
for node in node-{1..5}; do
  curl https://$node.example.com/api/health
done

Database Migrations

Critical: Run migrations on ONE node only before deploying to others.

# On deployment leader node
php artisan migrate --force

Other nodes will see migrated schema on boot. Do NOT run migrations on multiple nodes simultaneously.

Load Balancing

HTTP Load Balancer

Distribute control plane API requests across nodes:

upstream workflow_api {
    least_conn;
    server node-1.example.com:443;
    server node-2.example.com:443;
    server node-3.example.com:443;
}

server {
    listen 443 ssl;
    server_name api.workflows.example.com;
    
    location / {
        proxy_pass https://workflow_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Health Check Endpoint:

GET /api/health

Returns 200 if node healthy.

Worker Distribution

Workers connect directly to database and cache. No load balancer needed.

Horizontal Scaling:

  • Add more nodes → more polling capacity
  • Each node polls independently
  • Tasks claimed atomically (database fencing)
  • No coordination overhead beyond wake signals

Monitoring

Key Metrics Per Node

Task Throughput:

  • Workflow tasks claimed/sec
  • Activity tasks claimed/sec
  • Task completion rate

Poll Efficiency:

  • Wake signal hit rate (% polls triggered by wake signal vs timeout)
  • Timing hint hit rate (% polls using available_at hint)
  • Average poll latency

Cache Health:

  • Cache hit rate
  • Cache latency (p50, p99)
  • Cache error rate

Database Health:

  • Query latency (p50, p99)
  • Connection pool utilization
  • Deadlock rate

Aggregate Metrics

Cluster Capacity:

  • Total tasks/sec across all nodes
  • Total active workers
  • Queue depth (ready tasks not yet claimed)

Wake Signal Propagation:

  • Time from task creation to worker poll (p50, p99)
  • Cross-node wake latency

Troubleshooting

Symptom: Tasks Stuck in Ready State

Check: Are all nodes polling?

-- Count ready tasks by queue
SELECT queue, COUNT(*) 
FROM workflow_tasks 
WHERE status = 'ready' 
GROUP BY queue;

Fix: Ensure workers running on all nodes.

Symptom: Workers on Other Nodes Pick Up Tasks Only After Long-Poll Timeout

This is the wake-acceleration layer reporting degradation. Tasks still flow — pollers fall back to the configured long-poll timeout (default 30 seconds) and the durable task-repair loop continues to redeliver work — but the sub-second latency that wake signals provide is lost. Correctness is unaffected.

Check: Cache backend coordination

# Node 1: Create wake signal
php artisan tinker
>>> app('Workflow\V2\Support\CacheLongPollWakeStore')->signal('test-channel');

# Node 2: Check signal received
>>> $store = app('Workflow\V2\Support\CacheLongPollWakeStore');
>>> $before = $store->snapshot(['test-channel']);
>>> # (Node 1 signals again)
>>> $store->changed($before); // Should return true

Fix: Verify the shared cache configuration. Check network connectivity between nodes and cache backend. Inspect the backend_capabilities and long_poll_wake_acceleration health checks for explicit wake-layer status (acceleration-layer issues escalate to warning, never error).

If workers receive no tasks at all (not just delayed tasks), the symptom is durable-substrate failure, not acceleration degradation; investigate workflow_tasks repair, claim fencing, and worker compatibility instead.

Symptom: Duplicate Task Execution

This should not happen due to atomic claim fencing. If it does:

  1. Check database isolation level:

    SELECT @@transaction_isolation; -- Should be READ-COMMITTED or higher

  2. Check for clock skew between nodes:

    date -u # Run on all nodes, compare

  3. File bug report with reproduction steps.

Best Practices

  1. Start with 2-3 nodes for most workloads
  2. Scale horizontally by adding nodes (not vertically)
  3. Monitor wake signal latency to detect cache backend issues
  4. Use connection pooling for high-throughput workloads
  5. Keep active-region nodes in the same datacenter/region to minimize latency between API nodes, the workflow database, and Redis. For regional failover, see the self-serve active/passive contract in multi-region.md.
  6. Use blue-green deployment for zero-downtime migrations
  7. Test multi-node locally using Docker Compose before production

Example Docker Compose Setup

version: '3.8'

services:
  db:
    image: mysql:8.0
    environment:
      MYSQL_DATABASE: workflows
      MYSQL_ROOT_PASSWORD: secret
    volumes:
      - db_data:/var/lib/mysql
      
  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
      
  node-1:
    build: .
    environment:
      APP_NAME: workflow-node-1
      CACHE_DRIVER: redis
      REDIS_HOST: redis
      DB_HOST: db
      DB_DATABASE: workflows
      DB_USERNAME: root
      DB_PASSWORD: secret
    depends_on:
      - db
      - redis
      
  node-2:
    build: .
    environment:
      APP_NAME: workflow-node-2
      CACHE_DRIVER: redis
      REDIS_HOST: redis
      DB_HOST: db
      DB_DATABASE: workflows
      DB_USERNAME: root
      DB_PASSWORD: secret
    depends_on:
      - db
      - redis

volumes:
  db_data:
  redis_data:

Run:

docker-compose up -d
docker-compose exec node-1 php artisan migrate
docker-compose logs -f

Security Considerations

  1. Network isolation: Database and cache should not be public
  2. TLS/SSL: Use encrypted connections for database and Redis
  3. Authentication: Secure database and cache with strong passwords
  4. Firewall: Only workflow nodes should reach database/cache ports
  5. Secrets management: Use environment-specific secret stores (Vault, AWS Secrets Manager)

Performance Tuning

Database

  • Index optimization: Ensure indexes on workflow_tasks (status, queue, available_at)
  • Connection pooling: Use PgBouncer or ProxySQL
  • Query optimization: Monitor slow query log

Cache

  • Redis persistence: Consider RDB snapshots + AOF for durability
  • Memory allocation: Plan for 10MB per 10,000 active channels
  • Eviction policy: Use volatile-lru or allkeys-lru

Application

  • Polling concurrency: Increase workers per node for throughput
  • Task dispatch mode: Use queue mode for background processing
  • Lease durations: Tune based on task execution time (default: 10s)