Runbook: Control-Plane Rolling Rollout

August 4, 2026 ยท View on GitHub

Goal

Replace Duckgres control-plane replicas without breaking most existing sessions during a planned deployment.

Requirements

  • The Deployment uses rolling replacement with overlap:
    • maxUnavailable: 0
    • maxSurge: 1
  • terminationGracePeriodSeconds is at least the configured drain timeout
  • The control plane sets --handover-drain-timeout 15m (or another explicit value appropriate for the cluster)

Expected behavior

  1. The old replica receives SIGTERM.
  2. It closes local pgwire admission and fails /health.
  3. It publishes draining in runtime state.
  4. New pgwire sessions are rejected on the draining replica.
  5. Existing pgwire connections continue until they finish or the drain timeout expires.
  6. When the timeout expires, the replica force-shuts down remaining sessions and workers.

See Org connection admission for the mixed-version admission boundary during a rolling deployment.

Unplanned control-plane failure is different:

  • live pgwire connections are lost immediately
  • clients must establish a new pgwire connection and worker session

Rollout procedure

  1. Start the rollout.

    kubectl -n duckgres rollout restart deploy/duckgres-control-plane
kubectl -n duckgres rollout status deploy/duckgres-control-plane

  2. Watch old and new pods during overlap.

    kubectl -n duckgres get pods -l app=duckgres-control-plane -w

  3. Verify the old pod becomes unready before it exits.

    kubectl -n duckgres get pods -l app=duckgres-control-plane
kubectl -n duckgres logs <old-pod-name>

  4. Confirm the new pod is serving traffic and new sessions acquire workers.

    • sum(duckgres_worker_lifecycle_count{state="spawning"}) settles back toward 0 (on-demand spawns succeed)
    • sum(duckgres_worker_lifecycle_count{state="hot"}) does not drop unexpectedly
    • client reconnect errors do not spike

If a rollout stalls

  • Check whether the old pod is still draining active sessions: 
    kubectl -n duckgres logs <old-pod-name> | rg "drain|draining|shutdown"
  • Check whether the pod termination grace period is shorter than the configured drain timeout.
  • Check whether long-lived idle clients are holding pgwire connections open.

If the timeout is too short

  • Increase both:
    • --handover-drain-timeout
    • terminationGracePeriodSeconds

Keep those values aligned. If the pod is killed before the drain timeout elapses, Kubernetes will cut the drain short.