HA / failover playbook

April 28, 2026 · View on GitHub

What "high availability" actually means for an OpenSIPS deployment, the two patterns operators choose between, and the exact tool-call sequence for routine maintenance, planned failover, and unplanned failover.

Audience: intermediate-to-advanced operator with at least one production OpenSIPS node. If you don't have an HA cluster yet, the residential-PBX tutorial gets you a single node first; this doc layers HA on top.

What HA actually means here

There is no single answer. There are three orthogonal things operators mean by "OpenSIPS HA":

  1. Signalling availability — there is always a node that can answer SIP and proxy INVITE / REGISTER traffic. Solved by N-of-M nodes behind a load balancer, with a way to share dialog + usrloc state.
  2. State persistence — when one node dies mid-call, the in-flight dialog is recoverable on another node so the call continues instead of being torn down. Hard. Requires clusterer + proto_bin for replication, and the surviving node must have a compatible config.
  3. Configuration availability — when you push a config change, you roll it out one node at a time without dropping calls. Solved by per-node cluster_drain_node + reload-and-reactivate.

Most production OpenSIPS clusters do (1) and (3) well, with (2) as a best-effort. Full session preservation is a research-grade problem because the upstream / downstream UAs need to also accept the takeover via SIP-level mechanisms (Replaces / target-refresh / P-Asserted-Identity continuity).

The two patterns

Pattern A — N-of-M behind a load balancer

                  +------ haproxy / IPVS / route53 ------+
                  |              |                |
                  v              v                v
              opensips-1     opensips-2      opensips-3
                  |              |                |
                  +------> [shared MySQL] <-------+
                  +-------> [shared usrloc / dialog
                             via clusterer + proto_bin]

The simplest pattern. Each node is independent at the SIP layer; shared state is in MySQL (subscribers, dispatcher, drouting) and replicated via clusterer (usrloc, dialog) for sub-second visibility across nodes.

Use when:

  • 2-5 nodes.
  • Carriers / UAs can be pointed at a VIP / DNS name with multiple A records.
  • You can tolerate a brief (sub-second) gap on node failure.

OpenSIPS modules required:

  • clusterer — node membership, sharing tags, status broadcasting.
  • proto_bin — binary protocol used by clusterer for replication.
  • usrloc with db_mode=2 and replication_cluster=N.
  • dialog with replicate_dialogs_cluster=N (3.6+).

Pattern B — active / standby with a virtual IP

        carrier  --->  10.0.0.100 (VIP)
                              |
                +-------------+-------------+
                |                           |
            opensips-1 (active)       opensips-2 (standby)
                |                           |
                +---> [shared MySQL] <------+
                |                           |
                +---> keepalived heartbeat <+

Use when:

  • You need exactly one IP that's externally addressable (e.g. carrier requires a single peer IP).
  • You can afford keepalived / vrrp tooling.
  • Sub-second failover is non-negotiable.

The OpenSIPS layer is identical to Pattern A; the difference is in the network-level VIP, not OpenSIPS itself. We won't repeat that here — the keepalived configuration is well-documented elsewhere.

The clusterer baseline

Both patterns require clusterer configured on every node. Minimum modparams:

loadmodule "proto_bin.so"
loadmodule "clusterer.so"

modparam("clusterer", "db_url", "mysql://...")
modparam("clusterer", "current_id", 1)   # unique per node: 1, 2, 3, ...
modparam("clusterer", "my_node_id", 1)   # 3.4+ alias

The cluster table in MySQL holds the node list:

cluster_idnode_idurlstatedescription
11bin:10.0.0.10:55551proxy-1
12bin:10.0.0.11:55551proxy-2
13bin:10.0.0.12:55551proxy-3

state accepts only 0 = Disabled or 1 = Enabled (verified against opensips.org/html/docs/modules/3.6.x/clusterer.html). Earlier docs that suggested 2 = unreachable were wrong; the MI rejects status=2.

Routine maintenance — drain one node

Goal: take node-2 offline for a kernel upgrade without dropping live calls.

Step 1 — drain node-2 from the cluster

cluster_drain_node(cluster_id=1, node_id=2, timeout_sec=120)

The tool:

  1. Sets node-2's clusterer status to 0 (Disabled). Peers stop replicating to it and stop selecting it for new traffic.
  2. Polls dialog:active_dialogs every 5 seconds until it hits zero or the timeout fires.
  3. Returns drained: true when zero dialogs remain.

If the count plateaus at a non-zero number for the timeout, you have "sticky" dialogs — calls that nobody is hanging up. Your options:

  • Wait longer. Some calls last hours. Re-run the drain with timeout_sec=3600 and walk away.
  • Hangup remainder. dlg_list() to see them, then either dlg_end_dlg(dialog_id) per call (causes the UAs to see a BYE) or dlg_db_sync() to save state to DB and accept that they'll be abandoned at process exit.

Step 2 — capture a baseline before changes

Always:

snapshot_capture(out_dir="/tmp/ha-snapshots", label="before-upgrade-node2")

Records statistics, processes, memory, dispatcher / lb / cluster state. Use for snapshot_diff after the upgrade.

Step 3 — backup mutable data

db_table_backup(table="dispatcher", out_dir="/tmp/db-backups", label="pre-upgrade")
db_table_backup(table="address",    out_dir="/tmp/db-backups", label="pre-upgrade")

If the upgrade includes a migration that touches data tables, restoration is one tool call away.

Step 4 — perform the upgrade

apt upgrade / yum update / replace the OpenSIPS binary, etc.

Step 5 — restart the OpenSIPS process

Drained = no traffic = it's safe to restart the daemon at this point without dropping calls.

Step 6 — confirm health before re-enabling

health_check()
sip_options_ping(target="127.0.0.1:5060", count=3)
perf_hotspots()

sip_options_ping must return succeeded: 3 with low RTT. If MI is dead, the daemon didn't come up properly — investigate before re-enabling.

Step 7 — bring node-2 back into the cluster

cluster_promote_node(cluster_id=1, node_id=2)

Sets status back to 1 (Enabled). Peers resume traffic; replication catches up automatically.

Step 8 — confirm the cluster sees the node

cluster_sync_check(cluster_id=1)

Returns nodes / capabilities / sharing tags / topology. Check that node-2 appears with status=Enabled and that no unhealthy_nodes are flagged.

Step 9 — diff the post-upgrade state

snapshot_capture(out_dir="/tmp/ha-snapshots", label="after-upgrade-node2")
snapshot_diff(
    before_path="/tmp/ha-snapshots/snapshot-...-before-upgrade-node2.json",
    after_path="/tmp/ha-snapshots/snapshot-...-after-upgrade-node2.json",
)

Tells you exactly which counters changed during the maintenance window. Useful for the change ticket.

Unplanned failover — node-2 just died

Goal: the cluster keeps running while you investigate.

What the cluster does on its own

  • Other nodes' clusterer reports node-2 as unreachable within node_timeout seconds (default 60). Its replication endpoint is dropped.
  • Dispatcher OPTIONS pings to backends still flow from surviving nodes; the failed-node's pings just stop.
  • New REGISTERs land on surviving nodes; usrloc replication from the failed node was already done (if clusterer was set up correctly), so most contacts are still findable.

What you do

  1. Confirm the failure — cluster_sync_check(cluster_id=1) should show node-2 with non-OK status.
  2. Decide whether to forcibly remove it — only if you're certain it won't come back. Otherwise leave it; it will rejoin when the process restarts.
    • To remove: cluster_send_command(cluster_id=1, node_id=2, mi_cmd="clusterer_remove_node").
  3. Snapshot for postmortem — snapshot_capture(..., label="incident-NN").
  4. Investigate the dead node — logs, kernel ring buffer, OOM killer, etc. The MCP can't help here; this is host-level debugging.
  5. Restart and rejoin — once the daemon is up:
    • cluster_promote_node(cluster_id=1, node_id=2)
    • cluster_sync_check to verify
    • snapshot_diff against the pre-incident snapshot

Things that go wrong (and how to spot them)

Dialogs are not replicating

Symptom: an in-flight call is stuck on the failed node — the surviving node has no record of it. Consequence: BYE arrives at surviving node, gets a 404, caller is confused.

Check: dialog modparams. Both replicate_dialogs_cluster=N and db_mode=1 must be set on every node. cluster_sync_check shows the "dialog" capability on each node.

Usrloc inconsistent across nodes

Symptom: UA registered against node-1 cannot receive a call routed through node-2 because node-2 doesn't have the contact in its usrloc.

Check: usrloc.replication_cluster=N on every node, and db_mode=2 (write-through). ul_dump on each node should show roughly equal counts.

Split-brain

Symptom: the network between data centres dropped briefly, both "sides" wrote diverging usrloc / dispatcher state, then merged with inconsistencies.

Mitigation: OpenSIPS' clusterer doesn't have automatic split-brain resolution like Raft. The "fix" is procedural: when you detect a split-brain situation, designate one side as authoritative, drain the other, repopulate from authoritative DB, rejoin.

Replication queue saturation

Symptom: clusterer:bin_send_buffer_full stat climbing fast on peer nodes; lag increases.

Check: proto_bin workers + buffer size. Bump clusterer.replication_pkg_size (3.6+) or scale out the bin protocol listener.

Smoke test for a new cluster

Use this every time you build a new HA pair:

TestToolPass criteria
Node-1 sees node-2cluster_sync_check(cluster_id=1)Both nodes listed, status=OK
Replication activedispatch usrloc REGISTER to node-1, run ul_dump on node-2Contact appears within 1 second
Drain workscluster_drain_node(cluster_id=1, node_id=2, timeout_sec=10)Returns drained: true
Promote workscluster_promote_node(cluster_id=1, node_id=2)Tool returns promoted: true; cluster_sync_check shows OK
Failover with a synthetic callPlace INVITE, kill node-1 process, BYEBYE reaches the same dialog on node-2

A complete first-cluster bring-up walkthrough is on the project's todo list; in the meantime the residential-PBX tutorial scaled to two nodes with the modparams above is a workable starting point.