Post-deploy smoke test

April 28, 2026 · View on GitHub

The 90-second checklist to run after every deploy. Catches the boring 30 % of bugs that would otherwise be discovered at 03:14 by an oncall engineer who didn't push the change. None of these tests require user traffic; all of them use the project's own MCP tools.

The checklist

#	Test	Tool	Pass criteria
1	Process is alive and answering MI	`health_check()`	Returns OK
2	SIP listener is bound and receiving	`sip_options_ping(target="127.0.0.1:5060")`	`succeeded >= 1`
3	Memory headroom OK	`perf_memory_report()`	shmem usage < 70 %
4	Process count matches expectation	`perf_process_breakdown()`	`total_processes` == `children + tcp_workers + extras`
5	Dispatcher destinations reachable	`dispatcher_list_db()`	Every dest with state=Active
6	TLS certs not expiring soon	`tls_check_all_expiry(warn_days=30)`	`expiring=[]`
7	Anti-flood firing only on noise	`pike_list_blocked()`	No legitimate-looking IPs
8	Auth path works	synthetic REGISTER + check `registrar_stats()`	`accepted_regs` increments
9	Routing path works	synthetic INVITE via `sip_generate_sipp_scenario` + check `dlg_list()`	Dialog appears
10	Cluster healthy (if applicable)	`cluster_sync_check(cluster_id=1)`	`healthy: true`

If all 10 pass, the deploy is probably good. If 1-3 fail, roll back immediately with cfg_rollback.

Test 1 — `health_check`

health_check()

Returns {"ok": true, ...} if MI is reachable and responding to which. If MI is dead, every other tool in the checklist will fail too; fix this first.

Test 2 — `sip_options_ping`

sip_options_ping(target="127.0.0.1:5060", count=3, timeout_sec=2.0)

Pure-Python UDP probe. Validates that the SIP listener is bound, responding, and the process is running normally. If you get "timeout" for all 3, the daemon may have started but failed to bind (port collision, missing capability, address-not-available).

If the proxy listens on multiple sockets (udp:5060, tcp:5060, tls:5061), test each:

sip_options_ping(target="127.0.0.1:5060")
sip_options_ping(target="127.0.0.1:5061", allow_public=False)  # tls

(TCP / TLS probes via this tool aren't yet supported; for those use openssl s_client or the tls_connections() MI call.)

Test 3 — memory

perf_memory_report()

Look at mem.shmem_used versus mem.shmem_total. Healthy production deployment runs at 30-60 % shmem utilisation. > 70 % is a warning;

85 % is a "tune it before this becomes the incident."

If you used cfg_tune to size the deployment and dlg_size_bytes is unset, the calculation may have under-allocated; re-run with a realistic per-dialog size.

Test 4 — processes

perf_process_breakdown()

Group by type; verify the count matches the global config. A common silent failure: tcp_workers=8 was set but the daemon ignored it because of a config syntax error elsewhere, falling back to default. The process count exposes that.

Test 5 — dispatcher / load_balancer

dispatcher_list_db()

Every destination should show state=Active. After deploy, give the ping mechanism one full ds_ping_interval (default 10s) to mark destinations active before judging.

If lb_list() is meaningful for your scenario, run that too; same criterion.

Test 6 — TLS

tls_check_all_expiry(warn_days=30)

If anything is expiring within 30 days, deal with it now (rotate + tls_reload), not when the alert fires. See tls-from-scratch.md for the rotation procedure.

Test 7 — anti-flood

pike_list_blocked()

After deploy, the blocked list should be either empty or only contain hosts you recognise as scanner traffic. If a legitimate carrier IP appears, your pike thresholds are too tight for the change you just shipped. Tune reqs_density_per_unit upward or add an exception.

Test 8 — synthetic REGISTER

If the scenario serves UAs (residential PBX, registrar, multi-tenant, WebRTC, IMS S-CSCF, asterisk-trunking, MSRP gateway, RADIUS auth):

Provision a test user once:

gen_test_subscribers(domain="<your-domain>", count=1, dry_run=False)

Creates test0001 with password secret0001.

Generate a SIPp REGISTER scenario:

sip_generate_sipp_scenario(
    kind="register",
    target="<host>:5060",
    from_user="test0001"
)

Run the SIPp command on a separate host. Should see two REGISTERs (challenge + reply) and a 200 OK.

Verify on the proxy:

ul_show_contacts(table="location", aor="test0001@<your-domain>")
registrar_stats()

Contact present + accepted_regs incremented = pass.

Cleanup the test user when done:

cleanup_test_data(domain="<your-domain>", confirm=True)

Test 9 — synthetic INVITE

Same shape but for call setup. If the scenario routes calls through to a back-end (Asterisk, FreeSwitch, carrier trunk):

Provision two test users, or run a UAS-side scenario you control.

Generate a SIPp INVITE scenario:

sip_generate_sipp_scenario(
    kind="invite_uas_ringing",
    target="<host>:5060",
    from_user="test0001",
    to_user="test0002",
    calls_per_sec=1,
    total_calls=5
)

Run SIPp. While it runs, watch the proxy:

dlg_list()
perf_sample_window(duration_sec=10, sample_count=5)

Pass criterion: dialogs appear, dialog:active_dialogs ramps up then back down, tm:received_replies shows mostly 200 OKs.

Test 10 — cluster (multi-node only)

cluster_sync_check(cluster_id=1)

healthy: true and every expected node listed with status=OK. If any node shows a non-OK status, do not consider the deploy successful — the cluster is in a degraded state and a single more failure may take the service down. See ha-failover-playbook.md.

Snapshot for the change ticket

After all 10 pass, capture a baseline:

snapshot_capture(out_dir="/var/log/opensips-snapshots", label="post-deploy-vN")

Two reasons:

Evidence in the change ticket that the deploy was healthy at sign-off.
Baseline for the next deploy's snapshot_diff so you can see what changed between releases.

What to do when something fails

Failed test	Most likely cause	First action
1 health	Daemon didn't start or MI is misconfigured	Check `docker_logs` / journalctl; fix config; redeploy
2 sip_options_ping	Socket not bound	Check `listen=` directives + firewall
3 memory	shm under-allocated	`cfg_tune` with realistic `dlg_size_bytes`, restart
4 process count	Config syntax error eaten silently	`cfg_validate` the live config; check log for "WARN: ignoring"
5 dispatcher	Backends really down OR ping interval not yet elapsed	Wait one cycle; if persists, check backend & firewall
6 TLS	Cert expiring	Rotate now; `tls_reload`
7 pike	Threshold too tight	Tune `reqs_density_per_unit`
8 REGISTER	DB connection broken / auth misconfig / NAT broken	Check `acc:db_` + `auth:` log lines + sngrep the REGISTER
9 INVITE	Routing rule didn't match / dispatcher empty	`do_routing` log + `dispatcher_list_db`
10 cluster	clusterer / proto_bin not loaded on all nodes	`cluster_sync_check` per node; align modparams

If 1, 2, or 3 fail and you can't fix in <2 minutes, roll back:

cfg_rollback(
    backup_path="/etc/opensips/opensips.cfg.bak-<previous-deploy-ts>",
    target_path="/etc/opensips/opensips.cfg",
    validate_after=True
)

Then notify the change-ticket owner. A failed deploy that's rolled back in 5 minutes is recoverable; a failed deploy left to bleed for 30 minutes while you debug is an incident.

Embedding this in CI / CD

If your deploy pipeline can run MCP tool calls (most can via a small shim), make tests 1, 2, 3, 5 mandatory pre-flight gates. Tests 8, 9 need test fixtures and are best run as a separate post-deploy stage that can fail loudly without rolling back the deploy itself.

The full battery takes ~90 seconds; the gating tests take ~10. Both are cheap insurance against deploys that "looked fine" until users noticed.