Notifications (notify)

July 15, 2026 · View on GitHub

How to configure outbound alerts for pipeline start, success, error, and stalled events, and how to test channels without running a pipeline.

kzero sends HTTP POSTs in run.mode: live only (down, up, reset). dry-run does not fire pipeline notifications. Use kzero notify test to verify wiring at any time.

Quick test (no cluster mutations)

Enable at least one channel in your profile (or via KZERO_* env vars below).
Run:

kzero notify test --config /path/to/kzero.yaml

Expect exit 0 and stdout: notify test: sent event "notify.test" to enabled channel(s).

Preview real event formatting:

kzero notify test -c /path/to/kzero.yaml --event pipeline.start
kzero notify test -c /path/to/kzero.yaml --event pipeline.success
kzero notify test -c /path/to/kzero.yaml --event pipeline.error
kzero notify test -c /path/to/kzero.yaml --event pipeline.stalled

pipeline.error in test mode includes sample failed_step and error fields so you can check Slack/Teams layout before a real failure. Invalid --event values (for example bare error) fail with exit 1 (config).

YAML schema

notify:
  on_error: true          # default true when any channel is enabled
  require_delivery: false # when true, failed pipeline.error / pipeline.stalled POST fails the run (exit 4)
  slack:
    enabled: false
    webhook_url: ""
  discord:
    enabled: false
    webhook_url: ""
  teams:
    enabled: false
    webhook_url: ""
  pagerduty:
    enabled: false
    routing_key: ""       # Events API v2 integration key
  webhook:
    enabled: false
    url: ""
    headers: {}           # optional extra HTTP headers

Annotated reference: configs/kzero.sample.yml.

Events

Event	When (live pipelines)
`pipeline.start`	After the `Kubernetes target:` block, before the first hook or step
`pipeline.success`	After successful `post-down` / `post-up` / full `reset`
`pipeline.error`	On fail-fast, before `hooks.on-error` (includes step ref when available)
`pipeline.stalled`	When `run.api_watchdog` trips (API unreachable mid-wait)
`notify.test`	Only via `kzero notify test` (not sent by pipelines)

Set notify.on_error: false to suppress pipeline.error while keeping start/success.

Delivery failures and exit codes

Failed outbound POSTs always log [ERR] with redacted URLs (v0.8.0+). Process exit depends on command and require_delivery:

Situation	Exit	Notes
`kzero notify test` POST fails	4	Always (NotifyFailed)
`kzero notify test` no channel / bad `--event` / bad config	1	ConfigError
Live `pipeline.error` / `pipeline.stalled` POST fails, `require_delivery: false` (default)	unchanged	Pipeline exit stays from step/watchdog (usually 3); alert loss only in logs
Same POST fails, `require_delivery: true`	4	Fail-fast on alert delivery (`KZERO_NOTIFY_REQUIRE_DELIVERY`)
`pipeline.start` / `pipeline.success` POST fails	unchanged	Logged `[ERR]`; does not use `require_delivery`

Soft default: leave require_delivery: false when cron can retry and local logs are enough. Hard: set true when on-call must receive the error/stalled alert or the run should count as failed.

Full taxonomy: SPECIFICATIONS.md §5 (exit codes). Mitigations: pipeline-network-loss.md.

Channel examples

Generic webhook (JSON body)

Best for custom integrations, n8n, or internal receivers. The webhook channel sends the full structured payload:

{
  "event": "pipeline.success",
  "command": "reset",
  "mode": "live",
  "client_id": "ops-team-a",
  "cluster_name": "example-cluster",
  "started_at": "2026-06-05T12:00:00Z",
  "duration": "18m32s"
}

notify:
  webhook:
    enabled: true
    url: "https://hooks.example.com/kzero"
    headers:
      Authorization: "Bearer ${TOKEN}"   # expand in your secret manager / env before deploy

Slack incoming webhook

notify:
  slack:
    enabled: true
    webhook_url: "https://hooks.slack.com/services/T…/B…/…"

Slack receives a colored attachment with a vertical bar per event: blue (started), green (completed), red (error), yellow (test). The title follows kzero {action}; the attachment footer is kzero vX.Y.Z from the running binary (build metadata). Bullet fields include Cluster, Client, Time, Context, User, Mode, and Duration on success. Slack also adds its own incoming-webhook attribution below that footer.

Discord webhook

notify:
  discord:
    enabled: true
    webhook_url: "https://discord.com/api/webhooks/…"

Microsoft Teams workflow / connector URL

notify:
  teams:
    enabled: true
    webhook_url: "https://….webhook.office.com/…"

PagerDuty Events API v2

notify:
  pagerduty:
    enabled: true
    routing_key: "your-integration-key"

Errors trigger with error severity; start/success use info.

Environment overrides (`KZERO_*`)

Viper prefix KZERO_ with dots and dashes mapped to underscores (same as other config keys):

Env var	YAML key
`KZERO_NOTIFY_ON_ERROR`	`notify.on_error`
`KZERO_NOTIFY_SLACK_ENABLED`	`notify.slack.enabled`
`KZERO_NOTIFY_SLACK_WEBHOOK_URL`	`notify.slack.webhook_url`
`KZERO_NOTIFY_DISCORD_ENABLED`	`notify.discord.enabled`
`KZERO_NOTIFY_DISCORD_WEBHOOK_URL`	`notify.discord.webhook_url`
`KZERO_NOTIFY_TEAMS_ENABLED`	`notify.teams.enabled`
`KZERO_NOTIFY_TEAMS_WEBHOOK_URL`	`notify.teams.webhook_url`
`KZERO_NOTIFY_PAGERDUTY_ENABLED`	`notify.pagerduty.enabled`
`KZERO_NOTIFY_PAGERDUTY_ROUTING_KEY`	`notify.pagerduty.routing_key`
`KZERO_NOTIFY_WEBHOOK_ENABLED`	`notify.webhook.enabled`
`KZERO_NOTIFY_WEBHOOK_URL`	`notify.webhook.url`

Example: test Slack without editing YAML on disk:

export KZERO_NOTIFY_SLACK_ENABLED=true
export KZERO_NOTIFY_SLACK_WEBHOOK_URL="https://hooks.slack.com/services/…"
kzero notify test --config ./kzero.yaml

Never commit webhook URLs or routing keys. Use env vars, CI secrets, or a secret manager.

Live pipeline workflow

Recommended order for a new profile:

# 1. Plan only
kzero analyze --config ./kzero.yaml

# 2. Verify notify channels (no API mutations)
kzero notify test --config ./kzero.yaml
kzero notify test -c ./kzero.yaml --event pipeline.error

# 3. Dry-run pipeline (no notify, no mutations)
kzero down --config ./kzero.yaml   # run.mode: dry-run

# 4. Live run (notify fires on start / success / error)
export KZERO_RUN_MODE=live
kzero reset --config ./kzero.yaml

See also kzero-selfhosted/run/docs/automation-and-pipelines.md for CI/cron patterns.

Operator audit fields in payloads

When configured, notifications include client_id from client.id and cluster metadata from cluster.name. The Kubernetes target: block also prints os_user / os_uid (hooks receive KZERO_OS_USER / KZERO_OS_UID). See SPECIFICATIONS.md.

Troubleshooting

Symptom	Check
No message on `down` / `up`	`run.mode` must be `live`; `dry-run` skips pipeline notify
`notify test: no notify channel enabled`	At least one `.enabled: true` or matching `KZERO_NOTIFY__ENABLED`
HTTP 4xx from webhook	URL, auth headers, and firewall egress
Duplicate error alerts	`pipeline.error` fires before `on-error` hook; hook may send its own alert
Secrets in logs	kzero redacts webhook URLs in notify error messages; keep URLs out of committed YAML
Live reset ran but no Slack after API outage	Enable `run.api_watchdog` and confirm `kzero notify test --event pipeline.stalled`. Failed POSTs should log `[ERR]` since v0.8.0 — see pipeline-network-loss.md. Total bastion network loss still requires local logs + external watchdog.
Want fail when error alert cannot be delivered	Set `notify.require_delivery: true` (exit 4 on failed `pipeline.error` / `pipeline.stalled` POST)

Contract details: SPECIFICATIONS.md → notify and kzero notify test.

Production live reset: run kzero notify test --event pipeline.error (and pipeline.stalled if the watchdog is enabled) before destructive work; tee stdout to a log file; consider an external watchdog — pipeline-network-loss.md.