Monitors (Event-Triggered Watchers)

August 31, 2026 · View on GitHub

A monitor watches a SOURCE, detects a CONDITION change, and fires an ACTION.

A monitor is a routine whose trigger is a watched source instead of a clock. Monitor : routines :: event-triggered : time-triggered — one daemon, one dispatch seam. The monitor owns only the source → condition → action layer; everything below it (spawning the agent, device placement, run history, the execution path) is the routines backbone, reused verbatim. Monitor and scheduler lifecycle remain separate hosted-service controls.

The three-part model

   SOURCE                 CONDITION                ACTION
   (what to watch)        (did it change?)         (what to do)
 ┌───────────────┐      ┌────────────────┐      ┌──────────────────┐
 │ command       │      │ on-change      │      │ run: <agent>     │
 │ poll / http   │ ───► │ match: regex   │ ───► │ routine: <name>  │
 │ file / device │      │ every          │      │ notify: telegram │
 │ ws / webhook  │      │ + dedupe key   │      │ webhook-out: url │
 └───────────────┘      │ + state.json   │      │ ({event} → prompt)│
                        └────────────────┘      └──────────────────┘

Architecture

~/.agents/
  monitors/
    ci-red.yml                         # Monitor config (YAML)
    cert-issued.yml
  .history/monitors/
    ci-red/
      state.json                       # last-seen value/hash + fire bookkeeping (written only on fire/baseline)
      liveness.json                    # per-poll heartbeat: lastCheckedAt, checkCount, lastError (written EVERY poll)
      fires/<id>/event.json            # fire history

Each monitor is a YAML file in ~/.agents/monitors/ (the user layer) or, for a built-in shipped with the CLI, ~/.agents/.system/monitors/ (the system layer, from gh:phnx-labs/.agents-system). listMonitors()/readMonitor() union the two — the user layer shadows a system built-in of the same name, exactly like routines' project/user/system resolution. A system built-in is enabled by default, like every other system-layer resource (rules, hooks, commands, skills): it runs on every install unless you shadow it with enabled: false (via agents monitors pause, which materializes a user copy — pause/edit/delete always write the user dir; the system mirror is pull-only, and there is no enable/disable verb). Enabled-by-default is not permission to fire on every daemon: an unpinned system built-in is placed on a single owner in code (requiresSingleOwner / monitorRunsOnThisDevice) — treated as shared-input unless it sets sharedInput: false, so a built-in that polls a fleet-shared queue can never fan out across the fleet and double-fire, even if its shipped YAML forgot a device: pin (SING-9a). Each monitor carries its read-time scope (user/system), so agents monitors list tags a built-in (built-in) and --json emits scope + builtin. The shared background daemon hosts a monitor engine beside the cron scheduler. agents routines stop disables only the scheduler and does not stop monitor polling; use agents daemon services disable monitors for that service. On each tick it evaluates every enabled, device-owned monitor that is due, applies the condition through the native state-diff store, and on a fire dispatches the action through the exact executeJobDetached path cron and webhook fires use.

Built-in: `pr-merge-on-green`

On by default. Polls every 5 minutes for this user's open PRs that are CI-green and non-author-approved, then dispatches claude to rebase-merge them. Because it polls a fleet-shared queue (--author @me is identical on every box), exactly one daemon must fire it or every box would race to merge the same PR (SING-9). This is enforced in code, not just by the shipped YAML: an unpinned scope: system built-in is treated as shared-input and fires only on the resolved owner (interactive.host, else the sole box on a single-device fleet, else nowhere), so it stays single-executor even if the YAML omits the pin (SING-9a). The shipped YAML SHOULD still pin a device: owner (or set sharedInput: true) to document the intent and choose the specific owner box.

The built-in YAML lives in gh:phnx-labs/.agents-system (monitors/pr-merge-on-green.yml) and polls monitors/pr-merge-on-green.sh — gh search prs plus gh pr view --repo, verdict via pr-verdict.py (the same file merge-guard.sh calls). Companion: phnx-labs/.agents-system#347.

This CLI also ships a hidden helper with the same verdict rules (hasApproveVerdict / isCiGreen match that python), listing registered project slugs instead of a global search:

agents _internal mergeable-prs    # owner/repo#n, or empty

agents monitors test pr-merge-on-green    # dry-run: observation + would-fire
agents monitors pause pr-merge-on-green    # opt OUT — materializes a user copy with enabled: false

The built-in is on by default; there is no enable verb (pause/resume are the toggle, and they write the user dir only). A stale user copy from an older, never-shipped enable step still has the broken gh pr list command (no --repo, reviewDecision == APPROVED only); delete it to fall back to the current shipped poll: agents monitors remove pr-merge-on-green.

The one genuinely new piece: native state-diff

Routines persist per-run metadata but have no last-observed-value store. Monitors add one (~/.agents/.history/monitors/<name>/state.json) — this is what replaces the hand-rolled markdown memory files ad-hoc watchers used to need.

Monitor Config

# ~/.agents/monitors/ci-red.yml
name: ci-red
enabled: true
source:
  type: poll                 # command | poll | poll-http | file | device | ws | webhook
  command: gh pr checks 1119 --json name,bucket
  interval: 30s              # seconds supported (unlike routines' minute-granularity)
condition:
  mode: match                # on-change (default) | match | every
  match: fail                # required for match mode (a regex)
  dedupeKey: build (\d+)     # optional: first match is the "same event" signature
action:
  type: run                  # run | routine | notify | webhook-out
  agent: claude
  prompt: "CI failed on #1119: {event}. Diagnose and fix."
  mode: auto
device: yosemite-s0          # OWNER — the single machine that evaluates + fires (exactly-once)
rateLimit:                   # firehose guard — auto-pause if exceeded
  max: 5
  per: 1m

Definition vs. running state

Two different things wear the word "monitor", and they live in different places on purpose:

	Path	Rides the repo to every box?
Built-in definition — shipped watcher (e.g. `pr-merge-on-green`)	`~/.agents/.system/monitors/<name>.yml`	yes — via the npm-shipped system repo, available on every install
User definition — an agent's per-work-item watcher	`~/.agents/monitors/<name>.yml`	no — deliberately per-machine. Agents create these per PR/issue; syncing would push every ephemeral watcher onto every box (the accumulation the double-trigger guard exists to stop). Fleet visibility comes from `agents monitors list` fanning out, not from git-syncing the files
Running state — last-seen value, fire history, rate-limit counters	`~/.agents/.history/monitors/<name>/state.json` + `fires/<id>/`	no — it is per-machine and regenerable

Nothing but the definition is ever written into monitors/: the only writers of that directory are the monitor file's read, write, and delete. Runtime lives under .history/, which is excluded, so the split needs no extra rules. Because user watchers stay local, "which box is watching what" is answered by the list fan-out (below), not by pulling a synced dir.

The double-trigger guard

A monitor's NAME is not its identity. Two watchers polling the same source on the same interval and firing the same action are one trigger fired twice, whatever they are called — and writeMonitor overwrites by name, so nothing used to notice. One real box accumulated open-pr-watch, pr-ci-fail, three stale pr2222-* watchers and an agent-added lander, all polling the same PR queue, added without a single warning.

The check runs across the fleet, not just this box — reusing the same cross-machine fan-out sessions --active uses. Two agents on two machines creating a watcher for the same work item is the case a local check cannot see. When a peer is unreachable the command says which ones it could not consult, rather than treating that as "no duplicate".

agents monitors add refuses two collisions:

Same name — adding would overwrite an existing monitor.
Same behavior — an existing monitor (user or built-in) already watches that source and fires that action, under any name.

Identity is a fingerprint over the source, condition, and action. Name, description, and enabled are excluded — a duplicate under a new name is exactly the case being caught, and a paused duplicate is still a duplicate. Placement (device/devices/runOn) is excluded too: placement is who executes, not what runs, and hashing it would let the same watcher be re-added N times by varying only the owner.

Pass --force when the duplication is deliberate.

Commands

# Create (auto-starts the daemon; then WAITS for the engine's first poll and
# reports whether it was actually picked up — config acceptance is not "running")
agents monitors add ci-red \
  --poll 'gh pr checks 1119 --json name,bucket' 30s --match fail \
  --run claude --prompt 'CI failed on #1119: {event}. Diagnose and fix.' \
  --device yosemite-s0

# The SSL watcher, reduced to config (replaces a 70-line prompt)
agents monitors add cert-issued \
  --poll-http 'https://secure.ssl.com/team/.../co-ec1l5dgjofa' 8h \
  --match issued --notify telegram --device zion

agents monitors list                  # every monitor on every fleet device, tagged (built-in) + owning box; source, action, owner, liveness (checked Nx / never polled / STALLED / fired)
agents monitors list --local          # this device only — skip the fleet fan-out
agents monitors view <name>           # full config + liveness + current watched-state + recent fires
agents monitors test <name>           # DRY-RUN: evaluate once, print event + would-fire (no action)
agents monitors edit <name>           # $EDITOR on the YAML
agents monitors logs <name>           # action run logs (run actions; reuses routines run history)
agents monitors runs <name>           # fire history
agents monitors pause / resume <name> # disable / re-enable
agents monitors device <name> --set X # (re)pin the owner device; --clear to unrestrict
agents monitors remove <name>

Sources (`add` flags — exactly one)

Flag	Source	Observation
`--watch '<cmd>'`	command	the command's stdout
`--watch-pid <pid>`	command	`running` / `exited` / `notyetspawned`, polled by the engine — see below
`--poll '<cmd>' <interval>`	poll	stdout, re-run every interval
`--poll-http <url> <interval>`	poll-http	`<status>\n<body>` every interval
`--watch-file <path>`	file	file content (or dir listing) + mtime
`--watch-device <name>`	device	fleet device reachability + headroom bucket
`--ws <url>`	ws	each WebSocket frame (push)
`--on <src:event>`	webhook	a signed github/linear delivery (push)

--watch-pid <pid> is the reliable alternative to a harness's own backgrounded-shell exit hook (PHNX-3023): a "will re-invoke me" watch loop (gh pr checks --watch, a long sleep, a tick poll) never itself exits, so a harness that only re-invokes on process exit never wakes for it — the ticket's "5 shells still running" that never notified anyone. --watch-pid instead has the daemon's own poll loop check the pid's liveness, independent of the session that armed it, and defaults its condition to fire on exit (unlike a plain --watch, which defaults to on-change). It fails loud at arm time — refuses to create the monitor — when the pid is already dead, rather than silently arming a watcher pointed at a corpse. Its --run/--notify action still spawns a new headless agent conversation or sends a real notification; it does not resume the exact session that armed it (see --run below) — the fix is "you get reliably woken", not "the same conversation magically continues".

--force arms a watch on a pid that isn't alive yet (the caller's own concurrent step is about to spawn it). The poll then reports notyetspawned, never exited, until it has observed the pid running at least once (a per-monitor marker file records that transition) — otherwise the very first poll would already read "not alive" as "exited", fire a false notification immediately, and permanently poison the on-change dedupe baseline so the real later exit never fires again.

Conditions (how an observation becomes a fire)

--on-change (default) — fire when the observation differs from last-seen. The first observation establishes a silent baseline; a later change fires.
--match '<regex>' — fire when the observation matches; de-duped so it fires once per distinct matched token (silent while the match is unchanged).
--every — fire on every observation (no dedupe). Rate-limit this.
--dedupe-key '<regex>' — the first match is the "same event" signature (default: the full observation).

Actions (exactly one; the event is injected as `{event}`)

--run <agent> --prompt '…' — spawn a new headless agent conversation; it does not resume or re-invoke the session that created the monitor. It shares --mode/--effort/ --action-timeout with routines and is dispatched through executeJobDetached. Takes a native harness id or a custom harness name (agents harness list); a custom harness is delegated to agents run <name> and pins its own host version and auth. The sandboxed child inherits this host's GitHub CLI auth (GH_CONFIG_DIR / ~/.config/gh) the same way interactive agents run does — a --run that shells out to gh is not a hollow success when the daemon user is already logged in (RUSH-2860; see routines.md §Sandbox Isolation). Native Claude/Codex actions use balanced rotation by default. Claude verifies the selected version with its own auth status before spawn and skips stale identity-only accounts. Claude launches with the same operator HOME plus the balanced version's CLAUDE_CONFIG_DIR as an ordinary agents run, so the native login and portable ~/... hooks resolve through the same paths.
--routine <name> — fire an existing routine (attach a monitor to a routine).
--notify [channel] — notify the owner through the canonical owner-delivery seam. Without an override, every addressable channel in humans.yaml's owner.policy.normal is attempted; [channel] selects one channel for this dispatch. A channel with no registered provider fails that one dispatch (ok: false, the reason logged) and leaves the daemon evaluating every other monitor — the daemon never exits on a bad channel name.
--webhook-out <url> — POST the event JSON.

Placement (pin-to-one)

--device <name> — the OWNER: the single machine whose daemon evaluates the source and fires. This is the exactly-once guarantee for v1 (no distributed lock). If the owner is down, the monitor is down. A device/fleet is itself a valid watch source (--watch-device).
--devices <list> — allowlist (advanced): each listed device fires independently, like routines' devices.
--run-on <host> — execute the ACTION on a different machine over SSH, distinct from the owner that fires it. With no owner pin, it pins the owner to this machine to avoid duplicate fires across the fleet.
--cwd <path> — working directory for a --run action, home-relative or ~/…. Defaults to the execution target's home, which stays portable across a --run-on hop. A monitor owns no project, so without this the run would be blocked at readiness with execution_context_missing (RUSH-2681).

The `test` dry-run (the DX centerpiece)

agents monitors test <name> evaluates the source once and prints the emitted event plus the would-fire decision — without acting and without writing state:

Dry-run: ci-red

  poll: gh pr checks 1119 --json name,bucket @30s  ·  [match]  ·  run claude

Observation
  ... name ... bucket=fail ...
  meta: {"exitCode":0}

Would fire: yes

Emitted event
  summary: fail
  → would run claude

(dry run — no action taken, no state written)

Liveness & health (is it actually polling?)

A monitor can be enabled, owned by this box, and listed as on while the engine never touches it — and until it fires nothing on disk proved otherwise, because change-detection state.json is written only on a fire or a baseline. A --match monitor that polled steadily but matched nothing therefore showed state: null, indistinguishable from a monitor the engine never ran (RUSH-2485).

The engine now writes a liveness heartbeat (liveness.json) on every poll — fire or not, match or not — so "never checked" is visibly distinct from "checked N times, not matching":

agents monitors list
  ci-red      on  poll: gh pr checks 1119 … @30s
                  [match] → run claude   owner: yosemite-s0   checked 42x · last 12 sec ago · no match yet
  cert-issued on  poll-http: … @8h
                  [match] → notify       owner: zion          never polled          <- yellow: engine hasn't touched it
  stale-one   on  poll: … @60s
                  [on-change] → notify   owner: yosemite-s0   STALLED — last poll 2 hours ago   <- red

never polled (yellow) — the engine has no heartbeat for it. If it persists after a few seconds, inspect agents daemon services and agents daemon status.
checked Nx · last <ago> · no match yet — alive and polling, condition just hasn't matched. This is the state that used to look dead.
STALLED — last poll <ago> (red) — an enabled, locally-owned monitor whose last poll is more than three intervals behind. The engine has stopped checking it (dead engine, wedged source) even though it's still marked on.
checked Nx · error: <msg> (red) — the source is erroring every poll.

agents monitors view <name> shows the same under a Liveness block (last checked, checks, last error), and both commands expose it in --json as lastCheckedAt, checkCount, lastError, consecutiveErrors, and stalled.

Drought escalation. After 5 consecutive failed checks — a source that errors every poll, or an action that fails every fire — the engine notifies the owner once that the monitor is doing nothing (the streak clears, and can escalate again, on the first good check).

Fleet / device semantics (pin-to-one, v1)

--device <name> = the owner. Only that machine's daemon evaluates and fires. Everywhere else the monitor is inert (monitorRunsOnThisDevice returns false).
~/.agents/monitors/ rides the user repo, so a monitor syncs to every machine; the owner pin is what makes it fire exactly once.
Remote management (--device <device> on a monitors subcommand, like routines) is a follow-up: the top-level --device flag names the OWNER here, which collides with the --device routing flag, so monitors interpret --device locally rather than routing.
The owner pin is the ONLY ownership gate a monitor action passes. A run action synthesizes a one-off job and hands it to the routines dispatch seam (executeJobDetached), which normally also checks the per-device ROUTINES activation manifest (~/.agents/devices/<machine>/agents.yaml → routines:). A monitor is not a routine and can never be a member of that list, so the synthesized job carries dispatchedBy: 'monitor' and jobRunsOnThisDevice (lib/routines.ts) skips the manifest for it — without that marker every monitor action was refused as wrong_owner with an empty allowlist and never ran (RUSH-2681). A routine action fires a REAL routine, which keeps its activation gate: a routine defined but not activated on this device is still refused. Monitor names are never written into the routines manifest.

Hygiene

Rate-limit / firehose guard — rateLimit: { max, per } auto-pauses a monitor that fires more than max times per per.
Notify-on-change discipline — on-change is the default; silent on no-change.
Coverage lint on add — warns when a --match names only a success token (e.g. issued), since "silence is not success" if the source breaks.

v1 scope

The engine evaluates the poll model: command, poll, poll-http, file, and device sources. Push sources (ws, webhook) are accepted and validated but deliver through a persistent subscription / the webhook receiver, wired in a follow-up. Distributed single-owner lease + failover (true HA across --devices) and monitor→monitor chaining are also out of scope for v1.

Key Functions

Function	File	Purpose
`validateMonitor()`	lib/monitors/config.ts	Hand-rolled config validation
`writeMonitor()` / `readMonitor()`	lib/monitors/config.ts	Persist monitor config
`monitorRunsOnThisDevice()`	lib/monitors/config.ts	Owner-device eligibility gate
`hasChanged()` / `writeState()`	lib/monitors/state.ts	Native state-diff store (fire/baseline only)
`recordCheck()` / `readLiveness()`	lib/monitors/state.ts	Per-poll liveness heartbeat (every poll)
`MonitorEngine.runMonitor()`	lib/monitors/engine.ts	One poll: evaluate → decide → fire → record heartbeat
`shouldEscalateDrought()`	lib/monitors/engine.ts	Drought predicate: notify owner after N failed checks
`evaluateSource()`	lib/monitors/sources/index.ts	Source-type → evaluator
`decideFire()`	lib/monitors/engine.ts	Apply the condition to an observation
`dispatchAction()`	lib/monitors/dispatch.ts	Fire the action via executeJobDetached / notify / POST
`MonitorEngine`	lib/monitors/engine.ts	The tick/evaluate loop inside the daemon