agents-cli (the CLI)

August 31, 2026 · View on GitHub

@phnx-labs/agents-cli — the agents / ag CLI for managing AI coding-agent versions, config, sessions, and cloud dispatch (Claude, Codex, Cursor, OpenCode, OpenClaw, Grok, Droid, …).

New agent? Start with docs/README.md. It links the compact architecture, concepts, execution, and subsystem decision documents.

This is the internal architecture map. The user-facing feature tour is README.md (pin versions, run, sessions, hosts, teams, workflows, plugins, browser, secrets, routines, pty). This file covers the design choices, module map, build, and release mechanics the README does not.

Phoenix Labs · FSL-1.1-Apache-2.0. Repo-wide policy (conventions, code review, security) lives in the root AGENTS.md.

agents setup is the re-runnable onboarding hub. It reports live readiness for core, browser, computer, secrets, accounts, fleet, share, watchdog, and preferences, then delegates each selected phase to its existing agents setup <capability> wizard. agents setup status --json is the non-interactive view of the same probes.

agents reminders lists personal operating reminders kept in ~/.agents/reminders/reminders.yaml (each a short/full pair). They surface succinctly in the Claude statusline — one per session, chosen deterministically from the session id (pickReminderForSession in src/lib/reminders.ts) so concurrent agents each show a different one and it stays stable within a session. Presence of the file with at least one entry is the opt-in; there is no flag. The render is Claude-only (renderClaudeStatusLine appends the dimmed part after the usage windows) because the statusline is a Claude Code surface; other harnesses expose no equivalent always-visible line. A malformed file is swallowed by the statusline (a broken prompt is worse than a missing line) but surfaced by agents reminders. The file syncs across the fleet via agents repo push/pull.

agents artifacts share publishes an artifact under a single Phoenix ID identity (Google-only device-code OAuth via agents auth login, src/lib/identity/client.ts PhoenixSession — there is no separate GetRush account and no Supabase). The URL namespace is the signed-in email's local-part (handleFromEmail, src/lib/share/backend.ts). A publish stamps one of five visibility levels (ShareVisibility, src/lib/share/publish.ts): public (default, gallery + OG card), unlisted (= --private; capability URL, noindex, gallery-hidden — obscurity, NOT read-auth, so the CLI warns loudly), private (= --protected; token-gated read auth, PHNX-3654 — the Worker serves it only to a request whose ?k=/Bearer key hashes to the stored viewer-token-hash, else 404; the raw key rides only in the emitted …?k=<token> URL, works for BYO too), me (owner-only, Phoenix-gated), and org (anyone at the sharer's email domain, Phoenix-gated). Both unlisted and private force a 64-bit random slug tail so the capability URL can't be guessed from the title. org is refused on a public-inbox domain (PUBLIC_INBOX_DOMAINS in src/lib/share/worker-template.ts), so it needs a workspace-domain Google account, never a personal gmail.com. Managed publishes are metered per user against a free-tier storage quota, object limit, per-file size cap, and publish rate limit (a CAS ledger at __usage/<owner>; enforcement keys on the real request body, fails loud with 413/429), while a BYO WRITE_TOKEN publish to the operator's own bucket skips all four (PHNX-3542). The full model is docs/share.md; the publication boundary is docs/observability.md.

The managed OG renderer is a multipart module-Worker deployment: JavaScript and font bytes live in the main bundle, while Yoga and resvg are uploaded as compiled WASM modules (renderWorkerBundledeployWorker). Do not inline WASM with esbuild's binary loader: Node permits the resulting runtime compilation but workerd rejects it. The real-workerd render in worker-template.integration.test.ts is the contract test for this boundary.

agents artifacts share list mirrors the public gallery by default. Use --scope unlisted|private|me|org or --all (alias for --scope all) to list the authenticated owner's hidden pages; the CLI forwards the owner's bearer and a scope=mine hint to the Worker's JSON listing route, which includes hidden pages only after verifying that the bearer owns the requested namespace. The filter is --scope (not --visibility) because the parent share <file> command owns --visibility. agents artifacts share visibility <target> <level> re-scopes an already-published page in place through the same PATCH metadata-edit route as share edit: the slug/URL and body are preserved, so no revision is created; the result flag is --visibility-json (the same ancestor-collision rename as --scope/--for-user). agents artifacts share open <target> opens the owner's own page signed in, so the served page's inline visibility chip is a live control and not a static cue (PHNX-3370): the served chip is interactive only when isOwner (handleFromEmail(identity.email) === firstSeg), and a browser gets that identity from the __share cookie the Worker sets by redeeming a ?phoenix_ticket=. Nothing minted that ticket before, so share open has the Worker mint a short-lived, single-use, self-signed one at POST /__ticket (authenticated by the caller's Phoenix bearer; signed with the cookie's HMAC secret but domain-separated so neither can be replayed as the other — signSelfTicket/verifySelfTicket in worker-template.ts) and appends it to the opened URL. Managed-only; a pre-feature Worker 501s the mint into an agents artifacts share update hint.

agents feed watch --json is the canonical thin-client operator stream: it composes the existing session watcher with feed attention and activity, while agents sessions watch --json remains the compatible session-only stream. Answers go through agents feed answer <attention-key> so the CLI atomically claims the first reply and routes it over the recorded session reply rail.

Owner-addressed delivery is policy fan-out, not primary/fallback selection. agents send --to owner, deprecated agents notify, and an important feed's channel: owner sink resolve every addressable id in humans.yaml's owner.policy.normal, in policy order. Each destination is attempted independently; a partial failure is reported alongside successful deliveries. Rush-backed destinations that cannot deliver on the originating worker forward their explicit channel and target to a capable macOS peer, so the peer never re-expands the owner policy and duplicates another channel. Legacy notify.owner and a humans file without policy.normal retain the historical single-destination behavior.

Feed channel sinks may override their outbound body with a message: template using the same placeholders as command sinks ({message}, {ticket}, {ticket_url}, {project}, and the rest in feed-broadcast.ts). Placeholder resolution is fail-closed: if the post lacks a referenced value, that sink is skipped. This is the semantic routing primitive for destinations such as an engineering Slack channel: a template that includes {ticket} receives ticket-backed posts without leaking unrelated owner alerts into the team channel. The shared {message} also includes the canonical tracker URL whenever the session has a resolvable Linear ticket, so every selected sink (including the private owner destination) receives a clickable ticket link.

gh is overloaded so the fleet's trained gh pr checks escapes the shared GraphQL rate limit (PHNX-3501). The whole fleet shares one GitHub token, and gh pr checks/view/list are all GraphQL-backed, so the fleet drains GitHub's 5000-point/hr GraphQL budget while REST core (5000 req/hr) sits idle — then every agent's gh pr checks --watch dies with GraphQL: API rate limit already exceeded. A generated PATH shim (the browser-shim pattern, ensureGhOverloadShim in src/lib/installations/shims.ts, installed on agents sync) intercepts only the agent's trained gh pr checks and routes it to the hidden agents __gh verb (src/lib/github/gh-overload.ts), which answers over REST (src/lib/github/rest.ts: prHead + rollupForSha = commits/{sha}/check-runs/status, anchored to the PR's live head SHA); every other verb execs the real gh byte-for-byte. --watch owns the REST poll loop (head-SHA-anchored, so a superseded run's red is never reported — closes PHNX-3042); a one-shot runs real gh first and falls back to REST only on the exact rate-limit stderr. Self-healing: the shim execs real gh whenever agents-cli is gone ([ ! -x "$AGENTS_BIN" ]) or the AGENTS_GH_SHIM sentinel is already set (recursion), so a leftover/orphaned shim can never break gh, and uninstalling restores plain gh immediately. It shadows via PATH precedence and never touches the real binary. Tier-1 wraps only pr checks (clean REST mapping, highest failure volume); pr view/pr list and a proactive linear-rate-limit.ts-style budget are fast-follow. POSIX-only in v1. reviewDecision is deliberately never REST-derived (it is a GraphQL-computed branch-protection/CODEOWNERS decision; approximating it could let the merge loop bypass review).

agents traces sync publishes two redacted derived surfaces: a per-session SessionDetail at sessions/<id>.json (a meta summary — spanMs/activeMs/turns/tools/errorCount/tokens/cost/outcome/repo — plus a plain-language whereItWentWrong, the shape the Phoenix Evals console consumes directly), and a rich per-device index.json. activeMs is spanMs minus every idle gap > 120s (PHNX-3457) so a session resumed after hours, or left idle mid-turn, reads as effort rather than calendar span (the raw span stays available per session as SessionDetail.meta.spanMs). Every index statistic is computed over the AGENT corpus ONLY (PHNX-3474): each session is classified kind = utility when it is internal machine plumbing — no tool call AND ≤2 messages (a single-shot call), or its topic/label matches a known internal-prompt signature (title generation, watchdog, commit-message writer, factory worker; classifySessionKind / UTILITY_PROMPT_SIGNATURES in traces/sync.ts) — otherwise agent. Utility rows (~68% of the raw corpus: title-gen, watchdog ticks, commit-message writes, factory workers, all spawned under the claude harness by the Rush app) are tagged and excluded, never deleted from sessions.db, so sessionsImported is the real agent count, and the medians / needs-attention / tool-error-rate / topic-bucket counts all measure agent work; a top-level utilityCount reports how many were dropped. Each session ref (topic-tile sessions[], needsAttention) carries kind and harness (the producing agent) so the console can filter by both. The index contains duration/error statistics — medianMs/p90Ms are now the ACTIVE-time percentiles (same keys, new value; the fleet-aggregate worker keeps weighted-averaging them unchanged); those blended figures sit alongside segmented active-time stats (PHNX-3472) — agentMedianMs/agentP90Ms over AGENT runs (a session with any tool call OR more than 8 messages) and interactiveMedianMs over the one-shot INTERACTIVE remainder, plus measuredFraction (share of sessions carrying a non-null duration) — so the console can headline agent runs (~15min median) instead of the corpus blend the 63%-one-shot tail pulls to ~15s; every harness now carries a non-null span (resolveDurationMs in session/db.ts derives it at upsert for rush/grok/kimi/cursor/muse/antigravity, which previously left duration_ms NULL) — ranked attention flags, metadata/tool-mix topic buckets — the human task taxonomy the console treemap renders (Feature work · Bug fixes · Refactor · Debugging · Code review · Release · Blog & docs · Fleet / ops), an ordered rules table in classify.ts's classifyTopic, keyed within the five stable TraceTopicGroup groups, each bucket carrying up to 30 example sessions refs ({id,title,kind,harness}) so a treemap tile is drillable into its session list (PHNX-3408) — and structured tool failure cause buckets (real, guard, hook), and a rolling drift signal. The shard also carries a per-session sessions roster (PHNX-3483) — one flat scalar SessionRosterRow per AGENT session (id, title, harness, model, repo, mode, projectType, startedAt, durationMs, toolCount, errorCount, needsAttention, best-effort costUsd), the raw material the Rush console filters and re-aggregates client-side (a pre-rolled scalar can't yield a filtered headline). durationMs reuses sessionActiveMs (the value behind stats.medianMs; 0 when unmeasured) and mode encodes the AGENT-vs-INTERACTIVE segmentation (headless/interactive) so a mode-split median over the MEASURED rows reproduces stats.agentMedianMs/interactiveMedianMs — the segmented stats skip null-duration rows, which the roster still carries at durationMs: 0; utility rows are excluded, so the roster length equals stats.sessionsImported.

device=all is synthesized, not stored: each device syncs its own shard, and the worker merges them on read (worker-template.ts). That merge re-projects field by field — so a new per-device shard field reaches the default console view only if it is also added to the merge, else it silently vanishes from /all while still present per-device.

Group-by dimensions for the console insight bar are pure functions in src/lib/traces/segments.ts: deriveAgent (model × harness), classifyTaskType, failureTiming, and computeLatency (time-to-first-tool percentiles from steps[0].startMs). The sync integrator wires them into the shard.

Cross-session failure clustering + wasted-time attribution (PHNX-3141) is src/lib/traces/insights.ts's computeInsights() — the piece that turns flat per-tool error counts into ranked, time-weighted failure patterns: failed tool_calls rows are grouped by (tool, cause, normalized-error) — volatile tokens (ids, counts, countdowns) stripped so 47 near-identical "rate limit exceeded for user N" errors fold into one pattern — and each pattern accumulates wastedMs from two sources that sum. (1) The failed call's own blocking durationend_timestamp - timestamp in tool_calls (PHNX-3437) — booked whenever the end time is known, independent of whether another call follows: a call that hung for minutes and then failed wasted that whole time even as the last call in its session, the case the gap heuristic alone booked as ~0 (this is what makes a fail-fast fix like PHNX-3407 — a user_location stdin hang cut from ~5.5m to <1s — measurable, dropping from ~5.5m to ~0). (2) The inter-call gap (ordinal/timestamp, already indexed) whenever the next call repeats the same signature (a retry loop) or the gap itself is a stall (≥60s), measured from the call's END when known so (1) is never double-counted. Both are bounded per contribution to MAX_GAP_ATTRIBUTION_MS (30m) so one huge gap — or a corrupt/backwards end timestamp — can't be booked as failure-loop waste; a real active loop is many short gaps that still sum large. A NULL end_timestamp (rows an older extractor stored, before the paired TOOL_INDEX_VERSION bump re-derives them; or a call still pending at scan end) falls back to the original gap-from-START heuristic unchanged — no crash, no NaN. Patterns are bounded top-K, ranked by wastedMs (impact) — never by raw occurrence count — so a single rare multi-hour loop still outranks a frequent but cheap one. Cost stays proportional to this sync's row count (no transcript re-parsing of already-classified sessions), so it stays incremental at 10k+ session scale. Patterns also carry a phenotype dimension (false-termination / premature-completion / out-of-order / failure-to-act, phenotype.ts) folded into the group key alongside (tool, cause, normalized-error) (PHNX-3327). That classification needs the full derived trajectory, which flat tool_calls rows don't carry — so it is computed per-session ONCE (parse → trajectory → classifyPhenotype) and persisted in the mtime+size-keyed session_phenotypes cache (same shape as session_topics / session_insights, readSessionPhenotypes / writeSessionPhenotypes in db.ts), then read for the whole corpus on every sync. Reading the cache for all rows — never just this run's freshly-parsed batch — is what keeps two identically-signatured sessions in ONE cluster regardless of which incremental sync first saw each; the signature output itself ({ tool, cause, key }) is unchanged, phenotype rides alongside it.

FieldTypeDescription
bucketHistoryBucketStats[][]14-day rolling window. Each inner array is one day's per-bucket stats (errorRate = tool errors / total calls; stallRate = sessions with ≥1 stall / total sessions).
driftSignalsDriftSignal[]Buckets whose error or stall rate crossed ±0.20 vs the 7-day average. severity: `degrading
failurePatternsFailurePattern[]Top-25 cross-session failure clusters from computeInsights(), ranked by wastedMs desc. Each carries signature (tool/cause/normalized key), phenotype (the cluster's failure phenotype or null; folded into the group key + id so the same signature under two phenotypes is two clusters — PHNX-3327), sessions, occurrences, wastedMs (estimate — labeled, never presented as exact), exampleSessionIds (≤5), and drift vs the same pattern id in the previous shard.
wastedMsTotalnumberSum of wastedMs across every cluster found this sync — not just the top-25 rows in failurePatterns above.
latencyLatencyInsight{ firstToolMs: { p50, p90, p99, max } } — time-to-first-tool percentiles, reusing segments.ts's computeLatency() over each session's earliest tool_calls row.

Per-session SessionDetail also carries surfacedToolFailures — every failed step, listed unconditionally regardless of the run's overall outcome, so a session that ultimately succeeded (merged / tests-green) still shows the tool failures it hit along the way instead of hiding them behind a green run-level status (the failure this fixes: a coding-agent analogue of a monitoring tool reporting 0% error rate while a run burned hours in a failed-tool retry loop).

meta.outcome is the truthful run outcome (PHNX-3387): a run with tool errors reads completed ONLY when it causally recovered — a substantive, non-human-facing tool step succeeded strictly after the last error AND resolved the failed work (its work signature — the effective program for a shell step, the tool identity otherwise — matches an errored step's), the exact predicate the false-termination phenotype uses (recoveredAfterErrors in phenotype.ts, shared with deriveRunOutcome in sync.ts so the console outcome and the phenotype can never disagree about whether a run finished). A run whose last substantive step is the error, whose only post-error steps are human-facing (a punt to AskUserQuestion), or whose only post-error success is unrelated work (a failed bun test followed by an incidental ls), stays errored. This is what makes surfacedToolFailures on a completed run honest: those are failures the run recovered from, not a green status hiding an unresolved one. It never flips a genuinely-unresolved run to completed (no regression vs the old errorCount > 0 ? errored : completed).

On live sync, the prior shard is fetched from R2 before the PUT to seed history; failures (404, parse error) fall back to empty history. --dry-run --out <dir> seeds from the previous index.json in the output directory. Topic classification is lazily cached in the self-healing session_topics table by transcript mtime + size; it is not part of the hot session scan or SCHEMA_VERSION. agents traces sync --dry-run --out <dir> computes both surfaces from the local sessions.db and writes them to a directory (no Phoenix auth, no worker, no upload) — a local export for verifying the real signal before the hosted path is wired.

The sync gate is a per-device mtime watermark plus a failure ledger, both in traces-sync.json (PHNX-3267). The watermark alone cannot retry a failure — a later-mtime success advances it past an earlier failed row, which the next run then skips forever — so the ledger also records each failed session's identity + typed, redacted evidence and the row query unions those retry-worthy ids back in regardless of the watermark. Failures are typed transcript-unavailable (the file is gone; expected history, not re-read, aged out after 14 days), parse-failed, or upload-failed (both retried until they resolve); SyncResult carries the per-kind counts, agents traces sync prints the breakdown, and agents traces status lists the outstanding retry set. A --dry-run never touches the ledger.

Authentication — Phoenix ID

agents auth login signs in with Phoenix ID (id.byphoenix.com), the one family identity shared with the Rush CLI and prix.dev — signing in here creates no Rush account. It runs the RFC 8628 device flow: startDeviceAuthorization()POST /api/v1/auth/device/authorization, print the short user_code + verification URL (works on any browser/phone, including SSH/headless hosts), then pollDeviceToken() polls POST /api/v1/auth/device/token until approval (src/lib/identity/index.ts).

On approval the CLI stores the opaque Phoenix bearer verbatim — there is no exchange step. writeSession({ access_token, email, userId }) writes phoenix-session.json (mode 0600) in the runtime state dir (src/lib/identity/client.ts). Every backend call reuses it as Authorization: Bearer <access_token> against Phoenix ID; agents auth whoami resolves it at GET /api/v1/auth/me; agents auth logout deletes the file (local-only — signs out nothing else).

  • PHOENIX_ID_BASE (default https://id.byphoenix.com) points the CLI at a different account service for local/private backends.
  • The CLI never reads another product's credentials (e.g. ~/.rush/user.yaml); each surface holds its own copy of the same Phoenix bearer. Managed agents artifacts share publishes under this identity — the share Worker verifies the bearer at ${PHOENIX_ID_BASE}/api/v1/auth/me.

Core design choices (read this first)

Break these and downstream code drifts silently.

1. Three DotAgents repos, resolution is project > user > system

Resources AND agents.yaml resolve in that order. Same-name overrides, everything else unions.

PathRoleEdited by
<repo>/.agents/Project repo — project-pinned commands / skills / hooks / rules.Project maintainers
~/.agents/User repo — user resources + ALL operational state (versions, shims, sessions, agents.yaml, browser).You / CLI
~/.agents/.system/System repo — npm-shipped defaults ONLY.Maintainers (gh:phnx-labs/.agents-system, GitHub rename target phnx-labs/.agents)

The system repo was renamed phnx-labs/.agents-systemphnx-labs/.agents on GitHub (PHNX-3394): the -system suffix was redundant once the layering itself is the role. DEFAULT_SYSTEM_REPO still clones the pre-rename slug — GitHub's own redirect makes that resolve fine, so there is no forced cutover — and both names are recognized as the system origin everywhere a remote is compared, via isSystemRepoRemote in src/lib/git.ts.

The system repo ships hooks that run as shell, so its auto-pull is origin-verified (PHNX-2957). Both fast-forward paths — agents use (foreground) and the opt-in AGENTS_AUTO_PULL=1 background worker — route through tryAutoPullSystemRepo, which pulls only when origin passes isExpectedSystemRepoRemote: the canonical isSystemRepoRemote, or the exact AGENTS_SYSTEM_REPO the operator pointed at. A repointed/unexpected origin is refused (never fast-forwarded), since a pull from an attacker's fork would be remote code execution on the next command that loads a system hook. Never widen a system-repo pull to bypass this guard; a legitimate alternate upstream is expressed through AGENTS_SYSTEM_REPO, not an unverified pull.

Extra repos register via agents repo add <source> → clone into ~/.agents-<alias>/ and participate after the user repo. The companion extras repo phnx-labs/.agents-extras keeps its name but is now private and opt-in: it is not cloned by default and is added explicitly with agents repo add.

2. AGENTS.md is the canonical memory file

CLAUDE.md and legacy GEMINI.md are symlinks. Edit AGENTS.md only — editing a symlink target directly gets stomped on the next sync. The sync writes the right file name per supported agent (OPENCODE.md, .cursorrules, etc.).

"Memory" names three distinct, unrelated mechanisms — don't conflate them. Besides the rules file above, agents memory (src/lib/memory.ts) is a layered resource — ~/.agents/memory/*.md facts fanned out into each capable version home (.claude/memory/, .codex/memories/, …) by syncMemoryToVersionHome. Separately, Claude Code's own native per-project auto-memory — .claude/projects/<project-key>/memory/*.md, freeform notes Claude writes into itself during a session — is neither a ResourceKind nor populated by agents-cli at all. Because getVersionHomePath gives every installed version its own isolated HOME, that native dir used to be a THIRD, unmanaged, per-version copy: a note written under one Claude version was invisible under another (PHNX-2817). syncClaudeProjectMemoryDir (src/lib/memory.ts, wired into syncResourcesToVersion, src/lib/installations/versions.ts) fixes this the same way project-level rules are version-independent: every version home's copy is a symlink into one canonical dir (~/.agents/.cache/state/claude-project-memory/<project-key>/, keyed by the same cwd-to-dash-name encoding Claude Code itself uses, claudeProjectDirName in src/lib/project-key.ts) — no merge/copy step, no drift, a pre-existing real directory's content is migrated into the canonical dir once rather than discarded.

3. Capability table gates per-agent writes

supports(agent, cap, version?) in src/lib/capabilities.ts is the only place that decides whether an agent+version can receive a resource. Out-of-range versions are skipped silently — do not add per-call agent checks elsewhere; route through supports().

4. No fallback logic for legacy layouts

src/lib/installations/migrate.ts folds legacy paths ONCE at install time. The bootstrap gate that invokes runMigration() then writes the .migrated sentinel (MIGRATED_SENTINEL_FILE, src/lib/state.ts), keyed to the migration SCHEMA version, so the scan short-circuits next run — runMigration() itself only relocates a legacy sentinel via moveFileOnce, never writes one. Downstream code assumes the post-fold layout. "Just-in-case" branches re-introduce drift bugs; the migrator is the single source of truth for legacy handling.

5. Hooks live in a single layered hooks.yaml

System + user hooks.yaml merged, user wins on same name. Per-entry matches: predicates (prompt_contains, prompt_matches, tool_name, tool_args_match, cwd_includes, project_has, git_dirty, permission_mode, permission_mode_not) AND together at fire time — the two mode predicates are fail-open on absence, since only some harnesses report the live mode in hook input. Use permission_mode_not to gate a hook off in one mode: permission_mode is an allowlist, so expressing "everywhere except plan" through it means enumerating every other mode, and that enumeration silently stops matching when a harness adds or renames one — which for a guard means it quietly stops guarding. Per-entry enabled: false disables a system-shipped hook from the user side. The agents: field in ManifestHook is @deprecated — the capability table decides which agents register a hook.

6. Multi-agent work → agents teams

DAG-style, boundary contracts, --watch supervisor, --worktree isolation, optional --cloud dispatch. The old mcp__Swarm__* surface was folded into teams (migrateLegacySwarmToTeams() in src/lib/installations/migrate.ts). Don't reach for Swarm — gone.

A teammate opens PRs but MUST NOT merge its OWN PR without a posted non-author verdict (PHNX-3236). A write-capable teammate has gh pr merge and authenticates as the repo owner, so nothing about being a teammate stops it from merging its own PR straight past the required non-author review — which is exactly what happened in the RUSH-2988 wave-1 dispatch (PRs #1817, #1820 self-merged with reviews: []). The boundary is enforced in two layers, and neither is per-brief prompt wording (the failure mode: only one teammate in that batch was told not to merge, and its siblings weren't):

  • Hard enforcement — merge-guard.sh. The PreToolUse Bash guard the teammate inherits from the shared version home blocks gh pr merge on a PR that has no non-author verdict on it. Its verdict check (pr-verdict.py) excludes any review/comment authored by the PR's own author (landed in the same ticket, phnx-labs/.agents #395), so a teammate posting VERDICT: APPROVE on its own PR — every fleet agent shares one GitHub identity — no longer clears the gate.
  • Dispatch default — TEAMMATE_PR_POLICY. One helper, withTeammatePrPolicy(prompt, mode) (src/lib/teams/agents.ts), appends the boundary to the prompt of every write-capable (non-plan) teammate, fresh or resumed, applied at all three dispatch surfaces so they can't drift: the local launch and the --device remote launch (both through buildRunArgv), and the --cloud launch (cloudDispatchOptions in src/commands/teams.ts). It is the harness-independent layer — it reaches every entry in TEAM_AGENT_TYPES, not just the hook-capable ones — rather than depending on each dispatch remembering to say it. Pinned by agents.pr-policy.test.ts.

Coverage, and the residual gap (stated per §Surface parity). The two layers do not overlap everywhere, so name where each reaches:

  • Hook-capable local / --device remote teammates get both layers — the hard merge-guard.sh block plus the prompt. This is the common case.
  • Cloud teammates (--cloud) get only the prompt: a cloud teammate runs in the provider's sandbox, not the shared local version home, so it never inherits merge-guard.sh. Wiring the prompt into cloudDispatchOptions is what keeps this surface covered at all.
  • Hook-incapable harnesses (e.g. Warp/ozhooks: OFF and no allowlist in the capability table above, yet present in TEAM_AGENT_TYPES) get only the prompt, local or remote, because there is no hook surface for merge-guard.sh to occupy and no allowlist to express a Bash(gh pr merge:*) deny.

So for the cloud and hook-incapable cases the prompt is a soft control an agent can ignore under pressure — the exact reliability limit the ticket names. That residual is accepted deliberately: there is no code-level merge interception point on those surfaces short of provider-side branch protection (cloud) or a per-harness enforcement mechanism those harnesses don't expose. Enable server-side GitHub branch protection requiring a review to close it uniformly regardless of harness or dispatch surface — that is the only guard that does not depend on the client.

agents-cli itself never merges a teammate's PR (pr-watch only opens follow-up fixers or escalates to a human — there is no merge action); the teammate's own gh pr merge is the only merge path, which is why both layers target that call.

An agent conversation from a session-capable harness MUST remain a session, whether launched interactively, headlessly, as a teammate, or by a routine. A caller MUST preserve the harness transcript and make it discoverable by agents sessions; it MUST NOT replace or hide that conversation record with its own execution metadata. This applies to SESSION_AGENTS, not a harness such as Warp that exposes no local transcript. The indexed session row carries the relationship when the harness and launch path provide one:

Launch surfaceSession relationshipSeparate execution state
agents run, including headless and --deviceOrdinary indexed session for a session-capable harness; when its SessionStart hook records an id, the launch id joins a remotely coined session back to the dispatch. A hookless run remains unmapped rather than receiving a fabricated id.Dispatch/audit events
agents teams teammateIts own session id plus teamOrigin; parentSessionId links the orchestrator when the teammate was spawned inside an identified agent sessionTeam registry + teammate meta.json own DAG/task/process state
Agent/workflow routineThe transcript is archived under the run; supported archive readers index it with origin: routine, routineName, and routineRunId.history/runs/<routine>/<run>/meta.json owns the attempt outcome
Command-only, missed, blocked, or skipped routineNo session is synthesized because no agent conversation occurredThe routine run record is the complete canonical record

Source of truth: buildExecEnv / emitResolvedSessionId in src/lib/exec.ts, listTeamsActive in src/lib/session/active.ts, archiveRoutineTranscripts in src/lib/daemon/runner.ts, and decorateRoutineSession in src/lib/session/discover.ts. Enforced by src/lib/session/team-filter.test.ts (--teams includes team sessions with teamOrigin populated), src/lib/daemon/runner.test.ts (routine transcript archival), and src/commands/sessions.test.ts + src/commands/sessions.cli-live.test.ts (routine run history plus linked sessions, with no fake session for command-only runs).

Team-origin sessions are durable session rows but are excluded from the ordinary historical listing by default to keep an orchestrator's fan-out from flooding it; agents sessions --teams includes them, while live teammates appear in agents sessions --active with context: teams. That is a presentation filter, never permission to omit the transcript from the session index.

Current routine-archive gap: readRoutineArchiveMeta indexes Claude, Codex, and Cursor routine archives. Kimi's state.json + wire.jsonl are archived by archiveRoutineTranscripts but do not yet have a routine-archive reader, so they are not session rows today. Treat that as named drift from the invariant, never as precedent for a second run-only conversation model; adding another routine harness requires both archival and a parser in readRoutineArchiveMeta plus an indexing test.

8. Routine definitions and device activation are separate

Routine YAML under project, user, or system routines/ describes what runs and when. Whether it runs on one host is membership in that host's top-level devices/<hostname>/agents.yaml routines: list. Pause/resume and setup MUST write only the target host's device file; they MUST NOT rewrite a definition with enabled, devices, or runtime metadata. Run history belongs in .history/runs/<routine>/<run>/, and that run history — not the session index — is the canonical record of an attempt; sessions/logs/reports are optional children of a run (so a missed/blocked/skipped attempt is visible with no session).

projects (plural) is grouping metadata only; the singular project anchor is a separate concept. The plural list organises a routine in list/the menu bar and MUST NOT affect execution; the (planned) singular project/--project-anchor plus a routine-level cwd is the execution anchor, resolved on the execution target. A routine's repo is an external Git/cloud/webhook identity, never a local cwd. The reliability contract — context resolution, readiness/pause-on-blocker, single-fire (routine, scheduledFor) claim distinct from the active-run claim, and the blocked/skipped run statuses — is normative in docs/specifications.md §Routine execution & readiness (RT-1..RT-11) and §Scheduling & execution singularity (SING-13, SING-15, SING-16); much of it is [Intended] (RUSH-2290), and each requirement marks landed vs intended.

Routine execution context is separate from grouping and repository identity. Plural projects is metadata-only; singular project selects one ProjectDef execution base; cwd is resolved on the eventual execution device. A rootless Linear project may still use a relative cwd, which anchors at that target user's home. repo remains GitHub/cloud/webhook identity and MUST NOT be used to infer a local checkout. Readiness failures save valid definitions paused through device activation; they never write mutable activation into routine YAML.

9. Self-updating agents are ONE binary, not fictional version-homes

Some harnesses (droid, grok, antigravity, cursor, hermes, muse, kiro, goose) install via an official curl … | sh / brew install script that carries no version token — the installer only ever fetches the current release and the binary self-updates in place. isSelfUpdatingAgent() (src/lib/agents.ts) is the single predicate for "no pinnable semver"; route every such decision through it, never a scattered === 'droid'. Its narrower cousin isGlobalBinaryAgent() (src/lib/installations/store.ts) — computed by probing whether getBinaryPath ignores the version arg — is true only when the agent resolves to ONE global binary (droid). For those, listInstalledVersions collapses the phantom per-version dirs to a single canonical entry, reconcileStaleLatestForAgent folds the stale dirs into the survivor, agents view shows the live --version, and agents add droid@1.2.3 gracefully installs the current release instead of erroring. grok is self-updating but stores a real per-version binary under each version-home, so it is NOT a global-binary agent and is left uncollapsed. (RUSH-1321)

For a concrete self-updating spec, the requested token is the stable installation label/account slot even when the vendor installer can only fetch today's release. installation.json.releaseVersion records the probed vendor release separately. This identity/release split is load-bearing: two Cursor or Grok homes may carry the same current release while retaining different native credentials, and removing an isolated label must never target a normal installation merely because their releases match. @latest keeps the probed release as its convenient label; concrete labels stay exactly addressable.

10. Diagnostic command taxonomy — doctor is the umbrella (RUSH-2027)

Three diagnostics, distinct scopes. Don't blur them — each answers a different question, and a new health check goes in the one whose scope it matches.

CommandScopeAnswers
agents fleet statusCoarse device health across the fleetAre devices online, do they have the agent CLIs installed, are they signed in, what is the agents-cli version skew, how many agents are running on each box. NOT fine-grained resource divergence. Publish-own/read-union: each daemon publishes only its own row (no N² ssh probe, RUSH-2061); the reader unions peers on demand (--local --json is the per-host publish endpoint).
agents inspect <agent>[@version]Deep single-harness diagnosisPer-resource diff between one version home and its resolved sources; manifest staleness; orphans. One harness, one machine.
agents doctorUmbrella — overall fleet + harness healthLocal diagnostics (CLI presence, per-version sign-in, per-version sync, orphans) and cross-device divergence, rendered as the prioritized critical-at-top + per-computer hybrid below. The single command a user runs to discover problems before runtime.

The per-version resource diff is content-aware for EVERY kind syncResourcesToVersion writes (PHNX-3504). diffVersionResources (src/lib/doctor-diff.ts) — the engine behind agents doctor and agents inspect — no longer name-compares mcp/permissions/subagents or skip workflows/memory. DOCTOR_ALL_KINDS = commands, skills, hooks, rules, mcp, permissions, subagents, plugins, workflows, memory (promptcuts is dropped — it is a single version-unscoped file, not per-home). A byte change under an unchanged name is diff, never a false ok: mcp structurally compares the parsed home server def vs the resolved source (no claude mcp add shell-out); subagents re-render the source through the SUBAGENT_TARGETS transform and byte-compare; workflows compare layout-aware per harness (Claude's copied WORKFLOW.md tree vs the transformed Kimi/Antigravity/OpenClaw/Grok file vs the Goose recipe; antigravity's is the shared HOME-global dir); memory diffs the knowledge facts (~/.agents/memory/*.md, bounded by the .agents-cli-memory.json manifest — NOT the rules-preset list that overloads AvailableResources.memory); permissions compare per-rule in the harness's native vocabulary for the representable harnesses (claude/opencode/cursor/droid/openclaw) and stay presence-only with detail: 'format cannot verify content' for the lossy TOML/flag harnesses (codex/grok/kimi/kiro/antigravity/hermes/copilot) rather than faking ok. The shared byte compare (filesContentMatch/dirsContentMatch, src/lib/resource-content-diff.ts) is reused by skills and the plugin-drift describer. A completeness test in doctor-diff.test.ts binds DOCTOR_ALL_KINDS to the writer set so a future synced kind cannot silently become a doctor blind spot, and agents doctor --fix (src/lib/heal.ts) reaches the newly-covered kinds.

agents doctor <agent>[@qualifier] accepts symbolic qualifiers@latest, @oldest, @default/@pinned, @all, or an exact version — resolved through the shared agent-spec engine (lib/agent-spec/index.ts, resolveAgentTargets). Bare agents doctor <agent> (no qualifier) sweeps every installed version without setting versionExplicit; --fix then excludes isolated copies. Any explicit qualifier sets versionExplicit: true, scoping --fix to the resolved version set (including isolated copies for @all). AgentSpecError from the engine is surfaced as a user-facing error. Routing flags (--device/--remote-cwd) are stripped via stripRoutingFlags before target parsing, so agents doctor claude@latest --device remotebox resolves correctly on the remote. (issue #2058, src/commands/doctor.ts:parseTargetArg)

agents doctor is a prioritized, comprehensive-by-default hybrid (RUSH-2069). There is no --verbose. A top ✗ CRITICAL — needs you now (N) section lists every critical across the whole fleet worst-first; a ─── by computer ─── section then gives each device its warnings plus a compact accounts/versions line (every installed version + its account, provable ✓ / ✗). Single-machine agents doctor collapses to the CRITICAL section plus one ▸ <machine> block. Severity: critical is logged-out (provable), missing-hook, missing-plugin, unwired-hook, hook-runtime-broken, cli-missing, ssh-key-enrollment and owner-sink-unreachable (the feed/notify owner-delivery lane can't reach the owner from this box, RUSH-2262); warning is logout-unprovable, missing-resource, content-drift, never-synced, stale, repo-behind, repo-drift, version-skew, fleet-resource-gap, hook-runtime-visibility-unavailable, orphan, duplicate-hook, duplicate-hook-drift, host-cli-missing, host-cli-invalid, rc-secret-export, env-secret-export, auth-bundle-wrong-backend, exec-policy, stale-cli and binary-shadow. (RUSH-2162 moved never-synced and duplicate-hook-drift to warning — both are stale-sync states one agents sync resolves.)

FINDING_SEVERITY in src/lib/devices/doctor-findings.ts is the single source of truth: the builders read their severity from it, and a test asserts this list and the module docblock assign every kind to the same bucket it does. Change a severity there and the test names the docs to move with it. The findings model, builders, remediationFor, and the pure renderFindings live in src/lib/devices/doctor-findings.ts.

One root cause is one line. A readout the user cannot scan is as useless as no readout, so the builders de-duplicate before rendering — on a real machine this takes ~57 rows down to ~16, and the rules are unit-pinned in doctor-findings.test.ts:

  • Per version, per kind, one row. emitGroup names a lone resource in full (hook 'git-guard' missing) and otherwise emits a count plus two examples (32 hooks missing (incl. 'a', 'b')). Never one row per resource.
  • Per agent, one row across versions. collapseAcrossVersions folds findings with the same (device, agent, kind, severity, account, message) into a single row carrying versions, rendered claude (5 versions), and widens the remediation to the agent-wide sweep. Three exclusions, each because the widened remediation would be wrong: isolated copies (runFix skips them, so the sweep would leave one broken — the caller passes isolatedVersions from isVersionIsolated); findings with no agent (their version is a repo alias); and logouts (NEVER_COLLAPSED) — a login is inherently per-version, there is no @all for it, and dropping the version falls back to the bare native hint, which the shim points at the default version.
  • Orphans are one line per machine. They are cleanup-only and agents prune cleanup --all fixes every version at once — --all is load bearing: without it cleanup sweeps only each agent's default version (commands/prune.ts:351).
  • Duplicate version-home hooks are one line per (agent, severity). agents sync <agent>@all --yes reconciles every copy at once, and a machine with five installed claudes otherwise emits two dozen identical rows.
  • No vaguer restatement. A version that just listed its drifted/missing resources gets no sources changed since last sync row on top, and a never-synced version reports one warning (agents sync <agent>@<version> --yes) instead of one row per absent resource.

Every check the old overview printed is a finding now. renderOverviewText was the ONLY text renderer for several independent checks, so deleting it dropped each of them from the command — the top defect this redesign had to answer for, and it recurred three times during review. They all enter buildLocalFindings as plain inputs (never probes, so the module stays pure and every branch is testable without a shell, PowerShell, or an installed CLI):

CheckInputFinding kind
Credential-shaped shell-rc exports (RUSH-1968)rcSecretsrc-secret-export
The file-store master key live in the process env (RUSH-1968)masterPassphraseInEnvenv-secret-export
Windows exec policy blocking agents.ps1execPolicyexec-policy
Windows OpenSSH key path/content/ACL invalidwindowsSshEnrollmentssh-key-enrollment
Hooks duplicated across version homesduplicateHooksduplicate-hook{,-drift}
Declared host CLIs not on PATHhostClis.statuseshost-cli-missing
Host-CLI manifests the loader rejectedhostClis.errorshost-cli-invalid

Before deleting any renderer here, enumerate what it called.

A remediation must fix EVERY version in its row, and must be a command that exists. Three separate rounds of review here found remediations naming a command form that does not do what the row claims — agents sync <agent> (default version only), agents repo pull (skips the system repo), agents prune cleanup (default versions only), agents clis install <a> <b> (takes one name), and agents run <agent>@<v>, then <cli> login (the second command resolves through the shim to the default version, not the one that is logged out — use agents run <agent>@<v> -- login, since -- forwards verbatim into that version home). Open the command definition and check arity, flags, and scope before writing a remediation string. agents sync <agent> targets only the default/sole installed version (commands/sync.ts:8), so a row collapsed across versions uses the @all selector — agents sync <agent>@all --yes. A fleet resource gap is absent from that box's central repos, so the central-to-home agents doctor --fix cannot close it — and neither agents repo pull nor the sync umbrella touches the system repo (commands/repo.ts:1186, lib/sync-umbrella.ts:104), which moves with the npm package instead, so that row names both paths rather than one command that quietly covers half the cases. A repo-drift row carries the repo alias (user / system) rather than hardcoding one.

Sign-in is per installed VERSION, and a logged-out claim must be provable. credentialPresence(agent, versionHome) splits a credential's existence into the per-version home and the active/global HOME; a logged-out critical is emitted only when BOTH are absent (provable = !perVersion && !active). A version sharing the global login is signed in, not out; an agent with no inspectable identity (!supportsAccountInspection) never yields a logout finding, not even the hedged warning. Do not enumerate that set here or in testsACCOUNT_INSPECTION_AGENT_IDS and CREDENTIAL_FILE_SEGMENTS (src/lib/agents.ts) are the source of truth and agents move between them (antigravity and cursor both did, mid-review, each time turning a hardcoded list into a false doc claim or a red test). Derive it: ALL_AGENT_IDS.filter(supportsAccountInspection). Login remediation is version-targeted via agents run <agent>@<version> + the harness-native login (loginHint) — but ONLY for the per-version-isolated set (claude/codex/grok/kimi/opencode/copilot); gemini/antigravity/droid/cursor share their login, so the fix says so instead of faking a per-version repair. Per-version sign-in rides the device inventory (FleetInventory.signIn, populated by collectLocalFleetSignIn in src/lib/devices/fleet-inventory.ts); an older remote CLI that omits it degrades to an "older agents-cli — upgrade" warning.

Cross-device divergence lives in agents doctor --devices. It compares each device's self-reported harness inventory against the local baseline and flags a resource / agent-version / config-repo present on one box but missing on another (e.g. the swarm plugin on zion but not yosemite-s0). The data path:

  • Every device's top-level agents doctor --json emits a fleet inventory field (src/lib/devices/fleet-inventory.tscollectLocalFleetInventory): installed resource names per kind, installed version ids per agent, and .agents/.system repo state (readRepoState in src/lib/git.ts).
  • runDevicesDoctor (src/commands/doctor.ts) fans that payload out per device and runs the pure comparator compareFleetInventories — SSH-free, so it's unit-tested against fixtures with no live fleet — then maps the divergences and each box's per-version sign-in into the hybrid via fleetDivergenceToFindings / signInToFindings / renderFindings (src/lib/devices/doctor-findings.ts).
  • agents fleet status reuses the same comparator inside buildFleetHealthReport (src/lib/devices/health-report.ts) to add a per-device divergence warning to its rollup.

Read-only by default — divergence detection never installs or syncs. --json carries a stable fleet divergence block for the VS Code extension / Agency.

Per-device harness/account readiness lives in agents devices harnesses / agents devices accounts (RUSH-2003). A fourth fleet lens, distinct from the three diagnostics above: not "is the fleet healthy?" (fleet status) or "is a token live?" (fleet ping), but "what can each box actually run right now?" — per installed agent@version, its account, signed-in, quota, and a single ready verdict (signed in AND not rate-limited). harnesses is the per-install view; accounts collapses installs that share one account. The collector (collectLocalHarnessInventory) reuses getAccountInfo (identity), the daemon-warmed usage cache via getUsageInfoByIdentity({ readOnly }) (quota — never blocks on a per-account network fetch unless --refresh), and deriveUsageStatusFromSnapshot (throttle state). Claude's managed status-line command also writes the native five-hour and seven-day rate limits from normal interactive responses into that same identity-keyed cache; it never reads or refreshes OAuth credentials. Two invariants ride on that writer (PHNX-3392, spec GWT-E5c): a run that hits its weekly limit persists a rate_limited week window (claude-statusline.tsmergeClaudeUsageCacheWindows), so the next collectRunCandidates excludes the account; and a MISSING snapshot is treated as unknown, never as headroom — capacityWeight(null, …) draws UNVERIFIED_WEIGHT (1), not full capacity, so a blind account on a worker box (usage 403s, RUSH-2392) can't outrank a verified-healthy one, while an all-blind pool (no snapshots at all) still draws a pick. An all-stale pool does NOT (PHNX-2526): when every account carries a snapshot older than USAGE_DECISION_MAX_AGE_MS and none is verified, balanced/available refuse to auto-pick on a plausible-but-wrong number — resolveRunVersion returns noVerifiedUsage, an interactive run shows the account picker and an unattended one fails loud with NO_VERIFIED_USAGE (formatNoVerifiedUsageError); the stale candidates survive only in rotation.healthy for bounded post-rejection failover. The distinction is hasStaleUsage (a present-but-old number) vs blind (no number to be misled by). The fan-out mirrors fleet ping (probe self in process, SSH each peer's devices harnesses --local --json worker, same per-device + overall deadlines). Everything but the collector is pure and unit-tested (harness-inventory.test.ts). Agent coverage is ALL_AGENT_IDS-driven, so a new harness is included automatically.

Usage is a per-account fact, fleet-synced from headed boxes (PHNX-3392). A Claude rate limit is metered per ACCOUNT, so the 5h/weekly numbers are identical on every box — but only a headed device (personal/desktop) can READ them: the usage endpoint needs the user:profile scope only the interactive login carries (RUSH-2392), and the status-line writer above only fires on a box that runs Claude interactively. A headless worker has just the user:inference setup-token, which the endpoint 403s, so its claude-usage.json stays blank and agents view shows no S:/W: bars. The usage-sync daemon service closes the gap without a device-to-device SSH mesh: each headed daemon publishes one identity-keyed snapshot to its owned tracked file at ~/.agents/devices/<device>/daemon-state.json; each tick automatically commits only that owned file, fetches/rebases, and pushes the user repo under one cross-process lock and a 45-second hard process-tree deadline. Each worker daemon therefore receives the file without an operator running agents repo sync user, then reads its local checkout and merges every headed snapshot NEWEST-WINS (ingestPeerClaudeUsageRows, src/lib/accounting/usage.ts). Both sides need not be online together, a tick opens no device-to-device SSH mesh, and two headed publishers converge per identity regardless of order. auth-sync shares the same envelope for safe ready/missing/invalid metadata only; credentials never enter Git. A deterministic ready source uses the existing encrypted SSH transport only to provision a peer whose shared verdict says missing, and that exceptional push is async with a hard deadline that kills the direct SSH client and remote connection. The store path and newest-wins flow have real-file tests (fleet-shared-state.test.ts, usage-sync.test.ts); a real two-checkout bare-remote test proves the automatic Git delivery (fleet-shared-repo-sync.test.ts). The legacy hidden __usage-ingest/__usage-export verbs remain compatible with older installed peers and retain real-file / real-CLI coverage. A synced row reads as last_seen (cached), never a live fetch, so a worker's bar is honest about being propagated.

11. Session recovery is one decision on the origin device

resolveSessionRecovery in src/lib/session/recovery.ts is the only place that chooses native resume versus /continue. sessions resume and run --resume route through it — as do the retired focus/attach/reconnect spellings, which are hidden aliases that still run the same bodies. Native resume is valid only in the exact origin version's isolated home when that home still owns the indexed transcript AND some injectable credential for this harness is healthy (the origin login, or a provider rotated in on a usage limit). A removed, signed-out, revoked, trashed, backup-only, or same-number reinstalled origin — or a limited origin whose transcript is no longer in that home — uses a healthy account of the same harness and reads the indexed transcript with /continue. Claude native resume uses the earliest recorded transcript cwd, which selected projects/<cwd-key>, not the later cwd stored from its first user turn. Never add a caller-local fallback that native-resumes another version home, and never let run auto change harnesses during recovery.

Native-first with same-harness account rotation (PHNX-3626). An exhausted/rate-limited origin does NOT drop straight to /continue. When the origin version home is installed, native-resume-capable, and still owns the transcript, recovery first rotates to a healthy injectable account of the SAME harness — a durable provider setup-token/API-key account (RUSH-3182), the only kind that can authenticate a resume that must read the origin home's transcript, since a native login lives in its own isolated home and cannot be forwarded — and stays NATIVE in that same home (RecoveryAccount, injected via the --account spawn path). resolveSessionRecovery reads the provider-inclusive pool (collectRunCandidatesForRun) for exactly this. /continue is last-resort: a signed-out, revoked, trashed, backup-only, or same-number-reinstalled origin, or a limited origin whose transcript is no longer in that home. When that continue pick is a provider account, the target still carries RecoveryAccount and exec injects it the same way native does (PHNX-3674) — otherwise spawn would authenticate as the version home's native login, which is the exhausted origin when no healthy native sibling exists. The balanced picker (--strategy balanced) is the same weighted-by-headroom selector dispatch uses; there is no second scheduler.

The origin version is recorded forward at launch. buildExecEnv exports AGENTS_RUN_VERSION, the SessionStart hook joins it to the harness's real session id in the by-session/<id>.json sidecar (like mode), the scan joins it at row build (meta.version ?? actorRec.version in upsertSession), and the db upsert COALESCEs it — the same write-once launch-metadata treatment as mode/harness/ actor. So a session whose transcript carries no derivable version (codex's .codex-homes/<version>/ home, which extractVersionFromManagedPath also now reads) still native-resumes instead of degrading to /continue for a missing recorded origin. Because version is now launch metadata rather than a purely transcript-derived field, it is excluded from the incremental-scan parity check (alongside actor/harness/mode), since an incremental row legitimately preserves a recorded origin a from-scratch reparse of rewritten content cannot re-derive. This is what fixed the "origin version was not recorded" fallback.

Prefer-device, fall back to local. Resume runs on the recorded owning device; when that device is genuinely unreachable it falls back to a LOUD local /continue replay from the synced mirror rather than dead-ending (resumeLocalFallbackSource rewrites machine to self so the delegated run resolves locally). Both unreachable shapes trigger it: runOnPeerno-target (the device is not a dialable registered device) and runOnPeerunreachable (a registered device whose SSH connection failed — asleep/offline, ssh exit 255, classified by peerHopOutcome). Resolving ok on a connect failure was the bug that silently no-op'd resume against an offline registered box. Safe against the RUSH-2022 silent-fork hazard by precondition: the owner was proven unreachable, so there is no live process to fork.

Configuration surface

All persistent configuration that affects how agents run — default model, mode, effort, tier overrides, interactive host, browser profile, and per-device limits — lives under one command barrel:

agents config list
agents config get <key>
agents config set <key> <value>
agents config unset <key>

Keys use agent@version as the canonical harness identifier. Examples:

agents config set run.claude@*.model best
agents config set run.claude@*.tier.best claude-opus-4-8
agents config set run.claude@2.1.45.model claude-opus-4-8
agents config set run.claude@*.mode auto
agents config set run.claude@*.effort high
agents config set interactive.host zion
agents browser use work
agents config set auto.pool workers
agents config set devices.mac-mini.role worker
agents config set devices.mac-mini.max-agents 4
agents config set devices.mac-mini.scheduler off
agents config set devices.mac-mini.tmux off

The new command is a facade over the existing YAML storage (run.defaults, model.tiers, config.interactiveHost, defaultBrowserProfile, and deviceConfig). Fleet sync behavior is unchanged.

devices.<name>.tmux (stored as tmux.enabled) defaults off, so a LOCAL interactive agents run launch spawns the agent directly. Turn it on for a device to wrap eligible local launches in the shared-socket tmux session and give each agent an exact %pane address for agents message, injection, and agents focus. The setting is machine-local and cannot be set for a peer.

It does not govern a run dispatched here over --device (RUSH-3125). That run's stdio is an ssh link, so without the wrap a blink SIGHUPs the agent and the in-flight turn is lost — and lib/hosts/reconnect.ts re-attaches on the premise that it survived. Durability is not the local operator's mouse/clipboard/ scrollback preference, so the two are separate concerns: the interactive dispatcher exports AGENTS_REMOTE_INTERACTIVE=1 (REMOTE_INTERACTIVE_ENV, src/lib/types.ts) and the peer wraps on it regardless of tmux.enabled. A remote interactive run on a box with no tmux installed is refused (undurable) rather than started as something a blink would kill.

--no-tmux, --raw, and AGENTS_NO_TMUX=1 remain per-run opt-outs and beat the durability rule too, so the escape hatch keeps working over --device. The gate is resolveTmuxWrap (src/lib/exec.ts) — three outcomes (wrap / bare / undurable), reading isTmuxEnabled() and the marker.

devices.<name>.role (stored as role) says what a device is for fleet-wide — worker (agents run here) or personal (a machine you sit at). agents devices role <name> <role> is the task-shaped spelling. Marking any device worker turns automatic placement into an allowlist: --device auto then picks only from the marked workers, in every caller (run, teams, ssh auto, the generic --device auto passthrough, and the AGI EXT launch commands, which resolve placement through the CLI rather than scoring devices themselves); a personal or desktop box is never picked automatically. The rule is one function — filterAutoPool in src/lib/devices/pool.ts — read by listOnlineDeviceNames, and auto.pool (workers by default, or all) turns the allowlist off. When roles leave the pool empty, both resolvers throw (formatEmptyAutoPoolError): a null host means "run locally", which on a personal/desktop box is the outcome the mark exists to prevent. Unlike the machine-local keys, role is shared: it lives in that device's tracked devices/<name>/agents.yaml config.role and syncs with agents repo push/pull, because every box has to agree on where agents may land. The vocabulary is worker | personal | desktop. desktop is a headed always-on box — the release/credential home, e.g. a Mac mini: it is NOT headless fan-out capacity, so like personal it is excluded from --device auto (both are in NEVER_AUTO), but it IS a headed box with a real interactive login, so for the Claude auth strategy it sits in the SAME "headed" bucket as personal (isHeadedDeviceRole, device-config.ts) — it authenticates from its own per-version login, never the worker setup-token. Which credential a run injects is keyed on DEVICE ROLE, not run mode (isHeadedDeviceRole, not ctx.interactive, in applyExecConfigEnv): a worker authenticates EVERY run — interactive dispatched TUI or headless — from its user:inference setup-token (the non-interactive worker credential, minted via agents auth mint claude), while a headed box uses its native user:profile login for both. Keying on run mode instead is the PHNX-3502 (worker --interactive --device → login screen, the setup-token sitting unused) / RUSH-2395 (headless laptop run → hijacked the human's login) bug pair. Full model: cli/docs/credential-management.md §Which credential a run injects. The one display consequence: because a personal box is by definition the interactive seat, agents devices list folds the ★ interactive star INTO the personal role rather than printing both (the star still shows for a non-personal box pinned as interactive.host).

auto-launch.enabled / auto-launch.preferred are the per-device switches that modulate that SAME filterAutoPool pool — one placement rule, not a second one. agents devices disable <name> sets auto-launch.enabled = false and drops the box from the pool the same way a personal/desktop role does, so it leaves EVERY automatic-placement path (run, teams, ssh auto, the AGI EXT launch commands) at once; agents devices enable <name> (the default) restores it. agents devices prefer <name> sets auto-launch.preferred = true, which does NOT narrow the pool — it BOOSTS the box in the ranker: autoLaunchPreferredSet (pool.ts) feeds pickBestDevice (teams/scheduler.ts), which ranks a preferred device ahead of its load-equal peers, after the signed-in tier and before load, so the boost overrides load-based ordering without overriding hard health; agents devices unprefer <name> removes it. Both flags are shared device-scope keys living in devices/<name>/agents.yaml config.autoLaunch{Enabled,Preferred} (a fleet-wide default rides fleet.defaults.config, set with --fleet), so they sync with agents repo push/pull. The four verbs are task-shaped forwarding spellings for agents devices config <name> auto-launch.{enabled,preferred} <on|off> — the one canonical per-device settings surface — so they print a deprecation-style "running that for you" notice and defer to it. The menu bar surfaces auto-launch.preferred as a per-device ★ Favorite / Unfavorite toggle (writing that same canonical devices config … auto-launch.preferred on|off, no separate store): a favorited device shows a ★ and sorts directly below the current machine, above the rest, in the DEVICES section (PHNX-2376).

devices.<name>.description (stored as description) is the free-text sibling of role: one line saying what the box is FOR — "gpu box — cuda 12.4", "release runner" — where role is the two-value placement switch. Like role it is shared: it lives in the device's tracked devices/<name>/agents.yaml config.description and syncs with agents repo push/pull, and any box may set it for any device. agents devices describe <name> <text> is the task-shaped spelling (thin sugar over agents devices config <name> description — one store, two names), and the default agents devices list renders it as the tail column next to a spec cell (cores / total RAM / total disk) and load, mem, disk used columns (RUSH-3062). It is validated as a single line capped at 80 characters — a newline or an over-long value is rejected loudly, never truncated. It is NOT notes: notes stays an appended list of long-form operator scratch that is never shown in device listings.

agents devices ignored lists the tailnet nodes dismissed from auto-discovery — each with when it was dismissed and which machine dismissed it, read from the tracked fleet.ignored block that agents devices ignore writes and agents repo push/pull syncs. The agents devices list footer names how many are hidden so a dismissed box is never silently absent; agents devices unignore <name> puts one back (RUSH-3062).

Native-account labels (agents accounts label) are the same kind of fleet-wide fact: they bind to a stable (agent, identityKey) (email / org key), not a device or a version. They live on the central accounts.native rows in ~/.agents/agents.yaml — already classified central and synced by agents repo push/pull — so codex#personal selects the same login on every box. A git merge of two independently labeled boxes can union two UUID rows for one identity; accounts remove / rename / label operate on every matching row so a sibling cannot silently survive.

interactive.host is a user-level preference: it lives in central ~/.agents/agents.yaml under config.interactiveHost, syncs fleet-wide via agents repo push/pull, and answers "which device shows me artifacts?" It is intentionally not a per-device key. To see it in the per-device view, use agents devices config <name> --inherited.

The old commands still work but are deprecated and print a warning pointing to agents config:

  • agents models tieragents config set run.<agent@version>.tier.<tier>
  • agents devices set-interactiveagents config set interactive.host <name>
  • agents devices configureagents config set devices.<name>.<key>
  • agents browser profiles set-defaultagents browser use <name>

Implementation: src/commands/config.ts with key parsing in src/lib/config-keys.ts. Per-device config helpers live in src/lib/device-config.ts.

Supported harnesses

The supported harnesses are the entries in the AGENTS registry (src/lib/agents.ts) — the canonical list, gated through supports(); the full id union is AgentId (src/lib/types.ts). The table below is a snapshot of their per-harness capabilities — keep it in sync with the registry. Prioritized (first-class): Claude Code, Codex CLI, Kimi CLI, Antigravity CLI, Grok CLI, OpenCode — features target these six first.

Harnessidhooksmcpallowlistskillscommandspluginssubagentsworkflows
★ Claude Codeclaude
★ Codex CLIcodex≥0.116<0.117≥0.128≥0.117
★ Kimi CLIkimi≥0.29.0
★ Antigravity CLIantigravity≥1.0.16≥1.0.6
★ Grok CLIgrok≥0.2.111
★ OpenCodeopencode≥0.3.130≥1.1.1
Cursorcursor≥2026.1.22
OpenClawopenclaw
Copilotcopilot≥0.0.353
Ampamp
Kirokiro≥0.10≥2.8≥1.23
Goosegoose≥1.34≥1.25
Droiddroid≥0.57.5≥0.26
Hermeshermes≥0.11
Muse Codemuse
Pi (Oh My Pi)pi
Warp Agent CLIwarp

✓ = supported · — = not · version cell = only within that range (out-of-range = skipped silently). src/lib/agents.ts is canonical — keep this snapshot in sync. workflows is claude/kimi/goose/antigravity (≥1.0.6, written to the shared HOME-global ~/.gemini/config/global_workflows/, not a per-version home), openclaw (Lobster .lobster files under .openclaw/workflows/), and grok (≥0.2.111, native Rhai under .grok/workflows/); mcp is universal; allowlist is claude/cursor/opencode/antigravity/grok/kimi/kiro/droid/openclaw/copilot (Copilot writes per-location approvals to ~/.copilot/permissions-config.json; Goose is deliberately NOT allowlist-capable — its permission.yaml gates whole tools (developer__shell, developer__text_editor), so a canonical rule set could not be expressed or read back faithfully; OpenClaw is tool-level only — blanket rules map to ~/.openclaw/openclaw.json tools.alsoAllow/tools.deny, sub-command patterns skipped); subagents is claude/codex/kiro/kimi (≥0.29.0, Claude-shaped <name>.md in ~/.kimi-code/agents/; older kimi-code compiles its agent profiles into the bundle with no filesystem loader)/grok/openclaw/droid/copilot/antigravity/cursor (≥2026.1.22)/pi. Pi (Oh My Pi, omp) is Claude-compatible: it natively reads .claude/commands, .mcp.json, and Claude-shaped subagents, and keeps its own native resources under ~/.omp/agent/ (skills, commands, subagents agents/, AGENTS.md context, .mcp.json). mcp covers stdio + http + headers. hooks/allowlist/plugins are OFF: omp hooks are per-tool JS/TS extension modules (not event->shell-command registrations), approval is per-TOOL only (tools.approval, no command/path patterns), and plugins are npm/TS modules (not the Claude marketplace manifest). Its cross-provider model catalog (OpenRouter/OpenAI/Anthropic/xAI/DeepSeek/…) surfaces in agents view / agents models pi via omp models --json. Warp Agent CLI (oz) is the coding-agent CLI on Warp's Oz platform (the shared Warp binary invoked via the oz symlink). Install is self-updating via brew install --cask oz (macOS) / the oz-stable apt|yum|pacman package (Linux); config lives under ~/.warp/, the rules/context file is AGENTS.md, and auth is oz login (browser OAuth) or a WARP_API_KEY token for headless/CI (oz api-key create). Headless run is oz agent run --prompt "<task>" [--model <id>]; autonomy is governed by the selected agent profile (--profile), not a per-run permission flag, so the single edit mode maps to no flags (mirrors Hermes). mcp covers stdio + http + headers via the Claude .mcp.json schema at ~/.warp/.mcp.json (project <root>/.warp/.mcp.json); skills come from --skill + oz agent skills. hooks/allowlist/commands/plugins/subagents/workflows/memory are OFF: Oz has no event→shell hook registration, its permissions are profile-based (not a Claude tool allow/deny list), slash-commands are native/server-managed, and cloud agents/profiles are server-side (no installable subagent dir). Warp is intentionally absent from SESSION_AGENTS — Oz stores conversations server-side (retrieved with auth via oz run conversation get <id>), so there is no local transcript for agents sessions to index — and it exposes no usage/limits endpoint, so agents view shows no usage bar for it. Gemini is hard-deprecated. Keep the legacy gemini id only for parsing old sessions/config; agents add gemini, agents import gemini, and agents sync gemini fail and point users to Antigravity.

Codex's Linux sandbox needs unprivileged user namespaces (PHNX-3285)

Codex ≥0.146 is the only harness that sandboxes its own tool calls on Linux with a bundled bubblewrap — it extracts codex-linux-sandbox + bwrap per run and sets up mounts inside a fresh unprivileged user namespace (--unshare-user, then a /proc/self/uid_map write) for both read-only and workspace-write modes (only skip = --dangerously-bypass-approvals-and-sandbox uses no bwrap). The legacy Landlock backend is gone (use_linux_sandbox_bwrap is removed, use_legacy_landlock panics under the permission-profile model), so bwrap is the only Linux sandbox path. grok/kimi/cursor/etc. do not sandbox this way, so this is codex-specific.

Ubuntu 23.10+ ships kernel.apparmor_restrict_unprivileged_userns=1, which denies that userns to an unconfined binary — bwrap then dies with bwrap: setting up uid map: Permission denied, and a headless codex run (an agents teams codex teammate, or agents run codex — always headless + sandboxed) lands zero tools while still reporting a completed turn. spawnAgent preflights this before the spawn (codexSandboxPreflight in src/lib/exec.ts, probing src/lib/linux-userns.ts) and fails loud with the one-time fix instead of silently under-delivering. Scope is deliberate: codex + Linux + headless + a sandboxed (non-skip) mode only — an interactive TUI surfaces the bwrap error itself, --mode skip uses no sandbox, and macOS/Windows never hit it. The intended auto = workspace-write + approval_policy=never config is never weakened.

The one-time fix keeps the sandbox intact — re-enable unprivileged userns per box: scripts/enable-codex-sandbox.sh writes the kernel.apparmor_restrict_unprivileged_userns=0 sysctl drop-in, applies it, and re-probes to confirm it took. Run it once per fleet worker (sudo bash cli/scripts/enable-codex-sandbox.sh, or fan out with agents ssh <box> 'sudo bash -s' < cli/scripts/enable-codex-sandbox.sh).

Source layout

src/
  index.ts             # CLI entry (commander.js)
  commands/            # User-facing subcommands (one file — or a `<cmd>-*.ts` family, e.g. the `sessions*.ts` family — per `agents <cmd>`)
  lib/
    state.ts           # Path constants; agents.yaml read/write (serializeCentral preserves comments)
    manifest.ts        # Project/user agents.yaml Manifest read/write (comment-preserving Document round-trip; used by mcp add, etc.)
    resources.ts       # resolveResource() / listResources() — layered resolution
    capabilities.ts    # supports() — the per-agent write gate
    agents.ts          # re-exports agent-spec/agents.ts (per-agent capability table)
    agent-spec/        # version-resolution engine + AGENTS table
    subagents-registry.ts  # SUBAGENT_TARGETS — declarative per-agent subagent shape (dir/layout/transform); generic install/list/remove engine
    installations/     # versions.ts (install, remove, syncResourcesToVersion), migrate.ts (one-shot idempotent migrations), store/resolve/strategies, shims.ts (shim generation, config symlink switching)
    hooks/             # hooks.yaml parser + per-agent registrar (install.ts), `matches:` evaluator (match.ts), cache/profile adapters
    browser/           # browser IPC service hosted by the shared daemon + existing CDP connection pool; browser clients may enable/disable this service but never stop/restart the shared daemon; registry.ts is the leaf (who declared what — never whether this machine should); resolve-target.ts is three outcomes (local / tunnel / loud undeclared); task-index.ts binds --device at start so later verbs resolve the task; ipc.ts owns one-shot and persistent socket clients, stream.ts owns the NDJSON action loop; hygiene.ts is the abandoned-task reaper (session-dead + idle, RUSH-2622) the daemon's 5-min tick and `agents browser prune` (alias `gc`) both call
    monitors/          # event-triggered watchers; architecture in docs/automation.md
    projects.ts        # named multi-repo definitions and status projection; domain model in docs/concepts.md
    project-pull.ts    # fleet pull with fast-forward, clean-tree, branch, and repository-identity guards
    session/           # `agents sessions` READER — discovery/parse/render of agent transcripts; also `migrate-targets.ts` (the `sessions migrate` target scorer); `db.ts` `queryResourceUsageStats`/`backfillResourceUsage` back `agents sessions stats` + `sessions backfill resources` (skill/command usage rollup, session_resource_usage + resource_scan_ledger); `claude-accounts.ts` attributes each Claude transcript to the account that produced it (account_key) and `insights.ts` extracts the cached multi-harness friction/correction/automation facets behind `agents sessions insights` (`agents insights` alias) — including a shell-command-by-binary breakdown (`bashCommands`/`bashCommandFailures`, keyed by `bash-command.ts`'s `bucketKey`) that splits the flat `Bash` tool count into `git commit`/`gh pr`/`agents ssh`/… so the tool mix and failed-tool loops name the actual command, not just the harness tool
    terminal/          # Terminal launch engine — tab/split in iTerm/Ghostty/tmux/Terminal.app, local or --device;
                       #   preferred.ts resolves WHICH terminal for a GUI caller (from live sessions' host app)
    cloud/             # Provider registry (Rush / Codex / Factory / Antigravity)
    teams/             # `agents teams` orchestration
    computer/          # `agents computer` client (computer-rpc.ts → native/computer-mac Unix socket; ssh-tunnel.ts → native/computer-win over ssh -L) plus dispatch/download/loop
    menubar/           # Menu-bar helper installer (source in ../menubar)
    profiles.ts        # Host CLI + endpoint + model bundles

Note: src/lib/session/ here is the transcript reader. The live-session writer is a separate package, packages/session-tracker — different data, different consumer; see its AGENTS.md.

agents sessions preview architecture (map before you touch it)

The interactive UI is three picker variants in src/lib/picker.ts: itemPicker (single-select, space toggles preview), dynamicPicker (async data source, used by the session browser, tab toggles preview), and a multi-select variant. All render a right/bottom preview pane built by buildPreview(session) in src/commands/sessions-picker.ts — a header (title/agent/model/cwd/ticket/PR) plus formatCompactPreview's verb-led rows (RUSH-2757): Asked (originating prompt), Doing (checklist/team/sub-agents), Made (file deltas, artifacts, plan, PR), Health (errors + test verdict), Cost (msgs/tokens + tool mix), Latest (the full last message, wrapped), and one width-capped Details ▸ fold (session id, skills, plugins, hooks, links, dirs, repos). agents sessions preview <uuid-or-prefix> uses the same card without the picker. ID-shaped selectors go through the indexed fleet resolver, remote cards render on their owning peer, and the normalized digest is cached in SQLite against the transcript's actual mtime + size. Live status is deliberately outside that durable digest and expires after 15 seconds through session-cache.ts.

Indexing is lazy — only discoverSessions writes the index — so a session THIS box just started is "running" in --active before it is indexed. The id resolver (computeLocalMetadataMatches in sessions.ts) therefore unions the indexed rows with the LIVE registry on a cold id miss, so preview/resume/focus resolve a running session with no transcript row yet (the fan-out peer answers from the same union, so it works cross-device too — SES-9b). The daemon keeps the index current within seconds via runSessionIndexWarmTick, and the cold-miss repair waits for a concurrent scan rather than returning a stale read (discoverSessions({ waitForScan }) — SES-9c). This is why a running session no longer reads "No session matching" during the index-lag window (RUSH-2682).

Routing lives in src/commands/sessions.ts: isBareBrowserListing (+hasNoBrowserDisqualifyingFlags) gates the bare fleet-wide listing to the rich runSessionBrowser (src/commands/sessions-browser.ts); a query/filter falls through to pickSessionInteractivesessionPicker. --flat/--tree/--json/--no-interactive print non-interactive views with no preview. PICKER_RECENT_COUNT = 15 caps the picker's list rows.

Gotcha: the preview pane has no guaranteed heightavailablePreviewRows = terminalRows() - fixedRows (picker.ts), and limitPreviewHeight returns '' when that collapses, so the preview can silently vanish on a full/short terminal (the RUSH-2198 bug). See the §Contracts §Sessions spec for the non-empty-preview invariant (SES-8).

Resume is machine-bound — check the owner before you start a harness

Reading and resuming follow different machines, and conflating them is the bug. Reading follows the FILE — a synced mirror with a readable filePath is on this disk and previews locally, even when machine names its owner. Otherwise, a row whose machine names another box must be read on that peer. _remote is one explicit signal, not the gate: host-dispatch index rows and live-registry rows can name a peer without carrying it. transcriptOnPeerOf in sessions-picker.ts is the one predicate used by picker previews, direct preview <id>, and sessions <id>. Resuming follows the HARNESS STATE, which is on the owning machine whatever the transcript's location. A mirror is therefore readable and NOT resumable, and that is the trap: nothing fails until the agent is asked to continue a conversation it has never seen, and sessions-resume.ts's fs.existsSync(cwd) fallback then quietly resumed in process.cwd() (RUSH-2022, PHNX-3481).

sessionOwnerDevice (src/lib/session/resume-owner.ts) is the one answer to "may this resume run here?". Every path that starts a harness from a picked row consults it first: agents sessions resume and the agents sessions picker hop to the owner, and sessions attach hops as an attach (its detach record and the headless process it stops are both on the owner — hopping as a bare resume would skip the stop and leave two processes on one transcript). The batch sessions resume mostly inherits it for free: every TAB it opens runs the canonical agents sessions resume <id> (lib/session/resume-command.ts), whose docblock already promised source-device routing — this is what makes that true. Its no-tab-backend path (inplace, which any Linux box in a plain ssh shell lands on) never runs that command, so it routes explicitly via resumeOnOwnerIfRemote. resumeSessionInPlace is the LOCAL takeover and fails loud if it is handed a peer-owned session, since reaching it with one means a caller skipped its routing step.

The hop uses runOnPeer (src/lib/session/remote/remote-list.ts), not the --device passthrough. Two reasons: the passthrough re-discovers locally and dead-ends for a session that exists only on the peer, and it marks the run AGENTS_FLEET_REMOTE — a one-shot command may carry that consent marker, but a resumed session would inherit it for its whole life and agents browser start inside it would be refused as a cross-machine drive.

The signal is only as good as what wrote it: machine on a host-dispatched run is stamped by src/lib/hosts/session-index.ts from the dispatch host. Any new writer of an agents run --device-shaped row must set it, or the index will claim the dispatching box.

Native helpers (downloaded on demand, never bundled)

No native binary ships inside this package's npm tarball (RUSH-3100). Every helper is a signed + notarized GitHub release asset on its OWN tag, downloaded and verified on demand — see src/lib/helper-versions.ts for the per-helper version floors. That is what lets an ordinary CLI release be produced without a signing Mac.

HelperSourceShips in tarball?Resolver
Keychain brokersrc/lib/secrets/keychain-helper.swiftbin/Agents CLI.appNo (RUSH-3100) — signed + notarized Agents_CLI.app.zip GitHub release asset on the helper's own keychain/v<x.y.z> tag, downloaded on demand. The asset name is underscored, not spaced: GitHub rewrites a space to a dot at upload, so Agents CLI.app.zip was served as Agents.CLI.app.zip and every client 404'd. No DR pin (Team + notarization only, unlike the menu-bar helper): keychain items are gated by the access-group entitlement + biometry, not a DR-keyed grant.src/lib/secrets/install-helper.ts, src/lib/secrets/download-keychain.ts (shared machinery in src/lib/helper-download.ts)
Menu-bar helpermenubar/ (SwiftPM) → bin/MenubarHelper.appNo (RUSH-3100) — signed + notarized MenubarHelper.app.zip GitHub release asset on the helper's own menubar/v<x.y.z> tag, downloaded on demandsrc/lib/menubar/install-menubar.ts, src/lib/menubar/download-menubar.ts (shared machinery in src/lib/helper-download.ts)
Standalone CLI binarysrc/bun build --compilebin/agents-macosNo — dropped from the tarball (RUSH-3026); macOS installs fall back to the JS entrypoint until it returns as a per-release GitHub assetscripts/postinstall.js
computer-mac../../native/computer-macNo — signed + notarized GitHub release asset on its own computer-mac/v<x.y.z> tag, downloaded on demandsrc/lib/computer/computer-rpc.ts, src/lib/computer/download.ts (shared machinery in src/lib/helper-download.ts)
computer-win../../native/computer-winNo — computer-helper-win.exe GitHub release asset on its own computer-win/v<x.y.z> tag, downloaded on demandsrc/lib/computer/ssh-tunnel.ts

Path math: compiled resolvers run from cli/dist/lib/…. Files still in dist/lib/ reach repo-root native/ in 4 hops (../../../../native/…); files in dist/lib/computer/ need 5 hops. The co-located menubar/ is 3 hops up (../../../menubar/dist/…) because it moved into cli with the CLI. Recompute depth if you move files — don't blind-replace.

Build, test, dev

bun install && bun run build && bun test

Tests are *.test.ts next to source; integration in tests/. Every PR to main runs the real suite cheaply on Linux — test (../../.github/workflows/tests.yml) plus gitleaks; those two are the required checks. The full cross-platform matrix (ubuntu + macOS + Windows × Node 22/24, ci.yml) runs nightly plus manual workflow_dispatch — deliberately off the release path. It used to fire on release/** branches, but that was 16-53 min of macOS/Windows-billed work on every release while gating nothing (it is not a required check, and release.sh gates on the exact-tree attestation, never this matrix). Cross-platform regressions are caught on the nightly lane instead; a risky release can still run it on demand via workflow_dispatch. CI runs from cli via defaults.run.working-directory.

Live Windows --device e2e (opt-in): src/lib/computer/ssh-tunnel.e2e.test.ts and src/lib/browser/drivers/ssh.e2e.test.ts drive a real Windows box end-to-end (exe push + LOGON task, tunnel + RPC, screenshot, type/get-text round-trip, remote browser launch/stop). Gated on AGENTS_TEST_WIN_HOST=<registered device>; both suites skip cleanly when the var is unset, so CI needs no Windows runner.

Local dev build: scripts/install.sh --skip-tests builds the working tree, installs it at $HOME/.local/agents-cli-dev/, and exposes it as $HOME/.local/bin/agents-dev (plus ag-dev). Drive it by name — agents-dev sessions --active. Version stamps as 0.0.0-dev.<sha>[-dirty].

The production command is never created or overwritten: the script must not write $HOME/.local/bin/{agents,ag,browser}, and it deletes any such link an older revision of itself left pointing into the dev prefix (including a dangling one, which is what a cleaned dev prefix leaves behind). A dev build that answered to agents made PATH order decide which code ran — see the root AGENTS.md §Never install a dev build over the user's agents.

The routines daemon is shared (secrets broker, browser IPC, scheduler), so the install leaves it on production code. --bounce-daemon restarts it onto the dev build when you need that, and says plainly that it changes what the user's everyday agents talks to.

Bin entrypoints need chmod 755. scripts/build.sh chmods every package.json#bin entry after tsc emits. Newer npm preserves tarball file mode and does NOT auto-chmod — 644 surfaces as zsh: permission denied: agents.

The files allowlist in package.json is a whitelist — only dist/** (JS/d.ts/JSON/sh) and the postinstall scripts + README/LICENSE ship. No native binary is in it (RUSH-3100). Nothing from apps/, native/, or sibling packages/ can leak into the tarball.

Releasing

Self-routing, zero-config. Run it from ANY fleet box and ANY checkout state — no variables to set, no Touch ID, no hand-moved credentials, and no requirement that the caller be on a clean main:

scripts/release.sh <version>                      # dry-run: bump, type-check, tarball preview, detected state
scripts/release.sh <version> --apply              # tests on an auto-picked fleet worker -> PR + CI -> merge + tag -> build/sign/publish on the home base (mac-mini)
scripts/release.sh <version> --apply --device <mac>  # sign/publish on <mac> when mac-mini is down -- <mac> must ALREADY be a provisioned signing home base (see below)
scripts/release.sh <version> --apply --deploy-worker off  # publish only; skip the post-publish share-Worker redeploy

The release has three self-selected homes and prints a [n/6] phase tracker, each phase labeled with the box it runs on and a ✓/✗ result:

WorkRuns onHow it's chosen
Orchestrate: bump, changelog, PR, taga detached worktree on the box you invoked it onfresh origin/<default> under .agents/worktrees/release-v<version>-<pid>
CI / tests (Linux)an auto-picked fleet workerscripts/test.sh resolves it through agents devices pick — the least-loaded reachable POSIX box in the same auto pool agents run --device auto uses, so role=worker/role=personal marks govern it. --test-device <box> pins one; --crabbox still routes to a disposable sandbox.sh workspace. --shard <n> fans the suite across n workers (minimum 2 — for one worker use --device auto), and --devices a,b,c names them explicitly instead of auto-picking. The minimum-2 floor applies to both: a single-name --devices is refused the same way, since a one-shard fan-out is --device auto through far more machinery. Sharding is what meets the release latency target: the suite is throughput-bound, so wall time is CPU/workers. Dynamic either way, never a hardcoded or release-exclusive instance
Promote attested tgz + npm publish (+ computer-helper re-attach only with --with-helpers)the home base (any OS — promote-only, RUSH-3026)--device <name> in release.sh, defaulting to mac-mini; the script detects if it's already there (scutil --get LocalHostName / hostname -s), else reaches it over ssh

The home base holds the npm publish token + gh auth. It defaults to mac-mini and is overridable with --device <name>. Not an env var: a flag with a default. Since RUSH-3026 the home-base phase is promote-only (download the attested tarball, verify, install-smoke, npm publish) — nothing on it signs or notarizes, so any OS works as the home base, Linux included. Re-attaching the reused helper zip and verifying the helper input-digest manifest are opt-in (--with-helpers): an ordinary release publishes the CLI and nothing else, since helpers resolve from their own tags (src/lib/helper-versions.ts) and a manifest check would otherwise abort a good CLI release whenever a helper's sources moved. assert_promote_home_base preflights it (tools + gh auth + a headlessly readable npmjs.com NPM_TOKEN) before the release's first mutation. Helper signing is a separate, source-change-only path and still needs a provisioned Mac. The test worker is not hardcoded.

A release redeploys the managed share OG-cover Worker so prod can't drift from the shipped template (PHNX-3403). The Worker that renders share preview cards was deployed only by a manual agents artifacts share update run, decoupled from release — so a release could ship a worker-template.ts change while the deployed Worker stayed stale (the gap that made PHNX-2835 look shipped while every new share still 404'd its cover). --deploy-worker <auto|on|off> (default auto) closes it: after publish, in the home-base phase, deploy_share_worker runs the just-published @phnx-labs/agents-cli@<version> (via a pinned npx, never the box's installed agents, so the deployed Worker matches the released source) as agents artifacts share update --bundle cloudflare.com. auto first runs share update --check — a pure local render+hash that needs no Cloudflare credentials — and deploys only when the shipped template differs from what the endpoint deployed; on always redeploys; off opts out. The deploy uses bundle cloudflare.com (the name the home base holds, not the CLI default cloudflare), agents secrets exec resolves it headlessly like the npm token, and the promote preflight verifies the share endpoint + that token on the home base before publishing (so a Worker-touching release can't publish then fail to deploy). A deploy failure fails the release loud with the exact manual fallback. It is idempotent — the already-published early return redeploys too — so a re-run after a publish/deploy split finishes the deploy.

A --device fallback must ALREADY be a provisioned signing home base — it is not turnkey. Signing + notarizing + publishing needs, on that box: the Developer ID Application identity in a headless-unlockable keychain (rush-signing.keychain-db + ~/Library/Application Support/rush/signing.kcpass, the pass file that lets a headless SSH release unlock it — a cert that only appears after an interactive login does not count), and the apple.com (notarytool) and npmjs.com (publish token) secrets bundles. A box like zion typically has a Developer ID cert in its login keychain but none of the headless plumbing, so it cannot sign a release. Verify with the probe rather than assuming from this prosebash cli/scripts/signing-home-base-probe.sh is the authoritative test, and a box that has since been provisioned (dedicated keychain + signing.kcpass + the apple.com bundle) returns OK and can sign, whatever this paragraph says about it. Passing --device <unprovisioned> used to run the whole flow — merge the PR, push the tag — and only fail at the sign step, leaving a tagged-but-unpublished release (RUSH-2535; npm stuck at 1.22.35 with v1.22.36 tagged). release.sh now preflights the resolved home base BEFORE any mutation (scripts/signing-home-base-probe.sh, run on that box over agents ssh): an unprovisioned --device aborts at the preflight, before the test/PR/merge/tag phases, naming the exact gap, so a mac-mini outage no longer risks a half-finished release. cli/bin/embedded.provisionprofile is a committed input (commit 2567004b4) that self-heals — the preflight and the home-base phase both recover it from a freshly fetched origin/<default> ref when the box's own on-disk checkout predates that commit, so a brand-new home base never needs the profile hand-copied over (RUSH-2541). The keychain (Developer ID identity in a headless-unlockable keychain) and the apple.com/npmjs.com secrets bundles remain genuinely manual, per-machine provisioning steps — seed those first, then --device <that-mac> works.

The caller checkout is never mutated or gated. release.sh immediately fetches origin and re-enters the release from a detached, release-owned worktree at fresh origin/<default>. Version bumps, changelog folding, release-branch construction, CI orchestration, merging, and tagging happen there. The worktree is removed on every exit path, so a dirty shared main, an agent feature branch, or another branch already checking out main cannot block or contaminate a release. The isolated tree installs dependencies from its pinned lockfile; it does not borrow node_modules or staged files from the caller.

One releaser at a time — the lease. Because the script runs from any box, two agents on two machines could enter it at once; they then clobber the same release branch, tag, and publish, and the collision only surfaces at the publish gate (merged tree != built tree) once one of them has already merged and tagged. scripts/release-lease.sh holds exclusivity on origin — the only thing every box can agree on — by pushing an orphan commit to refs/release-lock/held. A second claimant's push can never be a fast-forward of the first's, so git's rejection is the failed lock acquisition: no polling, no second service.

scripts/release-lease.sh status     # unheld | held version=… holder=… age=…min holder-alive=yes|no|unknown
scripts/release-lease.sh claim <v>  # 0 = acquired, 1 = someone else is releasing
scripts/release-lease.sh renew      # prove this run is still alive
scripts/release-lease.sh verify     # 0 = still ours; fails CLOSED on any doubt
scripts/release-lease.sh release    # drop the lease this checkout claimed
scripts/release-lease.sh clear      # drop a lease with no live holder (any checkout)

release.sh claims it right after the confirmation (before the first mutation) and drops it from cleanup_all's trap on every exit path. Ownership is the lease commit sha, recorded in .git/release-lease.token — not the pid, so a release resumed by a second invocation can still drop its own lease, and a third agent can never drop one it did not claim.

The TTL is not "how long a release takes". It is "how long since the holder last proved it was alive" — a distinction that matters because a healthy release routinely outlives any sane TTL: the CI matrix alone has run 57 minutes, and release 1.20.77 took 186 minutes wall clock. So two things hold the invariant together:

  • Renewal. release.sh runs a background renewer for the whole release (renew every 10 minutes), so a live run's lease is never older than 10 minutes and cannot be reclaimed out from under it. The renewer is killed before the lease is dropped, so it can never re-push a lease that is being deleted.
  • verify before every irreversible step. require_lease gates the squash-merge, the tag, and the publish routing. It fails closed — no token, no ref, unreachable origin all mean "we cannot prove this is ours", so the release stops rather than merging alongside whoever holds it now.

A lease abandoned by a killed run stops being renewed, so it becomes reclaimable after RELEASE_LEASE_TTL minutes (default 30); reclaiming names the dead holder rather than silently overwriting it.

An externally killed run is detected, not just waited out (RUSH-2274). The TTL alone made a killed release indistinguishable from a healthy long one: for up to 30 minutes status read held while nothing was releasing. So the lease also records which process holds it — host, pid, and that pid's start time — and claim/clear/status probe it, reporting holder-alive=yes|no|unknown:

ProbeWhenWhat it licenses
deadwe are on the holder's box and that process is gonereclaim immediately, no TTL wait
alivethe recorded pid runs here with the recorded start timenever taken, at any age — stop that release instead
unknownthe holder is another box, or the lease predates these fieldsthe TTL, exactly as before

release.sh exports RELEASE_LEASE_HOLDER_PID=$$ so the recorded pid is the orchestrating release, not the 10-minutely renew shell (whose $$ is dead a second later — recording that would make every renewed lease read as abandoned). A lease with no recorded pid stays unknown, so a missing export degrades to the old TTL behaviour rather than to "instantly reclaimable". The start time is what makes dead safe to act on: a recycled pid would otherwise read as a live release forever. A zombie counts as dead — a SIGKILLed release whose parent never reaped it is still listed by ps, which is precisely the case this detects.

scripts/release-lease.sh clear drops such a lease without starting a release — the operator's unwedge path, since release only drops a lease this checkout claimed. It shares one predicate with claim, so it can never take a live holder's lease either.

Finish a stuck release before cutting a new one — with one exemption. release.sh refuses to start when an older v* tag exists that npm never received, and prints the re-run that finishes it. The single carve-out is a patch-from-main bump stepping over main's own version, because that stuck release cannot be finished at all: release.sh's catch-up guard rejects it and points at "cut the next patch", so without the exemption the two guards deadlock and nothing publishes (2026-08-10, npm at 1.22.35 with v1.22.36 tagged — its CI-tested tree predated the prepack version-gate fix, so its own npm publish rejected a correct binary). Only main's own version is dropped from the candidate set, and stuck-release.sh says so on stderr; any other stuck tag still blocks, under every bump kind. Without the guard, a release that died between tag and publish left the next run validating its bump against a registry that was behind, so it cut the next version and the gap widened by one every time — that is how npm sat at 1.20.78 while main carried 1.20.81.

A stuck EARLIER bump PR blocks the changelog fold, not just a stuck tag (PHNX-3084). The stuck-tag guard above is registry-vs-tag; this is its PR-side twin. The version-bump PR merges async/best-effort after publish (RUSH-2395), so a bump PR wedged on a CHANGELOG conflict leaves that version's .changelog/next/* fragments still queued on main — the drain only landed inside the unmerged branch commit. A later version releasing then re-reads those fragments and folds an earlier version's notes under the new version. The same-target STUCK_BUMP_PR retry only lands release/v<current-target>, so it never sees an OTHER version's PR. Before folding, release.sh detects any other open release/v* bump PR (scripts/release-other-bump-prs.sh, unit-tested) and fails loud with the exact gh pr merge to run first, rather than silently re-attributing the earlier version's release notes.

The privileged phase runs on the home base, always — from the TAGGED script. After the invoking box merges + tags (git + gh, which need that box's auth), release.sh routes build + sign + notarize + npm publish + computer-helper to mac-mini. Whether inline (you invoked it there) or over ssh, it first checks out v<version> into a throwaway worktree under .agents/worktrees/, then runs that worktree's cli/scripts/release.sh <version> --home-base-phase — so the script that publishes is the one carried by the release tag (with --home-base-phase + headless-sign-context.sh), never the home base's possibly- stale on-disk checkout. The worktree is removed on exit whether the phase succeeds or fails. --home-base-phase runs inside that worktree: it verifies the checked- out version == <version>, enters the headless context (scripts/headless-sign-context.sh — unlocks rush-signing.keychain-db + exports AGENTS_SECRETS_PASSPHRASE from the on-disk pass files, so codesign/notarytool and every agents secrets exec run with no Touch ID), builds + signs the artifacts, resolves the npm token on the home base (never borrowed to the trigger box), publishes, and pushes the computer- helper release asset. bun run build copies the signed helpers into dist/ on a presence gate ([ -d bin/… ]); prepack's sha gate is sha-tool-portable.

Tests: an auto-picked fleet worker for Linux; cross-platform runs nightly. The --apply flow runs the full suite on a worker before opening the PR; a failure prints the failing tests + the captured log path and halts before any PR/publish. That covers the Linux suite, and the exact-tree attestation is the functional proof the publish gates on. The cross-platform (macOS/Windows) matrix (ci.yml) is not on the release path — it runs nightly, not on the release PR, so a release no longer waits on it. Run it on demand via workflow_dispatch before a risky release. --skip-tests skips only the Linux suite run.

The attestation producer shards by default. release-attestation-produce.sh (the suite run that mints the attestation) now fans the ~13k-test suite across the fleet via test.sh --shard N instead of pinning one box — the suite is throughput-bound, so this is ~1/N the wall time (~269s on one box → ~31s on nine). N is resolved from the eligible workers agents devices pick reports, capped, and falls back to a single auto-picked box when fewer than two are eligible (test.sh --shard has no silent fallback, so the count is resolved before the call rather than demanded of a possibly-thin fleet). Each shard still runs vitest at --maxWorkers=2 --retry=2, so the RUSH-3015 per-box flake mitigation is unchanged — sharding adds machines, not per-box concurrency. Override with --test-shard <n> / --test-devices a,b,c, or pin one box with --test-device <box> / --test-here.

A release-tree attestation can inherit the suite instead of re-running it (PHNX-3237). A release runs the full suite twice — once for the default-branch tree, once for the chore(release) commit tree — even though the second differs from the first only by the version bump, the folded changelog, and the regenerated command-index. release-attestation-produce.sh --inherit-suite-from <base-attestation.json> mints the release-tree record from an already-green base without re-running the suite: it still bun run build + npm packs (so the recorded tarball is the real release tree's, carrying the new version), but the expensive suite run is inherited. The soundness gate is release-attestation.sh derive — it fails closed unless the tree diff between base and release touches only package.json, .changelog/**, CHANGELOG.md, and docs/command-index.{md,json} (the exact set release.sh stages), so a code change can never ride a stale pass. The derived record inherits the base's lockfile/policy/toolchain/suite identity, which the allowlist proves are byte- identical to the release tree's, so release.sh's require() still keys to it exactly. Inherit mode is incompatible with any --test-* flag (there is no suite to route). release.sh does not yet call this automatically — the release-tree attestation is still produced out of band — so use --inherit-suite-from when producing it to avoid the second full-suite run.

Idempotent re-runs. The script's git-scope reads use <ref>:cli/package.json (not root) since the package moved under cli. If a publish fails after the PR merges, rerun the same command: registry-truth short-circuits skip an already-published version, tag creation is idempotent against the verified release commit, and the catch-up guards (CI-tested-head match + merged-tree match + version match) refuse an unverified publish so later commits on main cannot leak into the already-versioned package.

scripts/remote-sign-mac.sh is no longer on the release path. The privileged phase builds signed artifacts directly on the home base. The script remains only for the narrow case of building + pulling back JUST the signed macOS artifacts from another Mac (no publish); it takes the same --device <name> flag as release.sh (default mac-mini), with no other env knobs or fleet discovery.

Provisioning the apple.com bundle on a headless sign host. A Linux-driven release offloads macOS signing to a sign host over SSH, which needs the apple.com secrets bundle on that host. Push it with the file backendagents secrets export apple.com --device <signer> --remote-backend file (no passphrase required — the remote keys it under a machine-local key it auto-provisions and reads it headlessly; AGENTS_SECRETS_PASSPHRASE is never forwarded) — not the default keychain backend: a macOS login keychain is locked under headless SSH, so a keychain-backed push lands the bundle metadata but no readable secret items (secrets export --device now read-back-verifies a keychain push and fails loudly if it didn't persist, pointing at this fix). --device / -D is the fleet routing flag (legacy --host is stripped but not registered) on the secrets remote commands. See docs/secrets.mdPushing to a headless sign host.

Why the tarball no longer needs a Mac (RUSH-3100). It used to bundle dist/lib/secrets/Agents CLI.app — a swiftc-compiled keychain helper, Developer-ID codesigned and notarized — and prepack (scripts/verify-keychain-helper.sh) refused to pack unless that signed binary matched a pinned sha. Since CI runners are Linux and cannot produce it, that gate, not the code, is what chained an ordinary release to a signing Mac. The bundles are gone from the tarball; the helper is a release asset on its own tag, downloaded and verified on demand. Both verify-*-helper.sh scripts are retained — they are still the right gate for a helper release — they just no longer block the CLI tarball. Rebuild a helper only when its own sources change; the input digest in scripts/release-manifest.sh is what decides that.

computer-mac records itself from its published release, not a local rebuild (PHNX-2943). Unlike keychain/menubar (which this producer builds+signs in the Darwin block), computer-mac is signed on the separate scripts/publish-computer-helper-mac.sh path and is never rebuilt in release-attestation-produce.sh. When its source drifts, the producer records the published computer-mac/v<floor> binary — but only after proving that binary was built from the current source: the publish script uploads a computer-mac-input-digest.txt sidecar naming the source it built from, and the producer records the row only when that equals the current source digest, verifying the downloaded zip against its .sha256 first. A mismatch, a missing sidecar (a pre-PHNX-2943 release), or an undownloadable release fails closed with the exact publish-computer-helper-mac.sh command — never binding a new source digest to an unproven binary. Before this, the producer just died on drift and the publish script recorded nothing, so the "republish then re-run" advice looped forever (hit live cutting 1.22.43).

Menu-bar helper (menubar/bin/MenubarHelper.app) is built and verified the same way — built into bin/, published as a release asset on its own menubar/v<x.y.z> tag, gated when that asset is cut by by scripts/verify-menubar-helper.sh (presence + codesign --verify + a stapled notarization ticket + a designated-requirement pin). The DR pin is what keeps the Accessibility grant alive across upgrades: macOS re-validates each new version against the requirement stored with the grant (identifier "com.phnx-labs.agents-menubar" … certificate leaf[subject.OU] = "2HTP252L87"), not the CDHash — so every re-signed release still satisfies it and the grant persists. The gate hard-fails a release whose DR drops the pinned bundle id or Team ID (a wrong/absent team, an ad-hoc signature, a CDHash-pinned DR), because that would silently revoke every user's grant and re-prompt them on the next paste. The compiled executable INSIDE the bundle is named "AGI Menu" (menubar/Package.swift's products: entry — RUSH-3101), separately from the bundle FOLDER, which keeps its historical name MenubarHelper.app. Before this, launchd's ProgramArguments exec'd the bundle's Mach-O directly (installedExecutablePath() in install-menubar.ts), bypassing LaunchServices name resolution, so macOS fell back to CFBundleExecutable — which read MenubarHelper — for both the Accessibility list row and the "would like to control this computer" prompt. Renaming ONLY the executable (not the bundle id, Team ID, or designated requirement above) fixes the label without touching what keeps the existing grant alive. Every basename-matching check moved together: classifyMenubarProcesses / installedExecutablePath (install-menubar.ts), SingleInstance.swift's liveHelperOwnsLock, and the HelperIdentity.swift constant the Swift side reads it from. It is Developer-ID signed and notarized + stapled (menubar/scripts/build.sh, run inside the release's agents secrets exec apple.com context): Gatekeeper on macOS 26+ rejects an un-notarized .app as "damaged" (crashing AppKit at launch), and the stapled ticket rides inside the bundle so it survives npm's tarball round-trip — so the installed helper launches with no per-machine re-signing (the old install-menubar.ts ad-hoc re-sign band-aid is gone; the launch guards now verify Gatekeeper acceptance and fail loud instead — RUSH-2134). Notarization is mandatory for any real (Developer-ID) build; an ad-hoc dev build can't be notarized and the prepack gate refuses to pack it. Keep it a separate bundle from the keychain app — a menu-bar crash must never take down the secret broker. Stage a freshly-built bin/MenubarHelper.app before any release or the menu bar ships code-only (the 1.20.22 bug the gate prevents).

Exactly one status item is an invariant, enforced in the helper. The bundle can be started from more than one place — launchd's KeepAlive service, a LaunchServices/open launch, a second agents menubar enable — so the helper takes an flock on ~/.agents/.cache/state/menubar.lock at launch (SingleInstance.swift) and holds the descriptor for its lifetime; a loser surfaces the incumbent's menu and exits 0. Do NOT re-derive liveness from a pid file or a ps scan — the kernel releases an flock however the holder dies, which a pid cannot express, and a process list cannot say which copy launchd will keep alive. On the CLI side, classifyMenubarProcesses returns live copies of the installed bundle as a LIST (own), never a boolean: collapsing them is what let a duplicate icon read as a healthy running: yes. agents menubar setup is the recovery path — it ends every live helper and re-kickstarts the service so the survivor is always launchd's.

Only the install that OWNS the helper may reinstall it. The startup self-heal (installMenubarLaunchAgentOnUpgrade, every darwin invocation) reinstalls when the version stamp drifted or the plist's baked entry names another install. Both of those record whichever copy acted last, so on a box with several agents-cli copies each one read the others' marks as drift and recopied the bundle over them — and recopying replaces the executable under the running helper, killing it, after which KeepAlive restarts it and the next copy repeats it. Observed: a new pid every 5-15s, 578 launches in one helper log, a status item that never stayed visible, and agents menubar status still saying running: yes because a pid always existed (#2109). mayInstallMenubarHelper gates it: the plist's AGENTS_ENTRY names the owner, and only the owner reinstalls freely. A same-install upgrade keeps its entry path, so npm update still lands normally.

Three escapes keep the gate from becoming a stuck state, which is how the first version of it regressed: (1) repairs are never gated — a missing helper executable or a Developer-ID heal proceeds from any install, since a bundle that isn't there cannot be contested and blocking it leaves the menu bar dead with no automatic recovery; (2) a non-owner takes over once the recorded owner is gone from disk — but, like escape (3), only a Developer-ID source may seize a healthy helper this way (an ad-hoc/dev build is refused, see caveat (a)); (3) otherwise a non-owner may still take over once per MENUBAR_TAKEOVER_COOLDOWN_MS (1h, stamped in .menubar-last-heal). Without (3) a stale-but-present copy — an old nvm node dir nobody runs — owns the plist forever while the user's actual daily driver upgrades and never heals again. The cooldown turns an every-invocation storm into at most one restart per hour while leaving every install able to make progress. agents menubar setup bypasses the gate entirely and stays the immediate manual fix.

Two caveats worth knowing before you tune any of this. (a) Escapes (2) AND (3) are both refused to a source bundle that is not Developer-ID signed: scripts/install.sh puts an ad-hoc dev build beside the npm global, and letting it recopy an un-notarized bundle over a good one both makes Gatekeeper reject the result as "damaged" (RUSH-2134) AND poisons the shipped helper's Accessibility grant — an ad-hoc signature carrying the production bundle id fails the code requirement macOS stored with the grant, so it revokes the grant and re-prompts on the next paste. This is why a dev build ALSO signs under a distinct com.phnx-labs.agents-menubar.dev id (menubar/scripts/build.sh), so even a running dev helper registers its own TCC entry rather than the production one. Refusing an ad-hoc healthy-helper takeover strands nothing: escape (1) still heals a genuinely broken (missing / ad-hoc-installed) helper from any source, which is the only real deadlock. (b) The cooldown bounds the loop but does not converge it: two installs that are both invoked regularly trade ownership every cooldown, so the helper restarts roughly hourly until one is removed. That is deliberate — the alternative is stranding one of them — and the real fix is a single install (#2147 expanded the multi-install banner beyond PATH to NVM, fnm, Volta, Bun, common npm prefixes, and the npm _npx cache). The banner also checks each copy for dist/lib/app-bundle-install.js; a copy without it is labelled an unsafe legacy helper installer and must be removed, because current code cannot make an older executable use the atomic installer it predates.

Do NOT "improve" this by comparing bundle content. It looks like the obvious gate and it does not work: the helper is rebuilt, re-signed and re-notarized on every release (menubar/scripts/build.sh via release.sh), so consecutive releases ship byte-different bundles from identical Swift source. Measured on 1.22.20/21/22 — same 2876288-byte executable, three different sha256s, and three different CDHashes (so stripping the CMS/timestamp blob doesn't rescue it either). Any digest gate reports "changed" for precisely the skew case it was meant to exempt. Ownership is the only signal here that is stable across independently-signed builds. Related: the secrets broker hit the same multi-install failure and answered it differently, by keeping a hot broker alive across version skew (shouldTeardownVersionSkewedBroker, src/lib/secrets/agent.ts; #435, PR #909) — same disease, and a third KeepAlive helper will need one of these two answers rather than a fresh rediscovery.

The lock fd is O_CLOEXEC, and acquire self-heals a stale lock. The lock is opened O_RDWR | O_CREAT | O_CLOEXEC so no spawned child can inherit the descriptor — a pre-fix doctor child that inherited it and orphaned at PPID 1 held the flock forever, and every relaunch then read "already running" and exited, so the menu bar stayed dead until reboot. O_CLOEXEC is the fd-level guarantee across every spawn path: ChildProcess.spawn sets only POSIX_SPAWN_SETPGROUP (no close-on-exec default), and the bare-Process one-shots (runDetached / runMonitored) are meant to outlive the helper — so the flag on the fd, not the spawn site, is what keeps the lock out of every child. The helper never execs itself, so it keeps the fd for life. And when the flock is held but no LIVE "AGI Menu" owns it — the lock-file pid is dead, or belongs to some other program by reuse (liveHelperOwnsLock checks liveness + proc_pidpath basename) — acquire reaps the leaked orphan and retries the lock once, instead of surfacing into the deadlock. Only a genuine live incumbent is ever surfaced, so a duplicate launch never reaps a live helper's in-flight children (the reap is reached only when the holder is provably not a live helper).

Every CLI child the helper spawns is bounded, group-killable, and reapable. The helper shells agents on a timer, and an unbounded Process there is not a slow menu — it is a machine-killer. doctor --json measures 136s on an idle box, the poll asked for it every 60s, and a helper that dies mid-call leaves the child reparented to launchd with nothing to reap it (plus the node -e probes that child forked). The deaths are not preventable from inside the app: NSApplication.shared segfaults in SLSNewConnection when WindowServer is too starved to hand out a connection, and KeepAlive restarts into another doctor. Observed: 38 orphaned doctors + 92 orphaned probes, ~13 of 18 cores, load 490. The property that made this fatal is accumulation, so the rule is scoped to what accumulates: every TIMER-DRIVEN, repeating CLI call MUST go through ChildProcess — that is the capture() path behind the cached refreshers (routines, recentSessions, activeSessions, doctorOverview, watchdog). A poller is the only thing that can stack 38 copies of itself.

User-initiated one-shots deliberately do NOTrunDetached, runMonitored, and runMonitoredWithInput keep a bare Process on purpose, because every one of their callers is a menu click (routines run/pause, devices register, open <url>, and the ticket-agent / quick-fix dispatches). Two reasons, and both would be violated by "bound everything": a deadline there would kill the user's headless agents run mid-work, and a fire-and-forget open/dispatch is supposed to outlive the helper. One click cannot stack, so there is nothing to accumulate. Do not "fix" these by routing them through ChildProcess — if a future caller makes one of them repeating, that caller is the bug.

ChildProcess holds three invariants:

  • Bounded. Every spawn carries a deadline (30s; ChildProcess.doctorTimeout 180s for doctor --json, above its real measured cost — a ceiling set below the true cost just makes every poll fail while still paying full CPU).
  • Killed as a group. The child is spawned as its own process-group leader (POSIX_SPAWN_SETPGROUP) so a timeout kill(-pgid)s the subtree. Signalling the pid alone is what left 92 probes running. Foundation's Process cannot set a process group — that is why this is posix_spawn and not Process.
  • Reaped by the NEXT launch. Live children are recorded in ~/.agents/.cache/state/menubar-children; reapOrphansFromPreviousLaunch() runs in main.swift before the first AppKit call, since the crash being recovered from happens inside that call. Do NOT move it after, and do NOT replace it with an exit handler — SIGSEGV runs none. The registry is a versioned, flock-serialized document: a launch intent and unguessable provenance token are persisted before posix_spawn, the token is inherited through the child environment and carried in a group-leading supervisor's argv (the process-table surface current macOS exposes), and PID/PGID/start- time/resolved executable/argv complete the record immediately after spawn. On the next launch, a token-marked PPID-1 process reconstructs an intent whose helper died in that spawn-to-registration window. Reaping acknowledges entries one at a time: unreadable or identity-ambiguous live records remain durable, and a signalled record is removed only after the process group is confirmed absent. A registry read or write error fails loud and never becomes an empty successful read. Legacy two-field records decode conservatively as unverifiable retained state.

Poll intervals must stay well above the call's real cost: StatusItemController.doctorRefreshInterval is 15 min against a 136s command (it was 60s — a >100% duty cycle). MENUBAR_CHILD_TEST=1 "AGI Menu" exercises all of it against real processes, including reaping a real surviving orphan and proving a spawned child never inherits the single-instance flock fd. Separately, doctor --json taking 136s on an idle machine is its own defect — the helper is now safe against it, not a reason to consider it acceptable.

These self-tests are a build gate now, not just manual modes. The helper's env-gated self-tests (MENUBAR_SINGLE_TEST, MENUBAR_CHILD_TEST, MENUBAR_GUARD_TEST, MENUBAR_ISSUE_TEST) are headless — they exit before the AppKit path (Guards.enforceForInteractiveLaunch) so they need no GUI or signing. menubar/scripts/test-menubar.sh runs all four against the just-built binary and build.sh invokes it before signing, so no helper artifact ships whose invariants regressed. Nothing ran these before — PR CI is Linux (can't build Swift) and prepack only checks the shipped bundle's signature — which is how the flock fd-inheritance deadlock escaped. Do NOT add MENUBAR_DUMP / MENUBAR_PROMPT_PREVIEW to the gate: those reach AppKit and need a GUI session.

Standalone agents binary (#315) — no longer in the tarball (RUSH-3026). The signed arm64 Mach-O (bun build --compile → Developer ID + hardened runtime + the JIT entitlement in scripts/bun-jit-entitlements.plist, notarized via scripts/sign-cli-binary.sh) embeds the release version, so shipping it inside the npm tarball forced a Mac rebuild + sign on every release — the single artifact that chained publishing to a provisioned Mac even when nothing native changed. It is dropped from package.json files and from the prepack gates; postinstall's existing run-probe falls back to the JS entrypoint when dist/bin/agents is absent, so macOS installs keep working (the pre-#315 status quo, at the cost of the EDR mitigation until the binary returns as a per-release GitHub asset — tracked in RUSH-3026). scripts/verify-cli-binary.sh and the signing machinery are retained for that asset path; a Mac producer still builds + signs the binary locally, it just no longer ships in the tarball.

The @swarmify/agents-cli shim is frozen at 1.19.x — do NOT "catch it up." It's a legacy re-export not published since v1.20.0; release.sh publishes only @phnx-labs. Bumping it would un-deprecate a retired package.

Conventions

  • Real services only — no mocking. Tests exercise the actual critical path.
  • agents repo push / pull operates on ~/.agents/ only. System updates ride npm update -g @phnx-labs/agents-cli.
  • No sensitive data in any DotAgents repo — use agents secrets (Keychain-backed).

Contracts (source-of-truth spec — read before touching sessions/secrets)

The major subsystems carry a normative contract in docs/specifications.md — what a human, an agent, or a downstream tool may rely on, written because features have regressed by quietly deviating from an unwritten contract. When code and the spec disagree, one is a bug; fix the drift. It uses RFC-2119 MUST/SHOULD language, cites the implementing file:line, and carries Given/When/Then scenarios that map to tests. Sections:

  • docs/specifications.md §Sessions — the agents sessions contract. Load-bearing invariants: discovery MUST parse every harness in SESSION_AGENTS (all 12) and a malformed line MUST be skipped, never thrown (SES-1, SES-3); every list row MUST show a non-empty preview — live turn → label → first-prompt topic'-' (SES-8; --flat and the interactive picker share the one unguarded renderer, SES-GAP-1); "where a session started" spans three fields (cwd + provenance + context), not one origin (SES-13); the --json shapes and SessionEvent union are a stability contract (SES-IF-1, SES-IF-4); tool-call evidence is always redacted/bounded, repeated clauses match distinct calls, tool queries never parse transcripts, and exact static program counts retain repeated sites with wrapper/effective roles; versioned tool envelopes do not replace the list/detail JSON contracts (SES-31..SES-37, SES-IF-4a); agents sessions insights emits aggregate-only actions and keeps agents insights as its top-level alias (SES-IF-4c); agents sessions stats emits its own versioned sessions-stats rollup of skill/command usage and never the list/detail shape (SES-IF-4b); agents sessions export --encrypt seals every transcript body client-side with AES-256-GCM under the shared r2.backups bundle key, or an ephemeral one when unconfigured (SES-24, SES-25).
  • docs/specifications.md §Secrets — the agents secrets contract. Load-bearing invariants: inject into the child, never materialize to the agent — every command is on one side of the boundary by construction (SEC-6, SEC-7); the master passphrase MUST be stripped from the child env (SEC-8); the "no-noise" rules — silent value-free list, batched single-prompt reads, silent broker miss, no console.* in the lib layer, no shell-rc pollution (SEC-11..SEC-17); all three desktop platforms are supported and the parity matrix names where guarantees are weaker (SEC-CROSS-1, SEC-CROSS-3).

Requirement ids are section-namespaced — SES-* / SEC-* / EXEC-*, with the -IF- (interface), -CROSS- (platform parity), -COMPAT- (stability) and -GAP- (known gap) families — and a requirement the code does not yet fully meet carries a trailing Status: [Intended] or [Drift] line naming its -GAP-.

Beyond the two above, the document also specifies §Agent execution, §Scheduling & execution singularity, and §Watchdog. It does not cover every command group — hosts, teams, and cloud have design docs but zero RFC-2119 requirements, and surfaces like wallet and sync/apply have neither. The coverage inventory says which row a surface sits in; check it before treating a behavior as guaranteed.

Every section enumerates known gaps (implemented-vs-intended drift) — a new feature MUST NOT widen them and SHOULD close the one it touches. A gap that has been closed stays as a (resolved) entry so references never dangle.

Detailed design

docs/ is the source-grounded reference. Start with architecture.md for CLI ownership boundaries and the session mechanisms, then concepts.md for the resource model. Extension-owned presentation design lives in the AGI EXT architecture. The normative contract (specifications.md) sits alongside the reference docs (sessions.md, secrets.md) — read the spec for the guarantee, the reference for the how-to.