Instances Module (multi-instance management over SSH tunnels)

August 24, 2026 · View on GitHub

Lets a single Kiro Crew gateway (the hub) manage and switch between several remote Kiro Crew instances (dev hosts, EC2, home servers) over SSH or AWS SSM Session Manager tunnels, embedding each remote dashboard as an iframe pane below a switcher strip. Opt-in: off by default (instances.enabled). The transport is per-instance (connection_method) — see §13.

Section numbers in this document are an API. src/kiro_crew/cloud/connect.py cites "instances.md §9" from two docstrings (the module docstring and ssm_proxy_ssh_host). Do not renumber existing sections; append new material as new trailing sections.

Code: src/kiro_crew/instances/ (registry, tunnel manager, port allocator, token mint, diagnostics, injection validation, run-marker) plus src/kiro_crew/dashboard/handlers_instances.py (control plane) and the frontend InstanceTabBar / InstancesViewport / Settings → Instances surfaces.

Table of Contents

1. Overview

A Kiro Crew gateway normally binds the dashboard to loopback only. The Instances feature lets the hub reach other gateways running on remote hosts by opening an SSH -L forward to each remote's loopback dashboard port, minting a short-lived dashboard token on the remote, and embedding the remote dashboard in an <iframe>. You switch panes from a dropdown (InstanceTabBar, plus Cmd/Ctrl+digit in the Electron shell); the hub keeps the most-recently-used set "warm" (tunnel + iframe live) and lazily reconnects the rest. The switcher is a menu rather than a row of chips by DEFAULT because the number of configured crews is unbounded: the closed trigger costs constant width, and unread counts stay visible on it as an aggregate badge over every crew that is not on screen. A user who switches between the same two or three crews can PIN those out of the menu into always-visible chips beside it, spending header width only on the destinations they actually use — see Pinned crew chips.

Key properties

  • Opt-in. Nothing changes until instances.enabled=true, and the flag is read at gateway startup, so it also needs a restart.
  • Owner-only. The control plane is never reachable via Slack and requires an authenticated dashboard session.
  • Loopback-only. Tunnels forward 127.0.0.1:<local> to remote 127.0.0.1:<remote>.
  • Warm, not persistent. Tokens are short-lived (20h cap) and re-minted before they lapse; iframes are evicted past the warm cap.

2. Enabling the feature

kirocrew config set instances.enabled true
kirocrew restart

Settings → Instances offers the same toggle (it PATCHes instances.enabled through /api/config/kirocrew) and then shows a "restart required" hint, because the flag is only consulted in the gateway's on_startup hook.

When enabled at startup, the gateway:

  1. creates the instances registry + SshTunnelManager and auto-reconnects every instance whose was_connected hint is set, and
  2. extends the dashboard CSP frame-src with http://*.localhost:* so an embedded remote dashboard on a *.localhost host can render.

Loopback origins (http://127.0.0.1:*, http://localhost:*, plus the https and 0.0.0.0 forms) are in frame-src unconditionally because the Web Preview panel needs them; only the *.localhost wildcard is instances-gated.

With the flag off, /api/instances/* returns 403 and the panel shows an opt-in card. GET /api/instances also reports active, which is true only when the SSH manager actually exists: enabled && !active means the flag was set after startup and a restart is still pending.

3. Architecture

 +----------------------- Hub gateway (this host) ------------------------+
 |                                                                       |
 |  Dashboard SPA                                                        |
 |   |- InstanceTabBar    switcher dropdown: Local + crews with intent   |
 |   |- InstancesViewport  warm <iframe>s: http://<host>:<port>/?token=  |
 |   +- Settings > Instances   add / edit / connect / diagnose / remove  |
 |            | owner-only JSON API (SEL-audited)                        |
 |  dashboard/handlers_instances.py                                      |
 |            |                                                          |
 |  instances/ package                                                   |
 |   |- registry.py         ~/.kiro/crew/instances.json                  |
 |   |- port_allocator.py   free-loopback-port probe (base 7778)         |
 |   |- token_mint.py       ssh <host> kirocrew token -> JWT (never logged)|
 |   |- validation.py       injection-safe ssh_host / remote_bin guards  |
 |   |- run_marker.py       <home>/run/gateway-<port>.bin launcher hint  |
 |   |- ssh_tunnel_manager  supervised ssh -N -L, probe, self-heal, refresh|
 |   +- diagnostics.py      ssh -> remote-dashboard -> local-forward ladder|
 +-----------------------------------------------------------------------+
        | ssh -N [-C] -L 127.0.0.1:<local>:127.0.0.1:<remote> <ssh_host>
        v
 +--------------- Remote gateway (dev host / EC2 / home server) ---------+
 |  kirocrew gateway bound to 127.0.0.1:<remote_port> (registry default  |
 |  5476, the port a stock gateway binds)                                |
 +-----------------------------------------------------------------------+

Module responsibilities:

ModuleResponsibility
registry.pyPersistent list of configured instances (~/.kiro/crew/instances.json) + last_active_id. Light charset check on ssh_host/remote_bin (SSH) or ssm_target/aws_profile/aws_region/ssm_run_as (SSM) at add/update, per connection_method; every mutation re-reads the file and writes atomically, so a live gateway and a CLI edit cannot clobber each other.
port_allocator.pyProbes for a free loopback port at or above tunnel_base_port (7778). The probe sets SO_REUSEADDR so a TIME_WAIT remnant from a just-closed forward is not a false "in use".
token_mint.pyRuns kirocrew token --ttl --port --embed-parent-port on the remote over SSH (run-marker first, then a bin-candidate ladder) and parses the JWT out of the printed URL. Token is returned in memory only, never logged.
ssm_token_mint.pyThe SSM sibling of token_mint.py: runs the same subcommand via aws ssm send-command through the launcher's cloud.ssm chokepoint, reusing the shared remote-command builders. Token in memory only, never logged. See §13.
validation.pyThe authoritative injection-safe guard on ssh_host / remote_bin, and on ssm_target / aws_profile / aws_region / ssm_run_as, applied immediately before any command line is built. See §11.
run_marker.pyRecords the running gateway's own kirocrew launcher (and pid) keyed by port, so a remote mint execs the same venv the live gateway runs from. Also backs zero-config client port discovery. See §12.
ssh_tunnel_manager.pySupervises one tunnel child per instance — ssh -N -L or aws ssm start-session — with readiness wait, health probe, 2-tier self-heal, proactive token refresh, stored-token liveness probe, remote restart. One state machine, two transports.
diagnostics.pyDependency-ordered failure probes; reports the first broken link. diagnose_instance (SSH ladder) and diagnose_instance_ssm (SSM ladder).
handlers_instances.pyOwner-only, enabled-gated, SEL-audited HTTP control plane.

The local forward port is allocated, not mirrored. connect() takes a free loopback port from PortAllocator (from tunnel_base_port, skipping ports the registry has already handed out) and does not require it to equal inst.remote_port. The embedded iframe loads from http://<host>:<local_port> and the remote gateway accepts that on any port because check_origin trusts a loopback Origin that equals the request's own Host — exactly the shape the iframe produces — build_allowed_hosts compares hostname only, and the session cookie is keyed off the browser-facing port (_cookie_port_from_host), so distinct local ports get distinct cookies in the shared 127.0.0.1 jar. This does not weaken CSE SEC-016: a malicious local page on an arbitrary port sends its own Origin while Host stays the gateway's, so the two differ and the request is rejected; browsers forbid scripts from forging either header.

Earlier revisions mirrored the port (local_port = inst.remote_port) for the Origin reason above, which imposed a hard constraint that every simultaneously-connected instance use a distinct remote port. The shipped defaults contradicted it — a stock gateway binds the same default port on both ends, so a stock hub already held the port a stock remote reported and two stock installs could never connect (#1972). Reconnects reuse the instance's own previous port so the iframe origin and its cookie stay stable.

Platform note. The hub side of this feature assumes a POSIX host with an OpenSSH ssh client on PATH, and run-marker port discovery refuses outright on non-POSIX (§12). Treat a Windows hub as unverified.

4. The connect → warm → self-heal lifecycle

  1. Connect. POST /api/instances/{id}/connect validates the ssh inputs, allocates a free local forward port, starts ssh -N -L, waits until the local forward accepts a TCP connection, mints a dashboard token on the remote over SSH, and returns the live status plus the token. Connect is idempotent: an already-connected instance returns its current status, and the handler then probes the stored token before handing it over (see below). The browser loads http://<dashboard-hostname>:<local>/?token=... in an iframe, deliberately reusing the parent's own hostname so the pane is same-site with the parent and SameSite=Lax auth cookies are not withheld.
  2. Warm set. Up to warm_set_cap (default 5) most-recently-used instances stay warm: iframe mounted (hide-not-unmount, so switching never reloads or re-runs the token handshake) with a live tunnel and WebSocket. Exceeding the cap evicts the least-recently-used non-active iframe. Eviction unmounts the iframe only: it does NOT disconnect the tunnel or clear was_connected, so the switcher entry persists and re-warms on the next click. Entries disappear only on an explicit disconnect.
  3. Health probe. While CONNECTED, a per-tunnel loop polls the loopback forward every DEFAULT_PROBE_INTERVAL_SECS (30s, not user-configurable; <= 0 disables the probe); after probe_failure_threshold (3) consecutive failures the child is terminated so recovery fires. This is what catches a tunnel that is alive but no longer forwarding.
  4. 2-tier self-heal. On unexpected child exit: Tier 1 rebuilds the tunnel reusing the existing token; Tier 2 re-mints the token over SSH and then rebuilds. Capped at max_recovery_attempts (8) consecutive attempts with a capped-exponential backoff (recover_backoff_max_secs, 30s; the wait grows 1, 2, 4, 8, 16 then holds at the cap), which spans roughly a two-minute window: long enough to outlast a transient drop (screen lock, proxy warmup). The counter resets on a successful rebuild or a successful connect(). If it gives up, the diagnosis ladder runs automatically. The slow SSH I/O runs without the manager lock so self-heal cannot stall a concurrent connect/disconnect/shutdown.
  5. Proactive token refresh. A per-instance loop re-mints the token at DEFAULT_TOKEN_REFRESH_FRACTION (0.8) of its TTL, ahead of the 20h cap. The frontend mirrors the same 0.8 threshold from token_ttl_remaining and skips the active pane, so a reload never interrupts the pane in use.
  6. Stored-token liveness probe. A token can go stale while the tunnel stays CONNECTED (a failed self-heal re-mint, or a remote kirocrew restart that invalidates tokens). An iframe loaded with a stale token gets a server-rendered 403, so the SPA never boots to fire the reactive mc-auth-expired recovery. connect therefore probes GET /api/status?token=... over the existing forward (no SSH, DEFAULT_TOKEN_PROBE_TIMEOUT_SECS = 2s) and is deny-by-default: anything short of a 2xx forces a fresh mint, and if that mint also fails the response is a clean 502 rather than a token the gateway cannot stand behind.
  7. Diagnose / restart. ?diagnose=1 runs the probe ladder on demand; POST .../restart runs kirocrew restart on the remote over SSH (itself service-aware), after which the local probe detects the bounce and self-heals.

Startup revive. When the feature is on, the startup hook reconnects every instance with was_connected set, serially (so they do not race to bind their mirrored ports) and each wrapped, so one unreachable host neither aborts the rest nor crashes startup. It runs as a background task rather than awaited, because on_startup fires before the HTTP port is bound and serial SSH connects would delay the bind past the desktop app's gateway-wait window. A failed revive leaves was_connected true and records the failure reason, so the entry persists showing why it is down.

5. Configuration

5.1 instances.* config keys

Transport defaults and bounds live in kiro_crew.instances.constants and are referenced from InstancesConfig, so the documented values and runtime policy cannot drift.

KeyDefaultMeaning
instances.enabledfalsePrimary opt-in, read at gateway startup. Also gates the CSP frame-src *.localhost extension.
instances.warm_set_cap5Max instances kept warm at once (bounds memory/sockets; each warm instance is a full dashboard SPA). Clamped up to 1.
instances.tunnel_base_port7778First local loopback port the allocator hands out. Out-of-range values fall back to the default.
instances.ssh_compressiontrueAdd -C to the tunnel argv. See §5.2.
instances.connect_timeout_secsunset (SSH 15.0, SSM 25.0)How long (secs) to wait for the local forward port to accept connections before declaring a connect attempt failed. Hosts behind a ProxyCommand or jump host need longer (the proxy handshake runs before ssh begins the forward). An explicit value applies to both transports, including a value equal to either transport's default. Values below 1 fall back to the transport defaults; values above 120 are clamped to 120.
instances.mint_timeout_secsunset (SSH 30.0, SSM 90.0)How long (secs) to wait for the remote kirocrew token mint before failing a connect. The mint rides the same ssh transport as the tunnel, so a host behind a ProxyCommand or jump host pays the proxy handshake here too (the connect flow spawns two proxy-bound ssh children: connect_timeout_secs budgets the first, this budgets the second). An explicit value applies to both transports, including a value equal to either transport's default — size it for the slowest transport in use. Values below 10 fall back to the transport defaults; values above 120 are clamped with a warning.
instances.max_recovery_attempts8Consecutive self-heal attempts before the tunnel is left disconnected. Below 1 falls back to the default; above MAX_RECOVERY_ATTEMPTS_CEILING (100) is clamped with a warning, so a pathological setting cannot turn bounded self-heal into a near-infinite retry loop.
instances.recover_backoff_max_secs30.0Cap on the per-attempt backoff. Non-positive falls back to the default; above RECOVER_BACKOFF_MAX_CEILING_SECS (300) is clamped, bounding the worst-case wall-clock recovery window.
instances.probe_failure_threshold3Consecutive health-probe failures before a non-forwarding tunnel is torn down. Below 1 falls back to the default.
kirocrew config set instances.warm_set_cap 3
kirocrew config set instances.ssh_compression false
kirocrew config set instances.connect_timeout_secs 45
kirocrew config set instances.mint_timeout_secs 60

Constants that are not user-configurable: the probe interval (30s), the token refresh fraction (0.8), and the stored-token probe timeout (2s).

5.2 instances.ssh_compression

Adds -C (zlib transport compression) to the supervised ssh -N -L argv. It is on by default, and the reasoning is specific to what travels over this one forwarded stream: the entire remote dashboard, meaning the SPA bundle on first connect plus every subsequent API and WebSocket frame. That payload is JS/HTML/JSON, which compresses well, and the gateway does not gzip its HTTP responses, so -C is the only compression anywhere in the path and nothing is double-compressed. The dominant deployment is a dedicated remote gateway host reached over a higher-latency link, where spending remote CPU to save bandwidth is the right trade. On a fast or local link the CPU cost can outweigh the bandwidth win, which is why it stays tunable.

The flag is read once, at startup, into the SshTunnelManager, and each _SshTunnel inherits it; changing it takes effect on the next gateway restart. Only the tunnel argv is affected. The token-mint and diagnostics ssh invocations do not compress (they are single short commands, so there is nothing to gain).

5.3 Registry file

~/.kiro/crew/instances.json, one record per instance:

id, name, ssh_host, remote_port (default 5476), local_port (0 = unallocated),
ttl (default "20h"), remote_bin, was_connected, forwarder_pid (0 = none),
forwarder_start ("" = unknown), forwarder_sig ("" = unsigned)

plus a top-level last_active_id. id is a slug (^[a-z0-9][a-z0-9-]{0,62}$) derived from name when not given, with a numeric suffix on collision. The file holds connection coordinates only: no credentials or tokens are ever written there. Every anchored validator in this package ends with \Z, never $: Python's $ also matches just before a trailing newline, so a "20h\n" would pass a $-anchored check and then reach an ssh/ssm argument list carrying an embedded newline. ttl is validated against the SAME bound the token minters enforce (^[1-9][0-9]{0,3}[hm]$): a value this layer accepted but they rejected would persist and then fail at the next connect, blaming the tunnel for a bad edit.

Two persisted hints drive lazy reconnect:

  • was_connected is sticky "connection intent". It is set when a tunnel opens and cleared only on an explicit user disconnect, deliberately surviving gateway shutdown and a failed auto-revive, so the frontend keeps the entry in an error / click-to-reconnect state instead of dropping it. It is also what the frontend keys entry visibility on (was_connected || connected || warm).
  • last_active_id records the instance most recently connected to. connect() writes it and remove() clears it, and any value that no longer resolves to a live record is dropped on the next write. Nothing in the gateway reads it: startup revive keys on was_connected and revives every intended instance, not just one, and the active pane is frontend state. get_last_active() is the only reader and has no production caller.

A third hint pair, forwarder_pid + forwarder_start, exists for crash recovery rather than reconnect intent: connect() records the spawned forwarder child's pid and its opaque platform_compat.process_start_time identity next to local_port (one write), and a successful self-heal rebuild moves both to the replacement child. A gateway hard-kill (SIGKILL/crash) never runs teardown, so the forwarder survives, reparented to init, still holding its port and its session to the remote. The next connect() reclaims exactly that child, best-effort (a failed reclaim never fails the connect). The registry is agent-writable state, so a recorded claim is honored only when it authenticates: the record must carry forwarder_sig, the gateway's own HMAC over (instance id, pid, start time, port) under a key derived from the SEL trust root (sel_hmac_key_path(), domain-separated exactly like the session_pid_sig sidecar protocol) — a root an agent can neither read nor replace, so a record written, edited, or re-pointed by anything but the gateway fails verification outright and nothing is ever signalled for it. Behind the MAC, defense in depth from kernel-owned facts: the candidate must be a genuine ORPHAN — not a pid this manager currently supervises, and reparented to init (get_ppid == 1), which no live gateway's forwarder is (subreaper hosts read as non-orphaned and merely miss the reclaim). Then, iff the recorded local_port probes occupied AND both identity halves are recorded AND the pid's live start time equals the recorded one AND its full argv exactly equals the forward command line the manager would construct for the recorded port (platform_compat.process_argv_matches_exact), it is signalled — SIGTERM escalating to SIGKILL on a bounded grace, with the start-time identity re-verified before the SIGKILL (the grace window is exactly where a pid can exit and be recycled); pid-scoped for ssh, whose child shares the dead gateway's process group; group-scoped for SSM, whose child owns its group, with completion judged by the whole group being gone and the port actually releasing. Anything short of full identity — either hint missing, start time differing or unreadable, argv unreadable (always the case on Windows, so the guard fails closed there), or any argv element differing — means the identity cannot be confirmed: the process is left alone (logged, and SEL-audited when anything was signalled) and allocation simply skips its port; the freed port returns to the pool at the next allocation rather than being re-taken by the same connect. Reclamation is keyed on the manager's OWN recorded identity, never a process-table scan — matching the table by argv pattern is what once SIGTERMed forwards operators had opened themselves (#1972).

disconnect() resets local_port to the unallocated sentinel together with was_connected and the forwarder identity pair in one write, so a freed port is never left reserved and a stale identity never reaches a later reclaim's checks.

6. API (owner-only control plane)

All routes are gated by _guard(): deny-by-default. It rejects a Slack-origin request (an X-Session-Key starting slack:) with 403, rejects a request with no request["user"] with 401, and rejects a disabled feature with 403. Every call, success and denial alike, emits a SEL audit event (instances_<operation>).

Method and pathPurpose
GET /api/instancesList instances + live status + warm_set_cap + active.
POST /api/instancesAdd an instance.
PATCH /api/instances/{id}Edit name/ssh_host/remote_port/ttl/remote_bin/connection_method/ssm_target/ssm_run_as/aws_profile/aws_region (id and internal hints are not editable). Editing a field the tunnel is BUILT from (everything except name and ttl) disconnects a live tunnel first, because it would otherwise keep forwarding the old port to the old host under the new label; the teardown passes keep_intent=True so it does not touch was_connected — that flag records a USER disconnect, so a reconfiguration leaves it alone and a real disconnect arriving mid-edit still wins. The crew therefore keeps its switcher entry and reconnects in one click. The teardown and the coordinate rewrite happen as ONE operation, SshTunnelManager.reconfigure(), which holds the manager lock across both. Done as two steps a connect can read the OLD record in between, and whether its tunnel is already CONNECTED or still CONNECTING when the write lands decides whether any after-the-fact sweep would notice it — so the window is removed rather than narrowed: a racing connect either completes before (and is torn down inside the section) or starts after (and reads the new coordinates). It also cancels and AWAITS that instance's in-flight self-heal first: recovery reads the record before it takes the lock, so a recovery already running carries the pre-edit coordinates and would reinstall a tunnel to the old machine. Because that cancellation itself awaits, a reconfiguration additionally raises a per-instance BARRIER before its first await; while the barrier is up the scheduling seams refuse to start work — _on_tunnel_exit will not begin a self-heal, a backed-off one returns without acting, and _schedule_token_refresh will not restart a mint loop — so nothing can slip into the window. Self-heal is cancelled AND awaited before the coordinates move, because it rebuilds from the record it read. The token-refresh loop is unwound by the teardown instead — after the stop succeeds — so a REJECTED edit leaves the live tunnel holding both its credential and its refresh; in both cases the cancellation is awaited, since a mint already in flight would otherwise store a token for a tunnel that is being replaced. A teardown that raises ABORTS the edit with 503 / code: tunnel_teardown_failed and persists nothing: a stop that failed leaves the old forward live, so advancing the record would describe one machine while the still-open tunnel serves another — and that tunnel is the one the user reaches. Nothing is discarded unless the stop succeeded — the tunnel keeps its place in _tunnels along with its token and refresh task — so a failed stop can neither leave an untracked process holding the port nor a live forward without a credential. The registry write is also shielded from cancellation: a client hanging up mid-write must not unwind the async with and free the lock while the write is still in flight. An edit sends only the fields that DIFFER from an IMMUTABLE snapshot of the record taken when its form opened (not the live polled record, which a concurrent CLI edit would move under the user), so the later of two concurrent saves cannot revert the earlier one's corrections; optional fields travel as explicit empty values, so emptying one clears it instead of being read as "leave as-is". The dashboard does NOT reconnect afterwards: any automatic reconnect races an explicit Disconnect arriving mid-save, so the row offers Connect instead. A crew CORRELATED to a cloud stack has its connection_method/ssm_target/aws_profile/aws_region frozen in the edit form and omitted from the request — Stop/Start/Delete resolve the machine through those, so editing them would strand a billing instance. That freeze is dashboard-side only: this endpoint still accepts those fields for any instance, because correlation lives in the cloud launch store rather than the registry. A non-dashboard caller (CLI, script, the agent driving this owner-only API) can therefore still rewrite the coordinates. This is a recorded ACCEPT rather than an oversight: the endpoint is owner-only, loopback-bound, SEL-audited, and rejected from the Slack path, so the caller is already the machine's owner or their own agent — and the damage is reversible by re-editing, though the EC2 instance bills until it is. Server-side enforcement is tracked separately; it needs correlation data that lives in the cloud launch store, not the registry. An SSM crew that cannot be correlated is offered no lifecycle action, so its fields stay editable — that identity is how the dashboard finds the machine to stop or delete, and editing it away would strand a billing instance.
DELETE /api/instances/{id}Disconnect then remove.
POST /api/instances/{id}/connectOpen tunnel + mint token. Returns the token.
POST /api/instances/{id}/refresh-tokenForce a fresh mint and return the new token. See below.
POST /api/instances/{id}/disconnectTear down one tunnel.
GET /api/instances/{id}/status[?diagnose=1]Live status; ?diagnose=1 runs the failure ladder and merges the result.
POST /api/instances/{id}/restartRestart the remote gateway over SSH.

Two routes cross the token boundary, not one. connect and refresh-token both return a minted dashboard token in their response body, and they are the only two that do. refresh-token exists because the browser needs to replace an embedded pane's credential without tearing the tunnel down: proactively at ~80% of the TTL for a non-active pane, and reactively when an embedded dashboard posts mc-auth-expired for the active pane (rate-limited client-side to one re-mint per instance per 10s so a persistently-rejecting remote cannot spin a reload storm). The invariant is the same on both: the token is delivered to the authenticated owner only, is never logged, and never appears in a list or status payload. The count is what to keep straight, since a single-route reading would leave refresh-token out of any audit of where tokens leave the gateway: the pair is connect + refresh-token, and nothing else.

Status codes worth knowing: 503 when the manager is not running (feature enabled after startup), 404 for an unknown id, 502 when a connect, refresh, or remote restart fails, and 400 on invalid add/update input.

restart is wired end to end (route, handler, restart_remote, and an api.restartInstance client method) but no dashboard surface calls it today, so it is reachable only by an authenticated owner driving the API directly.

A token is stored against a tunnel GENERATION, not just against "some tunnel for this instance". Every install bumps a per-instance counter; a mint captures the counter before it starts (mints run without the manager lock, so a slow one must not block connect/disconnect) and, under the lock, refuses to store its token if the counter moved. Without that stamp the only check available is instance_id in self._tunnels, which is true again for the REPLACEMENT tunnel — so a mint in flight across an edit-plus-reconnect, or across a self-heal reinstall, would overwrite the valid new token with one the current remote never issued, and the embedded dashboard would be handed a dead credential. The stamp covers every mint path rather than an enumerated list: the request-driven refresh_token() the embedded dashboard calls is not a task in _refresh_tasks and cannot be cancelled by name, so cancellation alone could never have reached it. A refresh also refuses to START while the reconfiguration barrier is up, since the coordinates it would read are about to move; the caller reports "no token" and the client retries after the edit.

6.1 Why a transport edit tears the tunnel down instead of answering 409

The obvious cheaper contract is to REFUSE a transport edit while a tunnel is live (409, "disconnect first") and check-and-write under the existing lock. That deletes reconfigure() and everything it carries — the per-instance barrier, the recovery index, cancel-and-await on two task families, the shielded write — from a manager that is already large. It was rejected for one reason: a transport edit is most often made because the tunnel is broken, and a broken tunnel is exactly what does not report itself as down. A crew whose host moved, whose port was taken, or whose AMI runs a different remote user sits in connected or connecting while being unusable; 409 would answer "disconnect first" to a user who is editing precisely because connecting is what stopped working, and it hands them a two-step where the failure mode of forgetting step one is a silent mismatch between the record and the live forward.

The machinery is also not paid for by this feature alone. Every piece exists because a tunnel's coordinates can change under a task that already read them — which is equally true of the pre-existing self-heal and token-refresh loops, and is where two of the bugs found during this PR's review actually lived. Making the edit path safe hardened those seams rather than adding a new hazard: the barrier is what stops a backed-off self-heal from reinstalling a tunnel to a machine the user has moved on from, with or without an edit in flight.

What the design deliberately does NOT do is reconnect afterwards. Saving ends disconnected and the row offers Connect, because any automatic reconnect races an explicit Disconnect arriving mid-save. So the cost is bounded: the save closes what its own edit invalidated, and never reopens anything on the user's behalf.

7. Security model

  • Owner-only, never via Slack. A Slack-origin X-Session-Key is rejected; an authenticated dashboard session (request["user"], set by the token-auth middleware) is positively required rather than assumed.
  • Loopback-only forwards. ssh -N -L 127.0.0.1:<local>:127.0.0.1:<remote>, with AddressFamily=inet to avoid an unexpected ::1 bind, BatchMode=yes so a missing credential fails fast instead of prompting, and ExitOnForwardFailure=yes so a forward that cannot bind is a detected failure rather than a silent hang. -N without -f is deliberate: -f would fork ssh into the background and leave the gateway unable to supervise or kill the real forwarder. The multiplexing pins in §9 close the same hole from the ssh_config side.
  • No local shell. ssh is always spawned with an argv list, so ssh_host cannot inject local shell syntax; ssh_host/remote_bin are injection-validated immediately before every command line is built (§11).
  • Tokens. Short-lived bearer tokens (MAX_SESSION_TTL_SECS caps the session at 20h) minted over SSH, returned only to the in-memory caller, never logged, and never present in list/status payloads. The mint's failure path carries a bounded stdout tail that is token-substituted and credential-redacted first, and the scan window is bounded because the redaction regexes hold the GIL.
  • postMessage relay. The parent validates every embedded-frame event.origin against an exact loopback http origin (127.0.0.1, localhost, or a single-label *.localhost) and requires the port to belong to a currently-warm tunnel before trusting any message. Only four message kinds cross the boundary: an unread count, an auth-expired signal, a switch-pane request (whose target is re-validated against the known instance list), and a readiness ping. The parent's outbound postMessage is addressed to the pane's exact origin, never *.
  • CSP. frame-ancestors is 'self' plus the exact parent origin carried in the minted token's signed embed_parent_port claim, never a wildcard and never a hardcoded port, so a local page with no validly-signed token can never frame the dashboard.
  • Untrusted ssh stderr. A proxy banner is ANSI-stripped, credential- and exfiltration-redacted, and truncated before it is surfaced in status, and it is a secondary detail only: failure classification keys on real ssh signals, so banner prose can never be read as an auth verdict.
  • Trust root. <data-home>/run/ (the run-marker dir) is on the is_sensitive_path floor, so agent file tools can neither read nor write it. See §12 and security.md.
  • SEL audit trail. Every control-plane action is audited, reads included.

8. Using it (step by step)

  1. Enable on the hub: kirocrew config set instances.enabled true && kirocrew restart (or the Settings → Instances toggle, then a restart).
  2. Open the dashboard and go to Settings → Instances. This panel is the control plane only; it does not embed remote dashboards.
  3. Add an instance:
    • Name: any label.
    • SSH host / alias: what you would type after ssh (see §9).
    • Remote port: the port the remote gateway listens on. Instances may share it — the local forward port is allocated independently.
    • Token TTL: default 20h.
    • Remote kirocrew path: only needed when kirocrew lives somewhere non-standard on the remote.
  4. Click Connect. The hub opens the tunnel and mints a token.
  5. Switch panes from the switcher dropdown in the top header (Local returns to your own dashboard). In the Electron shell, Cmd/Ctrl+digit jumps between panes in switcher order. Each row names its tunnel state in words on screen next to the status dot — colour is reinforcement, not the carrier, so the row that errored is findable without hovering every entry. Crews you switch between often can be PINNED beside the trigger as chips, so the switch costs no dropdown click — see Pinned crew chips.
  6. Diagnose a flaky instance (runs the ladder), or Disconnect from its row. Edit settings / Remove live in the row's overflow menu — a row shows two primary actions plus that menu, so everything past them is one menu deep.

An unsaved edit is held by the PANEL, keyed by crew, not by the form component. The crew list unmounts for any number of reasons the form cannot see — switching to the Set up a new one tab is enough — and an edit whose only home was the form's own state came back silently reverted to the stored record. Because a guard can only refuse the exits it enumerates, the values are lifted instead of defended: the form is re-seeded from that draft when it remounts. The draft carries its baseline — the record it was typed against — and not just the values: re-deriving the baseline from the current inst on remount would rebase a stale draft onto a newer poll, so a port someone changed from the CLI meanwhile would read as a difference and be written back to its old value by a save the user thought only touched the host. One snapshot anchors everything: dirty and the request body both measure against that baseline, so a restored draft is unsaved work rather than a clean form, and a field the user never touched is never a difference. Only three things clear it: Cancel (the user choosing to discard), a successful save, and the crew ceasing to exist. That last one is anchored to the crew's EXISTENCE rather than to the Remove button, so a removal from the CLI or a cloud Delete clears it too — ids are derived from the name, so a crew added afterwards can land on the same id, and a surviving draft would remount on a different machine and let Save overwrite settings the user never typed. It is gated on a SUCCESSFUL poll, so an errored fetch is not read as "all crews gone" and does not throw unsaved work away.

A live id is not proof of a live RECORD, though: a crew removed and recreated under the same derived id between two polls never leaves the list. So the draft is also checked against the record's machine-addressing fields (connection_method, ssh_host, remote_port, ssm_target, aws_profile, aws_region). When one of those moved externally, the form says so, names the fields, and withholds Save until the user adopts the current record. This is deliberately not a silent choice either way, because the two situations that produce the signal are indistinguishable from the client and want opposite outcomes: a concurrent CLI edit should keep the user's typing, while a replacement must never receive it — only the person looking at the row can tell which happened. A label or lifetime changed elsewhere does NOT trigger it: that cannot make this a different crew, and the baseline diff already stops the save from reverting it.

Adopting the record is a three-way merge, with the draft's original baseline as the merge base: fields the user typed are kept, every field they did not touch is taken from the record that exists now, and the baseline advances to it. Keeping the old values wholesale would convert untouched-but-stale fields into deliberate writes — the very clobber the baseline exists to prevent.

A save that tore the tunnel down also drops the crew's WARM pane. That pane is an iframe holding the old local port and token, so once the tunnel behind it is gone it cannot be revived by reconnecting — it would reuse a credential the new tunnel never issued and sit on 403. The decision reads the saved record's own status rather than guessing from which fields changed, so a name-or-ttl-only edit (which tears nothing down) keeps its working pane. Opening a DIFFERENT crew's editor while one holds unsaved changes is still refused outright — that is about two editors being open at once, not about the unmount — and the refusal renders at the row that was clicked, since the menu has already closed by then.

Prerequisite: you can already ssh <ssh_host> non-interactively from the hub (a valid key or cert in your ssh-agent, no password prompt), and the remote has kirocrew installed with a gateway running on its loopback port.

Pinned crew chips

Switching between two crews through the dropdown costs a click every time. Any entry — including Local — can be PINNED from the pin icon on its own dropdown row (lit for pinned, unlit outline for not), which lifts it out of the menu into an always-visible chip beside the trigger. Nothing is pinned by default, so a single-crew user pays no header width for the feature and sees no chip row at all.

The pin sits on the row rather than in a second list of the same crews below it, and it is a SIBLING menu item of the row's switch target rather than a button inside it: a menuitemradio may not contain another interactive element, so a nested pin would be invalid ARIA and unreachable by the menu's own arrow keys. Clicking a pin toggles it and leaves the menu open, so several crews can be pinned in one visit without navigating anywhere.

Pinning is per crew rather than one expand-everything switch because the header's budget is a PIXEL budget, not a crew count: three crews named after real hosts outgrow it while six short names fit. Choosing WHICH crews are worth header space is what keeps that budget spendable on the ones actually being switched between.

ConcernBehaviour
StoragelocalStorage key mc-crew-switcher-pinned, a JSON array of instance ids (__local__ for the local dashboard). A module-level store broadcasts changes, because several bars in one realm are mounted at once and hidden with display:none rather than unmounted — a per-component hook would leave a hidden bar on a stale value until it remounted. A remote pane's embedded bar is a separate cross-origin realm and so carries its own pin set.
MigrationThe predecessor was one expand-everything flag, mc-crew-switcher-expanded. On first read a '1' there migrates to a pinned Local rather than to an empty set: that user wanted chips, and migrating them to nothing would read as the feature having been removed. The legacy key is dropped in the same pass.
OrderThe crew on screen leads as its own chip, then the pinned chips, then the dropdown. The dropdown TRAILS the chips so it stays adjacent to the last one and reads as "and the rest"; it carries the aggregate unread for every crew not on screen, clipped ones included. The active crew is never also a pinned chip — two copies of one name would spend the budget twice.
Width boundNone of its own. The switcher sits in the topbar's left grid track (minmax(0,1fr)) inside .tb-left, which carries min-width:0 and overflow:hidden, so the track structurally prevents it from reaching the centered search column — see the three-track topbar. Earlier revisions of this feature carried a vw-derived max-width because the search overlay was absolutely positioned and a left-side cluster could squeeze it; the grid layout removed that failure mode along with the need for the cap.
OverflowThe row is a single nowrap line with overflow: hidden, and the chip at the boundary is CUT rather than dropped. Wrapping into a hidden second row would keep every chip whole, but a wrapped row still holds its full ALLOCATED width with the wrapped chips' space empty — which pushes the trailing dropdown away from the last visible chip by a gap that changes with the viewport. Filling the row keeps the two adjacent (measured at the 4px flex gap, asserted by the capture harness). A trailing fade marks the cut edge, so a cut chip reads as "there is more, in the dropdown next to me" rather than as a rendering fault. Cut chips stay reachable in the dropdown, whose row marks them no room so a pin with no visible chip does not read as a pin that failed.

Why the switcher needs no width cap. The topbar is a three-track CSS grid: minmax(0,1fr) | clamp(240px,22vw,480px) | minmax(0,1fr). The search column is a flow-internal track, not an absolutely positioned overlay, so a wide left cluster cannot reach it: .tb-left is min-width:0 with overflow:hidden, and the track simply gives the chips less room. That is the whole bound, and it holds at every viewport width and under the macOS Electron 84px header inset (which narrows the tracks rather than shifting content over them).

This is worth stating because the obvious alternative is wrong in a way that already shipped once: a hardcoded fraction. max-w-[42vw] on the chip row reached ~538px at 1280px, which under the previous absolutely-positioned search overlay pushed its available space under the minimum width and unmounted it outright. A fraction cannot track a viewport-relative sibling; a grid track does it by construction.

Counting which chips were cut off is read-only and one-directional: the result is consumed only by the dropdown's rows, which are portalled and contribute nothing to the header's width, so nothing sized by the measurement lives inside the thing being measured. The dropdown's own unread badge is absolutely positioned for the same reason — appearing must not change the button's width, since the chip row is sized from the space that button leaves. The rule itself (a chip whose trailing edge passes the row's visible width is cut) is a pure function, clippedChipIds, because jsdom performs no layout and a rendered test could never distinguish a fitted row from a clipped one. offsetLeft is only sound there because the row carries position: relative, making it the chips' offsetParent and putting both in the same coordinate space as its clientWidth.

9. Remote host types

The only thing that varies per remote is the SSH host you configure: the hub always runs a fixed ssh <ssh_host> ... argv (BatchMode=yes, ExitOnForwardFailure=yes, ServerAliveInterval=30, ServerAliveCountMax=3, AddressFamily=inet, ControlPath=none, ControlMaster=no, -L/-N, plus -C when compression is on). Anything ssh can reach non-interactively works. ssh_host accepts host, host.fqdn, an ~/.ssh/config alias, or user@host, and rejects any segment starting with - (ssh option-injection guard).

Multiplexing is pinned off; everything else is inherited. A tunnel is a supervised foreground child, and a forward the gateway cannot supervise or kill reports as ssh exited with code 0 while it is in fact still serving. A multiplexed session does exactly that — ssh hands the forward to an existing shared connection and exits. ControlPath=none is the enforcement: with no path resolved there is no socket to join. ControlMaster=no states the policy, and is not sufficient alone — an inherited ControlPath still routes into a shared connection.

Everything else per-host — User, IdentityFile, Port, ProxyJump, ProxyCommand — is still inherited from ~/.ssh/config; the registry carries no inline equivalents and depends on that. Pinning a directive the user may also set is not free: ssh takes the first value obtained and reads the command line first, so a pinned -o silently discards theirs. The two multiplexing pins are safe because a supervised tunnel must never share a connection, but the same move on, say, IgnoreUnknown would drop the pattern a cross-platform config relies on and turn a working setup into Bad configuration option.

The diagnostics probes are a different case and are left alone: they are one-shot commands whose exit status is the whole result, with no forward to own.

Dev host / home server (primary)

Use your SSH config alias or user@hostname. As long as a key in your ssh-agent (or the default identity) covers auth, BatchMode succeeds without prompting and no key path is needed.

EC2 (and other key-based hosts)

EC2 differs from a directly-reachable dev host in three ways that matter here:

AspectDirect dev hostEC2
Authkey in ssh-agent / default identitykey pair (-i key.pem), or SSM Session Manager
Login userresolved by your ssh configec2-user, ubuntu, admin, and so on: must be explicit
Reachabilitydirectoften via a bastion (ProxyJump) or SSM-only (no public SSH)

Recommended: configure an SSH alias. Because ssh_host accepts an alias, put the EC2-specific bits in ~/.ssh/config on the hub and reference the alias. The fixed ssh <alias> ... argv inherits all of it:

# ~/.ssh/config on the hub
Host my-ec2
  HostName ec2-1-2-3-4.compute-1.amazonaws.com
  User ec2-user
  IdentityFile ~/.ssh/my-key.pem
  # Optional: reach a private instance through a bastion ...
  ProxyJump bastion-host
  # ... or via SSM Session Manager (no inbound SSH needed):
  # ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"

Then add an instance with SSH host / alias = my-ec2. Prerequisites on the hub: a passphrase-less key (or an ssh-agent already holding it, since BatchMode will not prompt), and kirocrew installed with a gateway running on the instance's loopback port.

Simpler cases work without an alias: ec2-user@10.0.1.5 and ubuntu@ec2-1-2-3-4.compute-1.amazonaws.com are both accepted ssh_host values, provided the matching key is the default identity or in the agent.

The cloud launcher registers instances here. kirocrew cloud launch best-effort registers the box it created in this registry using the native SSM transport — connection_method="ssm" with the EC2 instance id as ssm_target, plus the launcher's aws_profile/aws_region (cloud/connect.py:register_instance). The dashboard then tunnels, refreshes tokens, and self-heals the box over SSM with no SSH key, no inbound port, and no hand-edited ~/.ssh/config. kirocrew cloud destroy unregisters it (matched by ssm_target) after deletion confirms. This is why cloud/connect.py cites this section, and why its numbering must not move. The legacy ssm_proxy_ssh_host helper (registering the id as ssh_host behind an ~/.ssh/config ProxyCommand) is retained for reference only and is no longer used by the managed path.

Provisioning from the dashboard (/api/cloud/*)

The Remote Crew settings page can create an EC2 crew in the user's own AWS account without dropping to the CLI. dashboard/handlers_cloud.py exposes the launcher behind the same owner-only guard as /api/instances/*: an authenticated owner (request["user"]), non-Slack, POSIX only, 403 otherwise.

Method and pathPurpose
GET /api/cloud/preflight?profile=&region=AWS reachability + the prerequisite checklist (the doctor checks as JSON).
GET /api/cloud/iam-policyThe minimum IAM policy document to paste into the user's account.
GET /api/cloud/launchList launch jobs, in progress and finished.
POST /api/cloud/launchStart a launch job; returns the job immediately. 409 when one is already in flight.
GET /api/cloud/launch/{id}Poll one job: per-step state plus the device-code prompt while signing in.
POST /api/cloud/launch/{id}/cancelRequest cancellation; honored between steps and inside the sign-in wait. A cancel during provisioning is acted on when the deploy returns, and the stack it created is rolled back.
POST /api/cloud/launch/{id}/signinAcknowledge the device-code prompt (409 when none is pending).
POST /api/cloud/{tag}/stopStop the instance behind a stack tag.
POST /api/cloud/{tag}/startStart it again.
DELETE /api/cloud/{tag}Terminate the stack (wait=False; a denied human-action check surfaces as 403).

A launch is a durable job, not a request. It outlives both the HTTP call and the browser tab: cloud/launch_job.py writes one JSON file per job under <config_dir>/run/cloud-launch-jobs/ (the run/ tree is on the sensitive-path floor, so an awaiting-sign-in job's device code is not readable by agent file tools) and rewrites it after every state transition, so progress survives navigating away and a reload. The steps are preflight → provision → signin → connect, and RealLaunchEngine (cloud/launch_engine.py) binds them to the existing iam.reachability_check, ec2.deploy, login.start_device_login and connect.register_instance — the dashboard path adds no AWS logic of its own, and registration lands in this registry exactly as the CLI's does.

A restart does not resume a launch — it terminalizes it. The worker is a daemon thread, so a gateway restart takes it with the process while the job file still reads running. LaunchJobStore.reap_orphans() runs on first store use in a new process and marks every non-terminal job it does not own as failed ("interrupted"), because the alternative is worse than an error: a progress card that can never advance, and a cancel that returns 200 while signalling a thread that no longer exists. Ownership is tracked (adopt()) so a live process never reaps its own in-flight jobs. The CloudFormation stack may well have completed in AWS, so the message points the user at their crew list rather than implying nothing was created.

Because the gateway cannot answer the device login on the user's behalf, a job parks in awaiting_signin with the verification URL and user code exposed as job state until the owner confirms it in the browser.

What is reachable through which mechanism

NeedWhere it goes
Custom login useruser@host in ssh_host, or User in an ssh-config Host block
FQDN / IP targetdirect ssh_host value
Identity fileIdentityFile in an ssh-config Host block (there is no inline field)
Non-22 SSH portPort in an ssh-config Host block (there is no inline field)
Bastion / ProxyJumpProxyJump / ProxyCommand in an ssh-config Host block
SSM-only instancesProxyCommand with aws ssm start-session

The registry deliberately carries no inline -i / -p / -J fields. The ssh-config alias path covers every case above, including bastions and SSM, which inline flags could not express, and it keeps the hub's argv fixed: a user-controlled -i path would be a new injection surface on a command line whose current variable parts are all charset-bound literals.

10. Troubleshooting

SymptomLikely cause / fix
Settings → Instances shows the opt-in cardinstances.enabled is false. Set it and restart.
Enabled but the panel says "not active"The flag was set after the gateway started; the SSH manager is created at startup only. Restart.
Iframe is blank or blackThe pane's embedded SPA never announced readiness within 15s, so the error panel with Retry appears (Retry force-reloads even an identical src). An iframe reports no load error to its parent, so this watchdog is the only signal.
Connect fails with an SSH auth errorRefresh your SSH credentials (re-add the key to ssh-agent); BatchMode never prompts, so a missing credential is an immediate failure. Tunnels self-heal once auth is restored.
Connect fails for another reasonUse Diagnose. The ladder reports the first broken link: ssh_unreachable (check SSH access or the host alias), remote_down (remote gateway not listening), not_connected (SSH and remote are fine, this instance has no tunnel yet: click Connect), or tunnel_down (reconnect).
"local port N was taken while connecting"The allocator picked a port that something grabbed in the moment before ssh bound it. Retry. If it persists, stop whatever keeps taking ports in that range or move instances.tunnel_base_port to a quieter one.
Instance keeps droppingThe health probe plus 2-tier self-heal retry over roughly a two-minute window (8 attempts, capped-exponential backoff). Tune instances.max_recovery_attempts / recover_backoff_max_secs / probe_failure_threshold; both recovery values are clamped so they cannot loop indefinitely. If self-heal gives up, diagnosis runs automatically. Check the remote gateway and SSH stability.
A pane vanished from the warm set but its switcher entry is still thereIt was LRU-evicted (warm set full). The tunnel is untouched: selecting the crew re-warms it. Raise instances.warm_set_cap if you want more panes resident.
Every token mint fails on one remote, though its gateway is healthyThe remote's ~/.local/bin/kirocrew probably points at an uninstalled checkout. See §12: the run-marker is what makes mint follow the running gateway's install.

11. Input validation (validation.py)

instances/validation.py is the authoritative injection guard for the two user-controlled strings that reach an ssh command line. It lives next to the tunnel manager rather than in the registry on purpose: the registry's _SSH_HOST_RE / _REMOTE_BIN_RE checks are an early reject for obviously malformed input at add/update time, while these functions run immediately before each command line is built, which is the only point where the value is actually dangerous. That ordering matters because the registry's load path (Instance.from_dict) is deliberately tolerant and does not validate, so a hand-edited or hand-migrated instances.json can hold anything at all until one of these functions sees it.

Two distinct attacks, two distinct rules:

  • validate_ssh_host() closes ssh option injection. Even with no local shell, an ssh_host like -oProxyCommand=... is parsed by ssh as an option and can run an arbitrary local command. It therefore rejects an empty value, anything over 255 chars, more than one @, an empty user or host segment, any segment beginning with -, and any character outside [A-Za-z0-9._-] (with the segment required to start with a letter, digit or underscore). It returns the stripped host so callers use the validated form.
  • validate_remote_bin() closes remote shell injection. remote_bin is embedded, double-quoted, into the single command string the remote shell evaluates, so it forbids every shell metacharacter, $ included (no command substitution or expansion the gateway does not control), and bounds the value to 512 chars of [A-Za-z0-9._/~ -]. An empty string is legal and means "use the candidate search".

Failures raise SshValidationError (a ValueError).

Where it is called, and what each caller does with a rejection:

CallerBehavior on rejection
SshTunnelManager.connect()Returns an ERROR status carrying "invalid ssh settings", retained in _last_error so the switcher entry can explain itself.
SshTunnelManager._recover()Aborts self-heal for that instance with a warning (no point retrying an unusable record).
SshTunnelManager._refresh_token_once()Aborts the refresh with a warning.
SshTunnelManager.restart_remote()Returns {ok: false, message: "invalid ssh settings: ..."}.
diagnostics.diagnose_instance()Short-circuits to an unknown diagnosis with a clear reason, before spawning any ssh.

The remaining variable parts of the remote command are bounded by their own validators in token_mint.py: _validate_ttl (<1-4 digits>[hm]) and _validate_port (an int in 1-65535). The bin candidates and the data-home path segments are trusted module constants.

12. The gateway run-marker (run_marker.py)

instances/run_marker.py writes and reads <data-home>/run/gateway-<port>.bin (the running gateway's own kirocrew launcher path) and <data-home>/run/gateway-<port>.pid (its pid). It has two unrelated consumers, and separating them is the point of the module.

Consumer 1: remote token mint targets the running gateway's install

Token mint SSHes to the remote and resolves kirocrew from a fixed PATH candidate list whose first entry is $HOME/.local/bin/kirocrew. When that launcher symlinks into an uninstalled checkout (no .venv), every mint fails even though the gateway itself is healthy, because the gateway runs from a different venv. Rebuilding and restarting the gateway does not fix mint, since mint never consults the gateway's own install, and the refresh loop then fails on every cycle, which surfaces to the user as a pane that periodically disconnects and reconnects.

The fix: at startup the gateway records the absolute path to its own launcher, keyed by the port it serves. The mint shell snippet reads that marker first and, when it names an executable file, execs it, so mint uses the same venv as the live gateway. The snippet probes three data homes in priority order, since the remote's non-interactive SSH shell usually does not export KIROCREW_HOME:

  1. $KIROCREW_HOME when set and non-empty,
  2. $HOME/<CONFIG_DIR_NAME> (the current default, .kiro/crew),
  3. $HOME/<LEGACY_CONFIG_DIR_NAME> (.kirocrew, for a not-yet-migrated remote).

Those two home segments are interpolated from the shared kiro_crew.config.paths constants, the same ones the marker writer derives its default from, so reader and writer cannot drift apart on a future data-home rename. An absent or stale marker, or one that does not name an executable, falls through to the candidate search, so nothing regresses on an older remote. An explicit remote_bin is never overridden by the marker: it is the user's deliberate choice.

restart_remote() resolves kirocrew restart through the same path, keyed by the instance's remote_port.

The launcher path is derived from sys.executable's sibling console script (kirocrew, or kirocrew.exe on Windows) and is deliberately not resolved through symlinks, because the console script sits next to the possibly-symlinked interpreter in the venv's bin/, not next to the real interpreter. When no such script exists (a source-tree python -m kiro_crew launch) the marker is written empty: the mint clause requires a non-empty executable path so an empty marker is inert there, but the filename still matters to consumer 2.

Consumer 2: zero-config client port discovery

The marker's filename advertises which port a gateway serves, so marker_ports() lets a local client command (token / status / logout / stop, via port_resolution.resolve_client_port) find a gateway on a non-default port with no configuration. That path reads only the filename and ignores marker contents entirely. Resolution order is --port, then KIROCREW_PORT, then a port named by dashboard.url, then the sole gateway-owned marker, then the default 5476.

A marker is not proof a gateway is there. clear_marker() runs only on graceful shutdown, so a crash or SIGKILL leaves the file behind and an unrelated process may since have bound that port. Because client commands send the local secret (X-Local-Secret) to whatever answers, the consumer must verify the listener before trusting a discovered port. port_resolution._gateway_owns_port() does that in four fail-closed steps: the recorded pid must exist, must be among platform_compat.find_listening_pids(port), must be owned by the caller's uid (which closes pid recycling into another user's process), and must look like a gateway by argv (defense in depth only, never the sole proof). Discovery is skipped outright on non-POSIX hosts, where no owner can be reported and the file-permission argument does not hold, so Windows users keep --port / KIROCREW_PORT. This module deliberately offers no bare "is something listening" helper, so no caller can mistake reachability for identity.

The live gateway prunes markers naming other ports on startup, EXCEPT any whose gateway passes the same ownership proof its readers use. gateway.lock makes a gateway a singleton per data home only when every start goes through it, and in practice one machine runs several that share a home (a second gateway launched by hand, one started from another checkout that inherits the default data home, a cutover overlapping its predecessor). A blanket prune deletes a LIVE gateway's marker and pid sidecar, which makes it undiscoverable to token / status / stop and destroys the evidence section 12.1 depends on. The ownership check fails closed by RETURNING FALSE rather than raising -- non-POSIX returns False outright, and a missing or throwing listener-lookup tool is folded into False as well -- so a False answer means "ownership not proven", NOT "process gone", and the two cases are indistinguishable from the caller. Such a marker still prunes, so markers do not accumulate forever -- but the prune removes only the marker and pid sidecar, never the credential, because treating False as death would strip a LIVE incumbent's credential on every Windows host and push its clients onto a shared file a newcomer may have replaced. clear_marker() owns credential deletion.

12.1 The internal-API credential is keyed by port

<data-home>/run/gateway-<port>.secret holds the internal-API credential of the gateway serving that port, written 0600 beside the marker and removed by clear_marker() with it.

The credential is generated per gateway start (os.urandom(16).hex()) and kept in memory as the value the auth middleware compares against, so it identifies ONE generation. Published only to the single shared <data-home>/.local_secret, it was last-writer-wins per home: a second gateway starting in the same home replaced the file while the first kept serving the port, the incumbent went on comparing against its own in-memory value, and every internal caller then sent the newcomer's credential to the incumbent. The whole internal channel answers 403 with a body of exactly Forbidden until one of them restarts: learn_add, spawn, session-keepalive, artifact writes, the task runner, all at once, with no warning and no metric.

Two rules keep the two halves paired:

  • The writer (dashboard.server._write_instance_credentials) always writes the per-port file, and writes the shared .local_secret only when no other gateway in the home is verifiably alive on a different port. The shared file is still written in the single-instance case because pre-per-port readers (an older CLI, a cron script from a previous install) know only that path.
  • The reader is ONE shared helper, config.loader.read_local_secret(port): it returns the credential for the port the caller is about to dial and falls back to .local_secret when no per-port file exists. It lives there rather than in each reader because every surface that implements its own read reintroduces the bug for itself. port is required. An optional port would resolve the dial target from process context, so a converted call site could read the credential for one gateway while dialing another -- the same desync, reintroduced one call site at a time and invisible in the hunk under review. A caller with no port resolves one explicitly and passes it, where the choice is reviewable. mcp_core, mcp_shared, cron_script, computer_use/screencast and the Sage review driver each name their dial target; a test greps for a no-argument call so the shape cannot come back.
  • The dialed port's own credential outranks any path a caller names. cron_trigger.trigger_cron_job reads the per-port credential for the port it posts to FIRST, and falls back to the secret_path its caller named only when that is absent. The order is deliberate and is dictated by the callers: both of them pass config_dir() / ".local_secret", the home-wide file, which is exactly the file a second gateway generation replaces -- so preferring the named path would reinstate the defect this module exists to prevent. The cost of that order, stated rather than hidden: a crash-orphaned run/gateway-<port>.secret (the prune never deletes credentials, see section 12) is preferred over a correct named path, so a caller that genuinely names another home's credential for a port this home once served would send the stale one and get a 403. No caller does that today -- both name the ambient home-wide file -- and closing it properly means the credential-path parameter going away rather than the order flipping.

A denial carries a machine-readable code (internal_auth_mismatch) beside the prose, because a genuine permission denial produces the same Forbidden body and a consumer matching on text misdiagnoses one as the other. It also names both sides by fingerprint (a short SHA-256 prefix plus length, never the value), so a cross-generation mismatch is distinguishable from a forged header and from a caller that had no credential at all; without it a real desync is unattributable from the log.

Why run/ is on the sensitive-path floor

The marker names a path that the gateway execs on the remote host outside the agent sandbox, and run/ also holds the sandbox launcher scripts. An agent that could write into this dir could point a marker at an attacker-controlled binary and get it executed unsandboxed on the next routine token refresh: a reachable sandbox escape, which the owner and -x checks do not stop because agent writes run as the same user. run/ is therefore classified read+write sensitive in security._SENSITIVE_HOME_DIRS, under every known data-home prefix. The dir is created 0700 (re-applied on an existing dir, since exist_ok does not re-apply mode) and both files are written 0600 through the shared atomic_write helper, whose unique mkstemp + os.replace closes the same-user symlink TOCTOU a predictable <name>.tmp would leave open. Every legitimate writer opens these paths directly and does not route through the file gate, so gateway startup and spawn are unaffected.

13. The SSM connection method (connection_method)

Each instance record carries a connection_method: "ssh" (default) or "ssm". SSM tunnels over AWS Systems Manager Session Manager, so it needs no inbound port, no sshd and no distributed key — reachability is an IAM decision (ssm:StartSession on the instance ARN) rather than a network one.

MethodTunnel commandClient prerequisitesMint path
ssh (default)ssh -N -L 127.0.0.1:LP:127.0.0.1:RP <ssh_host>non-interactive SSH accessssh <host> kirocrew token
ssmaws ssm start-session --document-name AWS-StartPortForwardingSession --target <ssm_target> --parameters portNumber=RP,localPortNumber=LPAWS CLI + session-manager-plugin; ssm:StartSession, ssm:SendCommand, ssm:GetCommandInvocationaws ssm send-command → kirocrew token

Records are back-compatible: an instances.json written before this feature has no connection_method and loads as "ssh".

SSM-only registry fields: ssm_target (an EC2 i-… or SSM managed-instance mi-… id), plus optional aws_profile, aws_region and ssm_run_as. Only the profile name is persisted — never a credential; the AWS CLI resolves credentials via its own provider chain.

ssm_run_as is the remote POSIX user SSM commands run as: cloud.ssm.run_command wraps every remote command in sudo -u <user> -i bash, and its default (ec2-user) is a launcher assumption that holds only for provisioned AL2023 boxes. Without a per-instance override, an Ubuntu AMI would bring the tunnel up and then fail the mint — and, because the readiness probe also runs through that same wrapper, diagnose_instance_ssm would report remote_down for a perfectly healthy remote gateway. The field defaults to ec2-user (so existing records and launcher-provisioned boxes are unaffected), is charset-validated as a Unix username like the other SSM coordinates, and an empty value resolves to the default rather than emitting a bare sudo -u ''.

One state machine, two transports

_SshTunnel builds either argv from the same class, and _TransportParams (resolved once per operation by _resolve_transport) carries the validated per-transport values so connect / _rebuild / _recover / _refresh_token_once / restart_remote share one code path. The health probe, 2-tier self-heal, proactive refresh, stored-token liveness probe and startup auto-revive are transport-agnostic.

Two SSM-specific behaviours:

  • Process-tree teardown (all platforms). The SSM child gets process-group isolation at spawn — start_new_session on POSIX, CREATE_NEW_PROCESS_GROUP on Windows, passed explicitly per the platform_compat recipe — and teardown reaps the whole tree through platform_compat.kill_process_tree (killpg POSIX / taskkill /T Windows). This matters because the session-manager-plugin grandchild is what actually holds the forwarded port: terminate() on the aws wrapper alone orphans it and wedges the port. Doing this with raw os.killpg/os.getpgid would silently degrade to wrapper-only termination on Windows, which Kiro Crew supports.
  • Readiness timeout. session-manager-plugin completes a WebSocket handshake with the SSM service before binding, so the SSM transport uses a longer default connect timeout than a direct ssh TCP connect. An explicit caller-supplied timeout still wins for both.

_ssm_exit_error classifies the child's exit with SSM vocabulary (expired credentials, ssm:StartSession denial, missing plugin, target not a connected managed node, local bind conflict) rather than running SSM stderr through the ssh auth/transport matchers, which would mislabel an AccessDenied as an ssh auth failure.

Diagnosis ladder

diagnose_instance_ssm mirrors the SSH ladder with an SSM first rung, so an offline agent is not reported as a dead remote gateway:

  1. managed node online? (describe-instance-information) → no ⇒ ssm_unreachable
  2. remote dashboard up? (send-command + curl on the remote loopback) → no ⇒ remote_down
  3. local forward reachable? → no ⇒ tunnel_down, else ok

ssm_unreachable is a new diagnosis code; the shared rungs reuse SSM-worded reasons so the copy never tells an SSM user to "check SSH access".

Reuse of the launcher's SSM primitives

Argv building and remote execution delegate to cloud.ssm (build_port_forward_argv, run_command) rather than duplicating them, so the two features cannot drift on the SSM document or parameter shape. Those calls run in the gateway process, which has no KIROCREW_SESSION_KEY, so the launcher's agent-session chokepoint does not apply; the hooks.py denied-command list gates agent tool calls and likewise does not gate the gateway's own children.

Trade-off: the mint transits SSM command history

The mint runs over send-command, so the token appears in that invocation's output, which SSM retains for up to 30 days and is readable with ssm:GetCommandInvocation. It is not in CloudTrail (which records the API call, not the output), and no S3/CloudWatch output destination is configured.

Bounded by: the token is TTL-capped and only usable against the remote's loopback, so using it requires ssm:StartSession — a superset of the access needed to read the history. The generated launcher policy also withholds ssm:ListCommandInvocations, so a holder cannot enumerate command ids hunting for tokens. The SSH transport has no equivalent exposure. This mirrors the accepted posture in cloud/connect.py::mint_token.

ssm_token_mint.py is listed in security_posture.NON_EGRESS_REDACTION_MODULES alongside its SSH sibling: it redacts remote output on the way into an exception, which is not an egress boundary.

Interaction with §9

§9 documents reaching an SSM-only instance through an ~/.ssh/config ProxyCommand — still valid as a manual option, and still connection_method="ssh": the reachability lives in ssh config and Kiro Crew is unaware of it. connection_method="ssm" is the direct alternative, requiring neither sshd nor a key on the remote — and it is now what cloud/connect.py's registry integration uses (register_instance sets connection_method="ssm", ssm_target=<instance-id>). The legacy ssm_proxy_ssh_host helper is kept for reference only.

14. Session transfer (send a session to another instance)

Copies one dashboard session from this instance to a connected peer. The user picks it from any session menu: Send a copy to ▸ <instance>.

Code: src/kiro_crew/dashboard/session_transfer.py (bundle + importer), SshTunnelManager.send_session_bundle (delivery), handlers_instances.api_instances_send_session (control plane), and the frontend SendToInstanceSubmenu mounted inside the shared SessionActionsMenu.

14.1 Why it needs no new transport

A session is a portable JSONL transcript (<data-home>/sessions/<key>.jsonl: a metadata line then {role, content, ts} records) and the receiving side already knows how to turn one into a live tab — that is what chat_persistence does on every gateway restart. So a transfer reuses two things that exist: the tunnel from §4 and the rehydrate path.

The gateway binds loopback unconditionally (dashboard/urls.py:is_local_only always returns True in the public build), so an instance tunnel is the only sanctioned way to reach a peer. Nothing here opens a socket.

14.1a Two layers — and why Layer B is what makes resume real

The transcript above is only the display copy (Layer A). The context the model actually holds — the compaction/turn state, keyed by a kiro-cli session id — lives in a second store outside the crew home: kiro_sessions_dir()/<sid>.json + <sid>.jsonl, joined to a slot through session_map.json. Call it Layer B.

This split is the whole fidelity story. Ship Layer A alone and the peer has a browsable history but no resumable context: SessionMap.get finds no usable sid and the next turn falls back to _build_history_prefix(), a condensed ~8K-char text prefix — no tool state, no real context window. Ship Layer B too and the peer resumes through session/load under its own fresh sid, which is the same fidelity a local gateway restart gives.

So bundle_version 2 carries an optional layer_b. It is optional by design: a v1 sender, or a session that never opened a kiro-cli context, ships Layer A only and the peer degrades to the prefix. Both versions stay accepted so a newer instance can still receive from an older one.

On import Layer B's host-naming fields are rewritten — a fresh sid (so copy-never-move holds and a repeat send cannot collide), cwd and the filesystem allowed_*_paths cleared (matching the project decision below — the session arrives unscoped), agent_name set to the target-resolved agent — while the conversation payload travels byte-exact. That distinction is forced, not stylistic: thinking blocks inside conversation_metadata carry a provider signature over their own content, which is validated when the conversation is replayed, so rewriting any covered byte makes the peer's next turn fail — long after the import reported success. An earlier revision scrubbed Layer B on both boundaries and, measured against one developer machine's 704 real sessions, altered a signature in 41% of them. Redacting this artifact and transplanting it cannot both hold; what bounds the exposure is the destination (the operator's own peer, over a tunnel they authenticated, stored 0600), not a scrub of the payload. Layer A keeps its redaction — that text is rendered and re-read as context. Inbound Layer B is validated structurally (parse-only, never rewritten) and refused whole if any record fails to parse. Materialisation is best-effort: if it fails, the import still succeeds as the transcript-only copy rather than failing an already-persisted session.

Sub-agent conversations deliberately do not travel. Their results were already injected into the parent conversation, so they are inside Layer B already; only spawn_continue against one specific sub-agent is lost on the peer.

14.2 Copy, never move

Import always allocates a new slot key and never mutates or deletes an existing session, on either side. Consequences worth stating:

  • the source tab is untouched, so a failed transfer costs nothing;
  • a repeat click sends a second copy rather than erroring, so the action needs no confirm step and no idempotency key;
  • there is no "move" verb and nothing in this feature can destroy a conversation.

14.3 What travels, and what deliberately does not

FieldTravels?Why
transcript (user / assistant turns)yesLayer A — the portable display copy. Tool and system frames are dropped from it: they reference local tool state.
layer_b (kiro-cli context: envelope + events)yes (v2)Layer B — the real context window, so the session RESUMES rather than replaying a lossy ~8K prefix. Only host-naming fields are rewritten on arrival (fresh sid, cleared cwd/allowed_*_paths, target agent); the conversation payload travels byte-exact and unredacted, because its thinking-block signatures are validated on replay. Optional, and best-effort on import.
sub-agent conversationsnoTheir results are already inside Layer B as injected context. Only spawn_continue on one specific sub-agent is lost.
memory (preferences, semantic KV, lessons)noA workspace's memory is a per-instance scope, and copying it across hosts is the risky, hard-to-undo part of a transfer. The peer keeps its own.
titleyesPrefixed ⇄ and suffixed (from <origin>) on arrival, so a transferred tab is never mistaken for a locally-born one. The prefix is stripped before re-bundling so a session bounced back and forth does not accumulate one prefix per hop.
agenthint onlyApplied only if the target has an agent by that name, else dropped. An agent template is a local object; carrying the name blindly would leave the slot pointing at nothing.
projectnoThe headline decision. The source's checkout path almost never exists on the target (a Mac worktree path on a Linux dev desk), and a slot pointing at a missing directory scopes file search and steering to nothing. The session arrives unscoped and the user re-picks a project.
modelnoAccounts differ in entitlement, so an id the source is served can fail at runtime on the target. The target resolves its own default (AGENTS.md § Model selection).
workspacenoWorkspaces are per-instance memory scopes; a matching name still means a different memory.
folder_id, tags, pinned, artifact, app, linked_session_key, forked_fromnoLocal-graph references that would dangle.

bundle_version is refused when outside the supported set ({1, 2}) rather than best-effort parsed: the two ends are independently-updated installs, and a silently misread field would land as corrupted conversation. Accepting both versions is what lets a v2 instance still receive a copy from a v1 one.

14.4 API

Method and pathPurpose
POST /api/instances/{id}/send-sessionSending side. Body {"slot": "<local slot key>"}. Bundles the local session and delivers it over that instance's open tunnel.
POST /api/chat/slots/importReceiving side. Accepts a bundle and materialises a new slot.

send-session goes through the same _guard() as every other route in §6 (owner-only, never Slack, feature-gated, SEL-audited as instances_send_session). It returns {ok, instance, remote_key, messages}.

Status codes: 404 unknown instance or unknown local slot, 400 a non-persistent source session or a malformed body, 503 the manager is not running or the source could not be persisted first, 502 the peer refused or was unreachable (the peer's own code is forwarded).

send-session is NOT a third token-crossing route. §6's invariant holds: connect and refresh-token remain the only two routes whose response carries a minted token. The transfer needs the credential but the browser does not, so the request is issued inside SshTunnelManager.send_session_bundle — the token never leaves the manager, is sent as a cookie (so it cannot land in the peer's access log), and is never logged. Any audit of where tokens leave the gateway still finds exactly two routes.

14.5 Trust model for an inbound session

An imported transcript is untrusted input that later becomes context an agent re-reads, so:

  • reaching /api/chat/slots/import requires a valid dashboard credential, which in practice means a token this hub minted on that host — a peer cannot push a session into an instance it has no credential for;
  • bundles are size-bounded before anything is written (5,000 messages, 1 MB per message, 20 MB of content total) and every message's role is checked against user/assistant;
  • assistant content is credential- and exfiltration-redacted on the way in, matching the fork path. User turns are left verbatim: redacting what the human typed would corrupt their own words;
  • import does not drive a turn. The session lands as a tab and waits for the user to type. This is the feature's main security advantage over an agent-facing "send a message to a peer" tool: no inbound text can make a remote agent act.

14.6 Direction and topology

The submenu on a given dashboard lists that gateway's registry, so a push runs hub → peer. Because each remote dashboard is embedded as an iframe (§3), a "send" driven from inside a remote pane would need that remote to reach back to the hub — usually impossible (a dev desk cannot SSH to a laptop). Sending in the other direction is therefore done by registering the peers you want on each host that should originate a transfer, and a hub-initiated pull (read a peer's session over the same forward) is the natural follow-on that would make remote → hub and remote → remote work without any reverse reachability.

15. Federated session search (search every connected instance at once)

GET /api/instances/search-sessions answers one query with sessions from the local gateway and every instance whose tunnel is currently CONNECTED. The dashboard's two search surfaces switch to it automatically whenever at least one warm connection exists (the ⌘K palette's Sessions tab and the sidebar's Older Sessions search); with no warm instance they keep calling the plain local /api/sessions/search, so a peerless install never pays the detour.

15.1 It is the hub-initiated pull §14.6 anticipated

The search reuses the transfer's transport shape exactly: the hub GETs a peer's own /api/sessions/search over the already-open forward — no SSH spawn, no new port, no reverse reachability. SshTunnelManager.search_sessions_remote follows send_session_bundle's credential rules to the letter: the token never leaves the manager (§6's invariant holds — connect and refresh-token remain the only routes whose response carries one), it travels as the port-scoped cookie so it cannot land in the peer's access log, and a 401/403 gets exactly one transparent re-mint retry, because a retained credential can go stale while the tunnel stays CONNECTED.

Each peer request runs under DEFAULT_SEARCH_PROXY_TIMEOUT_SECS (6s) — sized between the token probe (2s, a bare ping, which would produce false "unreachable" verdicts on a loaded peer doing real scan work) and the transfer budget (30s, which would let one dead tunnel stall a keystroke-driven search). Peers are fanned out concurrently, so the slowest peer bounds the whole reply.

15.2 Merging without a cross-instance score

The aggregator rank-interleaves: position k of the reply cycles through each source's k-th best hit, local source first. Raw scores are never compared across gateways — each instance may run a different ranking version (a newer hub searching an older peer, or vice versa), so a numeric merge would silently prefer whichever version inflates its scores. Interleaving needs no score wire format, keeps every source represented in the top rows, and preserves each source's own internal order.

An unreachable or refusing peer never fails the request: it is reported in the reply's unreachable array as {id, name, code} so a caller can tell what was NOT searched instead of having the result set silently narrowed. The shipped dashboard surfaces log the report (a visible "N instances unreachable" affordance is a follow-up); only CONNECTED peers are fanned out, so a miss here is a rare mid-search transient rather than the steady state for a down instance. Machine-readable codes distinguish a stale credential (search_unauthorized) from a dead tunnel (search_unreachable), a peer error (search_peer_refused), and a garbled reply (search_malformed_reply); the same codes are recorded in the SEL audit event for the request, so an operator can audit which peer failed and why without reproducing the search.

15.3 Peer replies are untrusted input

A peer's rows are re-shaped through a strict allowlist before they reach the browser: only known fields are copied, strings are type-checked, and title / snippet are re-run through the local credential + exfiltration redaction — the peer claims to have redacted, but this hub does not take its word for it. Rows from a peer additionally carry instance_id + instance_name; local rows carry neither, so the reply shape for a hub with no peers degrades to exactly the local search's own.

The endpoint runs behind the same _guard() as every §6 route (owner-only, never Slack, instances.enabled, SEL-audited as instances_search_sessions) and mirrors the local search's input contract (q sanitized, capped at 256 chars, min SEARCH_MIN_CHARS; limit default 50, max 200). The local rows are also redacted here: the aggregator calls conversation_log.search_sessions directly rather than going through the /api/sessions/search handler where the local redaction normally lives.

15.4 What the UI does with a remote row

A remote row's transcript lives on the other gateway, so the local dashboard can neither resume nor delete it:

  • Activation switches panes. Both surfaces route through useSelectInstance (the single owner of switch-to-a-pane semantics, §3), so clicking a remote row activates that instance's embedded pane — reconnecting it first if needed. Deep-linking to the specific session inside the embedded SPA is a follow-up: the iframe protocol has no open-session message yet.
  • The local delete action is hidden on remote rows. deleteHistorySession targets the LOCAL session file; with colliding keys across gateways it would delete a same-keyed, unrelated local conversation.
  • ⌘Enter (open in local split grid) is inert for remote rows in the palette — bound to an explicit no-op, because an absent handler makes the palette's Enter dispatch fall back to plain activation and the chord would silently switch panes.
  • Remote rows are badged with the instance's raw name (never translated — it is the user's own label, which also keeps the change i18n-neutral), and result ids are namespaced by instance so two gateways' same-keyed sessions cannot collide in the palette's keyed list. Snippet-highlight offsets are shifted by the prefix length so remote rows highlight the same match a local row would.
  • Any federated-endpoint failure in the UI — including the 403 when the instances feature is off — falls back to the plain local search, which is always the floor.