Iris Operations

All subcommands have --help. Use it.

Connection selectors:

• --cluster=NAME (preferred for known clusters): resolves a named config.
• --config=PATH: pins an exact YAML config file.
• --controller-url=URL: connects to an explicit URL.

How the controller is reached depends on the cluster. IAP-fronted clusters (marin, marin-dev) are reached directly over their IAP HTTPS URL — there is no SSH tunnel; require_controller_url returns the IAP URL and every request is authenticated at the edge (see Authentication and docs/iap-gclb.md). Non-IAP clusters open an SSH tunnel to the controller VM automatically.

Use iris cluster list to see named clusters. Use --config when you mean a custom or pinned file path.

Authentication (headless / CI)

For an IAP-fronted cluster the CLI authenticates every request at the IAP edge; the controller mints no token. Two ways to get an edge token:

• Interactive — iris --cluster=NAME login runs the desktop OAuth browser flow and caches a refresh token. Needs a browser; not usable headless.

• Headless / CI / agent — do not run iris login (it needs a browser). Instead give the process credentials for an IAP-allowlisted service account. The keyless way is to point your ADC at one by impersonation — no browser, no key file; iris reads it through the standard resolver with no flag or env:

  gcloud auth application-default login \
    --impersonate-service-account=iris-controller@hai-gcp-models.iam.gserviceaccount.com
  iris --cluster=marin-dev cluster status
  

  Needs roles/iam.serviceAccountTokenCreator on the SA (to impersonate) and roles/iap.httpsResourceAccessor on the cluster backend for that SA (the IAP allowlist — the impersonated SA email is the identity IAP authorizes). On a GCE VM whose service account is already allowlisted none of this is needed: ambient metadata credentials are used automatically.

See docs/iap-gclb.md ("The three caller paths") for the audience-vs-identity model behind this.

Cluster Lifecycle

iris cluster start|stop|restart|status
iris cluster dashboard              # open tunnel, print URL, block
iris cluster dashboard-proxy        # local proxy to remote controller (no tunnel needed)

Controller Restart

For a rollout across more than one cluster, use the use-iris skill. It fixes the order (marin-dev, marin, then CoreWeave smallest first), puts a human gate on every step, and drives scripts/iris/rollout_controllers.py for the credential preflight, the before/after snapshots, the 5-minute watch, and a one-job smoke test.

iris cluster controller restart restarts the controller only (seconds of downtime, workers unaffected). iris cluster restart tears down everything — controller + all workers. All jobs die. Never run the full iris cluster restart without explicit user approval.

Run controller lifecycle commands with the controller extra, which supplies the Kubernetes client used by CoreWeave prerequisite checks:

uv run --package marin-iris --extra controller iris \
  --cluster=<name> cluster controller restart

The base uv run iris environment omits this dependency and can report every CoreWeave prerequisite as missing even when the objects exist.

Workflow: confirm the tree holds exactly the code to ship (git status, git log -1) -> capture baseline (iris cluster status) -> restart -> verify.

The restart preflight resolves operator-side controller secrets before taking a checkpoint, building images, or writing a rollout record. CoreWeave keeps the resolved signing key in memory and uses that value when it projects iris-controller-env. A missing Secret Manager dependency or inaccessible secret leaves the running controller and rollout record unchanged.

iris cluster controller serve --dry-run is not a restart-validation step: it boots a full local controller that serves until killed (task dispatch, VM changes, and checkpoint writes suppressed) for interactive state inspection — e.g. replaying a checkpoint to debug scheduling. Rely on the unit suite / CI on the tree as the pre-restart gate.

If checkpoint times out: iris cluster controller restart --skip-checkpoint (restores from last periodic checkpoint; some recent state may be lost).

Restart builds and deploys your local working tree. iris cluster controller restart builds the images required by the configured runtime from your current checkout — HEAD plus any staged/unstaged changes (get_git_sha() is a tree-content hash), pushes them, pins the deploy to :<hash> in memory, and restarts the container in place. So the restart ships whatever code is in your tree; there is no separate image-rebuild step. To deploy a merged controller fix: update your checkout (git pull, or check out the fix) then restart — restarting from a stale checkout ships that stale code. Always confirm the controller is running the :<git-short-hash> you expect (iris cluster status), not just that it came back up; a stale-checkout deploy once cost ~5 red-canary days (incident record).

Restarts default to the fast Rust profile, which skips LTO and reduces native link time. Kubernetes clusters build the controller and task images for amd64+arm64 because task Pods run the controller image as the log-shipper sidecar; they skip the unused worker image. VM clusters build amd64 controller, worker, and task images. --image-platform overrides only the task image, for example when a Kubernetes dev cluster has amd64 nodes only:

uv run --package marin-iris --extra controller iris \
  --config path/to/dev.yaml cluster controller restart \
  --image-platform linux/amd64

Pass --cargo-profile release for an LTO build. Keep the default task image platforms when the deployed cluster includes arm64 nodes.

Rollout state is recorded automatically. Each controller restart writes a rollout record to gs://…/<cluster>/state/rollout-record.json — the image it deployed, the image it replaced, the pre-deploy checkpoint it took, and a phase (pending → committed for a forward deploy; rollback_requested → rolled_back for a revert). The rollback coordinates are captured as part of the deploy, so you never track them by hand. A forward restart also health-checks the new controller and auto-rolls back to the previous image + its pre-deploy checkpoint if the deploy fails to come up. (The first deploy after this landed has no prior record, so there is nothing to auto-roll back to — recover a failed first deploy by checking out known-good code and restarting forward, or use the on-VM procedure below.)

A failed SSH leg aborts the restart safely. On a GCE cluster the restart drives the VM over gcloud compute ssh --tunnel-through-iap; if that SSH fails (it retries 3×), the CLI prints Rollback restart failed: Command failed after 3 attempts: SSH exit code 255 and exits — but the running controller was never touched. Confirm with iris cluster status: the old version still healthy means nothing deployed and nothing needs rolling back; fix SSH and retry the restart.

GCE controller SSH auth is per-username, and agent/headless sessions may lack it. gcloud compute ssh connects as your local OS username; Permission denied (publickey) right after the IAP tunnel opens means the VM refused that username+key pair, not that the tunnel failed. The controller VM does not necessarily honor every key visible in project/instance ssh-keys metadata for your username, so a key that "should" work from metadata inspection can still be refused — and adding new keys (metadata or OS Login) from an unattended session is exactly the kind of credential change an operator should approve first. If the restart's SSH leg is refused from your session, run the restart from a session that already has working SSH to the VM rather than minting access. Note this only gates GCE clusters (marin, marin-dev); CoreWeave controller restarts go through the Kubernetes API (kubeconfig at ~/.kube/coreweave-iris, context pinned per cluster config) and need no SSH.

Rolling back a controller deploy (migration-aware)

Roll back the last deploy. iris cluster controller restart --rollback reads rollout-record.json, then redeploys the previous image and restores its pre-deploy checkpoint — no coordinates to look up. Run it while the controller is still reachable so it takes the in-place path.

uv run --package marin-iris --extra controller iris \
  --cluster=marin cluster controller restart --rollback

Why it restores a checkpoint, not just the old image. A restart runs forward-only migrations in place on the on-VM state DB (schema_migrations tracks applied stems; there is no down-migration), and some are destructive — e.g. 0039_drop_api_keys, 0040_drop_users. Redeploying the old image alone would leave it loading a schema it does not understand, hitting missing-table errors at runtime. So a correct rollback must also restore the pre-deploy (pre-migration) checkpoint — the one taken while the old code was still running. --rollback does both from the record: it writes rollback_requested and restarts the previous image; on boot the controller restores that checkpoint over its migrated local DB, then marks the record rolled_back. That consume-once step is a one-shot — a later crash or VM reboot reuses the restored DB instead of rewinding to the checkpoint again.

For a wedged/unreachable controller, or a deploy with no prior rollout record, use the fully-manual on-VM procedure below instead, which never risks recreating the VM.

Controller Checkpoint Rollback (wedged / OOM recovery)

When. The controller is wedged by a bloated local DB — typically a controller-VM OOM after a large job backlog: RPCs hang and the healthcheck times out. A plain restart does not help: startup reuses the local DB whenever it is present (download_checkpoint_to_local only runs when the db dir is absent — see controller/main.py), so docker restart / gcloud compute reset just reload the same bloated DB and re-wedge.

The fix is to roll the local DB back to a pre-spike checkpoint by hand. Run the steps below on the controller VM. Do this only when the user has asked you to recover a wedged controller.

Definitions used below — read them from the cluster config (config/marin.yaml):

• STATE_DIR — controller local state dir, default /var/cache/iris/controller (override: storage.local_state_dir). The DB lives in $STATE_DIR/db.
• REMOTE — storage.remote_state_dir (e.g. gs://marin-us-central2/iris/state). Checkpoints live at $REMOTE/controller-state/<epoch_ms>/{controller.sqlite3.zst,auth.sqlite3.zst}.

# 0. SSH to the controller VM (the GCE instance labelled iris-<prefix>-controller=true),
#    then set STATE_DIR/REMOTE from the cluster config so the commands below resolve.
gcloud compute ssh iris-controller-marin --zone <zone> --tunnel-through-iap
export STATE_DIR=/var/cache/iris/controller
export REMOTE=gs://<bucket>/iris/state

# 1. Pick a pre-spike checkpoint. The DB size is a good proxy for backlog/health:
#    a checkpoint much larger than its neighbours was already bloated — pick an
#    earlier, smaller one. Each subdir is named with its epoch_ms.
gcloud storage ls --long --readable-sizes "$REMOTE/controller-state/**/controller.sqlite3.zst"

# 2. Stop the controller (frees the RAM the bloated DB is consuming).
sudo docker stop iris-controller

# 3. Move the bloated DB ASIDE — never delete it. Startup reloads $STATE_DIR/db
#    if present, so this is what forces a fresh restore; keeping it makes the
#    rollback reversible.
sudo mv "$STATE_DIR/db" "$STATE_DIR/db.bloated.bak.$(date +%s)"

# 4. Restore the chosen checkpoint into $STATE_DIR/db using the controller image's
#    own download_checkpoint_to_local (handles the GCS pull, zstd decompress, and
#    the paired auth DB). Run it in a one-shot container so it reuses the VM's
#    ambient GCS credentials. Substitute <epoch_ms> from step 1.
IMAGE="$(sudo docker inspect --format='{{.Config.Image}}' iris-controller)"
sudo docker run --rm --network=host -v /var/cache/iris:/var/cache/iris "$IMAGE" \
    .venv/bin/python -c "from pathlib import Path; \
from iris.cluster.controller.checkpoint import download_checkpoint_to_local as restore; \
ok = restore('$REMOTE', Path('$STATE_DIR/db'), checkpoint_dir='$REMOTE/controller-state/<epoch_ms>'); \
raise SystemExit(0 if ok else 1)"

# 5. Confirm the restore actually produced a DB BEFORE starting (if it didn't, the
#    controller would reload the latest — often still-bloated — checkpoint on start).
test -f "$STATE_DIR/db/controller.sqlite3" || echo "RESTORE FAILED — do not start; move the backup back"

# 6. Start and verify it serves.
sudo docker start iris-controller
curl -sf http://localhost:10000/health && echo " controller healthy"

Rollback cost. Jobs and state created after the chosen checkpoint are dropped. Workers on separate VMs and other infrastructure are unaffected — they re-register with the recovered controller.

If it goes wrong. The previous DB is preserved at $STATE_DIR/db.bloated.bak.<ts>. To undo the rollback, docker stop, rm -rf $STATE_DIR/db, mv the backup back, and docker start.

Job Management

iris job run -- python train.py           # submit + stream logs
iris job list --state running             # filter by state
iris job logs /user/job-name -f           # follow job + child logs
iris job cancel /user/job-name            # exact job name + its children
iris job complete /user/job-name          # mark unfinished descendants successful, then stop them
iris job cancel --prefix /user/job-prefix # all jobs with this ID prefix
iris job describe /user/job-name          # per-task state, exit, duration, peak memory

The workload command hierarchy is:

Resource	Inspection	Actions
Job	list, describe, logs, wait	run, cancel, complete
Task	list, describe, events, logs, wait	exec, profile, preempt, fail
Attempt	describe, events, logs, wait	profile, preempt, fail

All three workload logs commands accept the same filters and --follow. Use job logs for the aggregate, task logs across a Task's Attempts, and attempt logs for one numbered Attempt. process logs is reserved for controller, worker, and task-runtime diagnostics.

Task actions without an attempt suffix target the current attempt. Attempt actions require /user/job/task:attempt and are accepted only while that attempt is still current. exec remains a Task action because its request identifies a Task rather than a numbered Attempt.

job logs returns the last 1000 lines by default. A multi-rank gang emits that many in seconds, so a grep for anything earlier in the run comes back empty:

iris job logs /user/job/child --max-lines 400000 --no-tail --substring "Saving checkpoint"

Submitting a GPU gang from a workstation

--cluster cw-* connects the CLI to that cluster's controller, which needs a kubectl port-forward the read-only CoreWeave token cannot open. It fails after a 90s timeout, while plain kubectl get works. Submit through the hub, which federates the job to the peer:

uv run iris --config lib/iris/config/marin.yaml job run --no-wait \
  --enable-extra-resources --target-cluster cw-us-east-08a --priority interactive \
  --cpu 2 --memory 8GB --disk 32GB --timeout 7200 \
  --job-name my-run-coord \
  -e WANDB_API_KEY "$WANDB_API_KEY" -e IRIS_PORT_JAX 32731 \
  -- python -m experiments.<launcher> --run-id my-run --dp-racks 1 --run

• The GPU gang is not this job. The submitted job is a small CPU coordinator that runs an experiment launcher; the launcher dispatches the accelerator gang via Fray as a child job at <coord-job-id>/<gang-name>. Pass --cpu/--memory for the coordinator alone and let the launcher size the gang.
• The working tree ships with the job. job run bundles the current workspace, so uncommitted and branch-only code runs as-is. The submit log prints the bundle size.
• IRIS_PORT_JAX must be unique per concurrent gang. Rank 0 binds and registers it for the JAX coordinator, and the default is shared cluster-wide.
• Only task_env reaches the container. Each cluster config's defaults.task_env carries MARIN_PREFIX and the object-store credentials, and nothing else. WANDB_API_KEY is not among them: pass it, or set WANDB_MODE=disabled for a run whose metrics do not matter.
• SchedulingGated on every task means the gang is queued. Kueue admits a gang all at once, so a busy cluster holds all of it. Same-band jobs queue behind each other; they do not preempt.
• --timeout covers the queue wait, and killing the coordinator kills the gang. A contested fleet can hold a gang for hours before admitting it; when the coordinator's deadline passes, its children are torn down mid-run (the tasks report killed with a preemption each). Size the timeout for wait plus run.

Reading a job's output needs a CoreWeave object-storage key exported as CW_KEY_ID/CW_KEY_SECRET; without one, s3://marin-us-east-02a raises NoCredentialsError. fsutil buckets reports which backends are reachable, and fsutil reads them:

uv run fsutil ls -l s3://marin-us-east-02a/tmp/my-run/step-1

For machine-readable job data, use the Iris Python client (IrisClient) directly.

job run gotchas

• Remote jobs only see env vars you put in the job spec. The submitter's shell env is not copied into the container. Pass required values explicitly: iris job run -e HF_TOKEN "$HF_TOKEN" -e WANDB_API_KEY "$WANDB_API_KEY" -- python train.py.
• --memory not --ram — unrecognized flags silently pass through to the command string.
• -e KEY VALUE uses two positional args. If $VALUE is unset, the parser eats the next token. Always quote: -e KEY "${VALUE}".
• --gpu requests hardware; --extra gpu requests the Python dependency extra. Need both for GPU JAX jobs.
• A job that dies in BUILDING with a uv sync error is failing setup before your command starts. The default is uv sync --all-packages --no-dev. Scope it with CLI --sync-package <member> or SDK EnvironmentSpec(sync_packages=[...]); skip setup entirely with CLI --no-sync or SDK EnvironmentSpec(setup_scripts=[]) for a bring-your-own image. The build log labels each step ([iris setup] step N/M) so you can tell which script failed. See "Task Setup" in AGENTS.md.
• Use --gpu or --tpu to request accelerators, instead of --region or --zone. Let Iris handle scaling group constraints. Use --region or --zone when you are trying to pin data to a particular location.
• --reserve is a hard zone constraint: it confines the job to a zone where the named accelerator has actually been obtained (empirically — a live, non-erroring slice in the region), and the job waits if none exists yet (an availability probe meanwhile scales the accelerator up). It does not hold capacity and does not attach accelerator devices. Use --tpu/--gpu on the task that needs hardware.
• executor_main parent jobs (e.g., canary ferries) submit GPU sub-tasks via Fray. The parent must be CPU-only (--cpu 1 --memory 2g), otherwise it hogs the GPU node and deadlocks. Memory at or above 4 GB requires --enable-extra-resources (see "Validator opt-in" below).

Task Operations

iris task describe /user/job/0                 # state, attempts, backend object, root cause
iris task events /user/job/0                   # retained backend + controller action timeline
iris task events /user/job/0:2                 # one attempt only
iris task exec /user/job/0 -- bash             # shell into running container
iris task exec /user/job/0 -- python -c "import jax; print(jax.devices())"

task events is the first stop when a pod or Kubernetes Event has already been garbage-collected. It queries iris.task_event across every retained attempt in the task's current job incarnation in one call and shows both backend observations (k8s/kueue, k8s/container) and controller decisions (iris/controller) in chronological order. Events are retained for up to 30 days. Kubernetes DisruptionTarget conditions are recorded as k8s/disruption events with the pod and node identity plus a snapshot of the node's taints, conditions, hardware/topology labels, and CoreWeave health annotations.

Default timeout is 60s. Use --timeout 300 for slow commands, --timeout -1 for no timeout (last resort).

The exec session is non-interactive and buffers output. To run a command that survives disconnect, wrap with nohup + &:

iris task exec /user/job/0 -- bash -c "nohup bash -c 'your-command > /tmp/out.log 2>&1' &"
iris task exec /user/job/0 -- cat /tmp/out.log   # check later

Task with no logs or W&B progress

Determine whether the task is pending, building, running, or terminal before treating missing logs or W&B metrics as a training stall:

iris task describe /user/job/0
iris task events /user/job/0
iris job logs /user/job

A task without a running container cannot produce task-local logs or W&B updates. For a running task, compare the current attempt and exit reason with driver/container logs and the resource request. A live process can still be compiling or blocked by host/device memory before its first optimizer update.

Keep inspection read-only. Record the state, attempt, exit reason, and resource request before restarting or signalling anything.

Log filtering

The Iris dashboard's full-log filter accepts a regular expression. For example, rank0.*train/loss matches log contents; an invalid expression such as an unbalanced [ is rejected. Fix the expression instead of treating an empty result as evidence that the task produced no matching logs.

The dashboard's find box searches only the lines loaded in the browser. Use the full-log filter when the target may be outside that segment; use find for visible lines.

Acting on a wedged Task or Attempt

When a scheduling bug or stuck node strands a task on a machine, force its current attempt terminal without touching the rest of the job:

iris task preempt /user/job/0                   # current attempt; reschedules if budget remains
iris task fail /user/job/0                      # current attempt; no retry
iris attempt preempt /user/job/0:3              # accepted only if attempt 3 is still current
iris attempt fail /user/job/0:3 --reason "bad output"

The action is queued on the controller and applied on the next control tick through the same finalization path the scheduler's preemptions use, so it shares one write transaction with the scheduler instead of racing it. Only tasks running on a worker (ASSIGNED / BUILDING / RUNNING) can be changed; pending or already-terminal tasks are rejected with a reason. preempted charges the preemption budget; failed is terminal with no retry.

Use job complete only when the workload should be recorded as successful. It marks the Job and every unfinished Task and Attempt SUCCEEDED, then stops their runtimes. job cancel records the Job as KILLED instead.

task preempt, task fail, and job cancel also read IDs from stdin (--stdin, or a literal - target) and take --dry-run. This is the query→act bridge: select the targets with SQL, preview, then fire. See "Bulk actions: query → act" below.

Recovering a stuck terminating Kubernetes pod

Use the use-iris skill when a CoreWeave pod remains after its Kubernetes deletion deadline. The Grafana K8s control plane dashboard classifies overdue pods; its alert fires only for node-bound, nonterminal GPU pods without finalizers.

The recovery order is safety-critical: record the node's existing cordon state, cordon it, quiesce the exact Iris attempt and every sibling workload, then use a CoreWeave force reboot if targeted graceful deletion still cannot stop the pod. Never force-delete the pod object while the old process may still be running. Kubernetes does not wait for kubelet confirmation, so replacement work can start while the old process still owns the GPU. Force-delete a stale object only after CoreWeave confirms the reboot completed (or process death is otherwise proven).

Process Inspection & Profiling

iris process status                         # controller resource usage
iris process status -t /system/worker/<id>  # worker process status
iris process logs -f                        # follow controller logs
iris process logs --level WARNING           # filter by level
iris process profile threads                # thread dump (prints to stdout)
iris process profile cpu -d 10              # 10s CPU profile (writes .speedscope.json)
iris process profile mem                    # memory flamegraph (writes .html)
iris process profile cpu -t /user/job/0     # profile a running task container

Prefer iris process profile over SSH for profiling — it uses the /system/process RPC and avoids direct VM access. SSH is a fallback only when the RPC doesn't cover your needs.

GPU environments set NCCL_RAS_ENABLE=1, NCCL_DEBUG=INFO, and NCCL_DEBUG_SUBSYS=INIT,BOOTSTRAP,ENV,NET,GRAPH,TUNING,RAS. The default timestamp is [%F %T.%3f]. Short debug-smoke jobs may additionally select COLL,PROXY,NVLS,REG; do not use TRACE or CALL for normal runs.

GPU Levanter runs persist NCCL's job-global communicator view from JAX process 0 every two minutes. The probe is bounded and records unavailable, failed, and timed-out polls explicitly. See docs/ops/training-stall-alert-contract.md for metric semantics and a bounded Finelog query.

For a read-only one-shot check, first confirm the target task is RUNNING; a BUILDING task has no NCCL listener. Query from the task's own container and network namespace:

iris --cluster=<cluster> task exec <task-id> --timeout 15 -- \
  python -m rigging.telemetry.probes.nccl_client --timeout 8

The command sends one status request, does not retry or write inside the task, and limits both socket time and response size. Exit code 3 means the task-local listener was unavailable. Do not restart, signal, or otherwise modify the task to make this check succeed.

Scheduler & Autoscaler

iris rpc controller get-scheduler-state        # pending queue, resource constraints, priority bands
iris rpc controller get-autoscaler-status       # per-group demand, backoff, failures, quota
iris rpc controller get-provider-status         # scheduling events, cluster capacity
iris cluster vm status                          # scale groups with slice counts

Priority bands: PRIORITY_BAND_SYSTEM (admin-only Iris, Finelog, and hero work), PRIORITY_BAND_PRODUCTION (admin-only critical work), PRIORITY_BAND_INTERACTIVE (default), and PRIORITY_BAND_BATCH (opportunistic). CLI SYSTEM submissions require --system-reason containing hero, finelog, or iris. See docs/priority-bands.md.

get-scheduler-state's running_buckets is a live DB projection (tasks where state=RUNNING AND current_worker_id IS NOT NULL), not an independent in-memory set. It is self-consistent within a single call but skews across separate RPC calls on a busy cluster — a task can move workers between two calls seconds apart. Do not diagnose a "worker running a task the tasks-table doesn't show" by diffing running_buckets against a separately-timed iris query; that mismatch is snapshot skew, not a leak. To check for a genuine leak, use one atomic query (e.g. RUNNING tasks whose current_worker_id is absent from workers).

SQL Queries

The controller exposes its SQLite DB via RPC:

iris query "SELECT state, count(*) FROM jobs GROUP BY state"
iris query "SELECT state, count(*) FROM tasks GROUP BY state" -f csv

Never modify the controller database without explicit user approval — read-only queries only, even on offline checkpoints.

State codes: 1=PENDING, 2=BUILDING, 3=RUNNING, 4=SUCCEEDED, 5=FAILED, 6=KILLED, 7=WORKER_FAILED, 8=UNSCHEDULABLE, 9=ASSIGNED (tasks only), 10=PREEMPTED (tasks only), 11=COSCHED_FAILED (tasks only — a coscheduled sibling bounced when its gang-mate went down; terminal, not charged preemption budget), 12=MISSING.

Sharp edges

• Active states: 2 (BUILDING), 3 (RUNNING), and 9 (ASSIGNED) — not just RUNNING. Forgetting ASSIGNED causes resource attribution misdiagnosis.
• Committed resources: workers has committed_cpu_millicores, committed_mem_bytes, etc. Total capacity is in metadata_proto (serialized protobuf). Available = capacity - committed.
• request_proto: serialized protobuf in jobs.request_proto. You need protobuf to decode — plain SQL cannot inspect task constraints.

Useful queries

-- Failed jobs with errors
SELECT job_id, error, exit_code FROM jobs WHERE state=5 ORDER BY submitted_at_ms DESC LIMIT 10;

-- Quota-blocked scale groups
SELECT name, consecutive_failures, quota_reason FROM scaling_groups
WHERE consecutive_failures > 0 OR quota_reason != '';

-- Active slices (GCP)
SELECT slice_id, lifecycle, scale_group, worker_ids FROM slices WHERE lifecycle='ready';

-- Task attempt history (debugging retries)
SELECT task_id, attempt_id, state, exit_code, error FROM task_attempts
WHERE task_id LIKE '%<job_fragment>%' ORDER BY attempt_id;

Controller audit events (event=<action> entity=<id> trigger=<trigger> <k=v ...>) are emitted as structured logger.info lines — query them through iris process logs with its built-in --substring filter, not via SQL.

process logs has no --since flag (its only options are -t/--target, --level, -f/--follow, --max-lines, --substring). Do not pipe the raw output through grep — an unrecognized --since is dropped and a post-hoc grep over the default window silently returns nothing. Filter server-side instead:

iris process logs --substring='event=worker_failed' --max-lines 200
iris process logs --substring='<slice-or-worker-or-job-id>' --max-lines 40   # trace one entity's whole lifecycle

Useful event names (the action passed to log_event): worker_registered, worker_failing, worker_pruned, assignment_queued, task_preempted, task_unschedulable, task_timeout, job_submitted, slice_ready, slice_pruned, reconcile_rpc_failed. task_preempted records reason=Preempted by <preemptor-task-id>, so substring-tracing a victim shows exactly which higher-priority job evicted it.

Full table list: iris query "SELECT name FROM sqlite_master WHERE type='table'".

Bulk actions: query → act

iris query is admin-only and read-only, so it is the safe surface for finding the exact set of tasks/jobs you want to act on. iris task preempt, iris task fail, and iris job cancel read IDs from stdin (--stdin, or a literal -), so a query pipes straight into an action — no hand-copying ids. Stdin parsing is CSV-tolerant: it takes the first field of each line and keeps only ids (leading /), so a -f csv header row and trailing columns are dropped automatically.

Always --dry-run first to confirm the set, then re-run without it:

# Drain everything EXCEPT one protected job off a slice, so it can bind its ports.
SLICE=marin-tpu-v4-reserved-2048-us-central2-b-...
SEL="SELECT t.task_id FROM tasks t JOIN workers w ON t.current_worker_id=w.worker_id
     WHERE w.slice_id='$SLICE' AND t.state IN (2,3,9) AND t.job_id NOT LIKE '/larry/%'"

iris query -f csv "$SEL" | iris task preempt --stdin --dry-run          # preview
iris query -f csv "$SEL" | iris task preempt --stdin --reason "drain slice for /larry"

task preempt reschedules under the Task's preemption budget; task fail does not retry. Select Task IDs (t.task_id, task index kept) when you only need to clear specific workers. Canceling the Job would stop all of its Tasks.

Canonical joins (the schema doesn't pre-wire these, so keep them here):

-- Which scale group is the size-N slice? (find the slice_id to target)
SELECT scale_group, device_variant, count(*) AS workers, count(DISTINCT slice_id) AS slices
FROM workers WHERE device_type='tpu' GROUP BY scale_group, device_variant;

-- Everything occupying a slice's workers, by job and task state.
SELECT t.job_id, t.state, count(*) FROM tasks t
JOIN workers w ON t.current_worker_id=w.worker_id
WHERE w.slice_id='<slice_id>' AND t.state IN (2,3,9) GROUP BY t.job_id, t.state;

-- Co-tenants sharing a worker VM with a given job (CPU tasks bin-packed onto
-- TPU hosts show up here — a common source of host-global port collisions).
SELECT t.job_id, t.task_id, w.md_tpu_worker_id FROM tasks t
JOIN workers w ON t.current_worker_id=w.worker_id
WHERE w.worker_id IN (
  SELECT current_worker_id FROM tasks WHERE job_id LIKE '/larry/%' AND state IN (2,3,9)
) AND t.job_id NOT LIKE '/larry/%' AND t.state IN (2,3,9);

To dump rather than act, feed the same selection to iris job logs / iris job describe per ID, or read the task rows directly with a wider SELECT.

Offline checkpoint analysis

For slow queries, query offline. Never run expensive queries against the live DB — they stall the controller.

# Download the checkpoint file (path printed by command above)
sqlite3 /tmp/controller.sqlite3 "SELECT ..."

Prefer to use the last checkpoint from GCS. Only take a new controller checkpoint if this is too old:

iris cluster controller checkpoint

Stats Namespaces

Time-series measurements live in finelog stats namespaces, not the controller SQLite DB (see AGENTS.md "Decisions vs measurements"). The controller bundles a StatsService alongside its log server (started by _start_local_log_server in controller/controller.py); both are mounted on the same uvicorn app and reachable at the /system/log-server endpoint advertised by cluster_config.endpoints (or, in fallback mode, at the URL printed as Local log server ready at <addr> on controller startup).

Namespaces:

• iris.worker — per-tick host utilization (cpu, mem, disk, running task count, net bps), keyed by ts.

• iris.task — per-attempt task resource snapshots, keyed by ts. Worker daemons write their process readings directly. On Kubernetes, each iris-node-agent samples the task containers on its node from kubelet /metrics/resource every 30 seconds. CPU is derived from consecutive cumulative counter samples, so the first row after an agent start reports zero CPU. Memory uses the working-set gauge and an agent-local peak; an agent restart resets that peak. Kubelet resource metrics do not expose container filesystem usage, so Kubernetes rows report zero disk usage. The node-agent service account requires get on nodes/proxy.

• iris.task_event — up to 30 days of deduplicated backend verdicts and state-changing controller actions per task attempt. Query all attempts with iris task events /user/job/0, or directly:

  SELECT attempt_id, ts, type, reason, message, source, count
  FROM "iris.task_event"
  WHERE task_id='/user/job/0' AND attempt_uid='<uid from iris task describe>'
  ORDER BY ts ASC;
  

  Kubernetes disruptions use source='k8s/disruption'. node_provider_id and node_system_uuid identify a physical machine; node_boot_id distinguishes reboots. To inspect taint-manager evictions across the fleet:

  SELECT cluster, ts, task_id, attempt_id, pod_name, node_name,
         node_uid, node_provider_id, node_boot_id, node_system_uuid,
         reason, message, pod_conditions_json, node_taints_json,
         node_conditions_json
  FROM "iris.task_event"
  WHERE source='k8s/disruption' AND reason='DeletionByTaintManager'
  ORDER BY ts DESC;
  

  One node drain can disrupt many pods. Treat rows with the same cluster, node_system_uuid, node_boot_id, and nearby timestamps as one machine incident before counting repeat offenders.

• iris.task_state — controller-emitted (every 30s) task counts by state per root job, plus oldest_pending_age_ms / oldest_building_age_ms wait ages, keyed by root_job_id. The root_job_id="" row is the per-cluster rollup, written even when idle — its absence means the controller is down. Feeds fleet-wide stuck-BUILDING alerting and queue-depth history.

• iris.admission_probe — on Kubernetes clusters, the outcome (every 60s) of a dryRun=All canary pod apply that traverses the full admission chain, keyed by outcome (ok/failed with error_class, latency, truncated message). failed rows (or silence) detect fail-closed admission webhooks before any task pod exists.

• iris.profile — per-capture profile blobs (cpu/memory/thread, periodic or on-demand), keyed by source so the dashboard's per-source list query prunes via parquet row-group min/max. Filter on source (a task path like /user/job/.../<index>, /system/worker/<id>, or /system/controller) and type (cpu/memory/thread). format is the blob encoding — the GCE/TPU worker's periodic CPU captures are py-spy speedscope JSON; the k8s backend's periodic captures are py-spy thread dumps (type=thread), since a hung collective samples no CPU but a thread dump pinpoints where every rank is blocked. vm_id is the writer VM (worker id, controller-self, or k8s/<node-or-pod>). To find a hang, read the last periodic thread capture per source before the freeze.

Retention is finelog segment-based. Target for iris.profile is 7 days.

Get a profile for a task — open the dashboard task page and use the "Profile history" panel; rows are CPU captures from the worker's 10-minute periodic loop plus any on-demand captures, click to download. To capture on demand, hit the "Profile now" button on the task page, the worker page (/system/worker/<id>), or the controller status page (/system/controller).

Profiles are written by the worker (periodic CPU + on-demand all types), by K8sTaskProvider (periodic thread dumps of every running pod + on-demand all types), and by the controller for /system/controller self-captures. The k8s backend has no per-node worker daemon, so its PeriodicProfiler runs the equivalent 10-minute loop controller-side (profile_poll_interval), dumping each running pod's threads off the reconcile path.

Query the namespace directly with the finelog CLI (opens a tunnel to the cluster's finelog deployment named by finelog.config):

cd lib/finelog
uv run finelog query marin "SELECT source, type, format, count(*) FROM \"iris.profile\"
  WHERE source LIKE '/user/job/%' AND type='cpu' GROUP BY 1,2,3"

To aggregate a whole job's CPU profiles into a per-worker-sub-job breakdown + merged flamegraph, use scripts/job_profile_summary.py — it resolves the cluster's finelog deployment, pulls every CPU capture under a job (and its descendant sub-jobs), parses the speedscope stacks, and reports where CPU is spent:

uv run python scripts/job_profile_summary.py /user/job/id          # per-sub-job + top leaves
uv run python scripts/job_profile_summary.py <dashboard-url>       # accepts iris.oa.dev URLs
uv run python scripts/job_profile_summary.py /user/job/id --subjob <name> --show-stacks
uv run python scripts/job_profile_summary.py /user/job/id -o merged.folded --svg flame.svg

Users & Auth

iris login                            # IAP clusters: cache the IAP edge refresh token locally
iris rpc controller list-users        # active users with task/job counts
iris user budget list                 # per-user budget limits

Users authenticate only through IAP: the GCLB validates an OIDC token at the edge and forwards a signed assertion the controller verifies; the controller mints no user token. iris login runs the browser desktop-OAuth flow once and caches the IAP edge refresh token (each RPC silently re-mints the short-lived edge token from it). Authorization is config-driven — roles are resolved per request from an in-memory RolePolicy built from the cluster config at controller start (admins from auth.admin_users, the IAP unprovisioned_role for everyone else). There is no users table and no reconciliation: config is the sole source of truth. To deprovision a user, remove them from auth.admin_users and reload/restart the controller; the rebuilt policy resolves them to the non-admin default on their next request (no token to revoke — the role is resolved per request). The only fleet-wide credential kill switch is rotating the cluster signing key (iris cluster init-keys + redeploy), which re-auths every worker.

Calling the IAP endpoint with curl

The built-in Marin desktop OAuth client is configured as an IAP programmatic client. The first command opens a browser and caches a long-lived refresh token in ~/.config/marin/credentials/marin.json:

uv run iris --cluster marin login

Mint a short-lived IAP ID token from the cached credentials and send it in Proxy-Authorization:

IAP_TOKEN="$(uv run python -c 'from rigging.credentials import iap_edge_provider; print(iap_edge_provider("marin").get_token())')"
curl --fail-with-body \
  --header "Proxy-Authorization: Bearer ${IAP_TOKEN}" \
  https://iris.oa.dev/proxy/system.log-server/health

Proxy-Authorization is reserved for IAP. Keep Authorization available for an Iris JWT when a controller route requires one. When auth.iap.signed_header_audience is configured, the controller accepts the identity assertion added by IAP and resolves the caller's Iris role by email.

The path proxy encodes / in an endpoint name as .. The finelog endpoint /system/log-server is therefore system.log-server in the public URL. /proxy/system/finelog addresses an endpoint named /system with a finelog subpath and does not reach the controller's finelog server.

Troubleshooting

Symptom	Diagnostic
Job stuck PENDING	iris rpc controller get-scheduler-state for constraints. Check quota: iris query "SELECT name, consecutive_failures, quota_reason FROM scaling_groups WHERE quota_reason != ''"
Workers not joining (GCP)	iris cluster vm status for slice lifecycle. SSH to VM, check bootstrap logs.
Autoscaler not scaling	iris rpc controller get-autoscaler-status — check backoff_until_ms, consecutive_failures.
Task retrying	iris job describe /user/job — per-task state and exit codes; iris job logs /user/job for the per-attempt errors.
Task failed with exit 137 / suspected OOM	iris job describe /user/job — per-task peak memory + exit code. If most shards peak near the container memory limit, raise --memory on resubmit.
Dashboard unreachable	Verify tunnel is alive. curl -sf http://localhost:10000/health.
ArchMismatchImageExecuted alert, or tasks die instantly with exit 255	See Image architecture mismatch.

Image architecture mismatch

An image built for the wrong CPU architecture fails at exec. The pod log shows exec /usr/local/bin/python: exec format error and the container exits 255 in under a second. The ArchMismatchImageExecuted alert reports an evidence column on /k8s/arch_mismatch: message means the kubelet captured that text and the mismatch is confirmed, signature means only the exit-255-in-under-a-second pattern matched, which an unrelated instant failure also produces. Confirm a signature row before acting.

Only containers set to terminationMessagePolicy: FallbackToLogsOnError yield message evidence — Iris sets it on task and stage-workdir. Everything else reports signature.

Confirm the tag really lost the architecture. A multi-arch tag is an index listing every platform:

TOK=$(curl -s "https://ghcr.io/token?scope=repository%3Amarin-community%2Firis-task%3Apull&service=ghcr.io" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['token'])")
curl -s -H "Authorization: Bearer $TOK" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  https://ghcr.io/v2/marin-community/iris-task/manifests/latest \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print([m.get('platform') for m in d.get('manifests',[])] or 'SINGLE-ARCH')"

A platform: unknown entry is buildx provenance, not a real architecture. If linux/arm64 is missing, the tag itself is broken — rebuild and push a two-platform index before touching any node. Evicting first only re-pulls the same broken image.

Then check what the node actually cached, since imagePullPolicy: IfNotPresent means a node never refreshes a tag it already holds:

kubectl --context <ctx> -n iris debug node/<node> --image=busybox --profile=sysadmin \
  --attach=false -- chroot /host crictl inspecti ghcr.io/marin-community/iris-task:latest

Once the registry is confirmed good, drop the stale copy so the node re-pulls:

kubectl --context <ctx> -n iris debug node/<node> --image=busybox --profile=sysadmin \
  --attach=false -- chroot /host crictl rmi ghcr.io/marin-community/iris-task:latest

Do not evict a node whose cached image still works unless the registry tag is confirmed good. On 2026-07-29 the only arm64 builds left in ghcr were the ones cached on the nodes; the tag had been overwritten with an amd64-only manifest, so an eviction would have been unrecoverable. Pinning the config to an index digest avoids both the drift and the overwrite.

GCP (TPU) Operations

Connecting

# SSH tunnel (IAP)
gcloud compute ssh iris-controller-marin --zone=us-central1-a \
  --project=hai-gcp-models --tunnel-through-iap -- -L 10000:localhost:10000 -N

# Then: iris --controller-url=http://localhost:10000 ...
# Or preferred named-cluster auto-tunnel: iris --cluster=marin ...
# Exact-file form for custom or pinned configs: iris --config=lib/iris/config/marin.yaml ...

Configs: marin.yaml (production), marin-dev.yaml (dev, smaller scale caps).

GCP Resources

# Controller VM
gcloud compute instances list --project=hai-gcp-models \
  --filter="labels.iris-marin-controller=true" --format="table(name,zone,status)"

# Iris-managed worker VMs
gcloud compute instances list --project=hai-gcp-models \
  --filter="labels.iris-marin-managed=true" --format="table(name,zone,status)"

# TPU VMs (all zones)
gcloud compute tpus tpu-vm list --project=hai-gcp-models --zone=- \
  --format="table(name,zone,state,acceleratorType)" | head -30

TPU Bad-Node Recovery

Trigger patterns (bad node, not a code bug):

• RuntimeError: No accelerator found. Please run on a TPU or GPU.
• FAILED_PRECONDITION
• Device or resource busy

Recovery: extract worker IP from logs -> map to VM name (gcloud compute tpus tpu-vm list --zone <ZONE> --format="table(name,networkEndpoints[0].ipAddress)") -> delete bad node (gcloud compute tpus tpu-vm delete <NAME> --zone <ZONE> --quiet) -> resubmit job.

Only delete the specific bad node. If multiple nodes fail simultaneously or the same node fails again, escalate to the user.

GCP State

State dir: gs://marin-us-central2/iris/<cluster>/state/ — contains bundles/ (code packages) and controller-state/ (SQLite checkpoints). Per-task log parquet segments are shipped separately by finelog under <finelog.remote_log_dir>/log/ (see lib/finelog/config/<cluster>.yaml).

GCP Gotchas

• Quota is the primary scaling bottleneck. The autoscaler backs off exponentially per scale group. Check with iris rpc controller get-autoscaler-status.
• Stuck TPU VMs. Occasionally a TPU VM gets stuck in DELETING for days. Check: gcloud compute tpus tpu-vm list --project=hai-gcp-models --zone=- --filter="state=DELETING".

---

CoreWeave (GPU) Operations

docs/coreweave.md describes the Kubernetes architecture. The named cw-* configs in lib/iris/config/ are the source of truth for each cluster's kubeconfig, context, namespace, and accelerator groups.

Access and read-only status

CoreWeave configs pin a Kubernetes context, so no kubectl config use-context step is needed. The configured kubeconfig path is used unless KUBECONFIG is set in the operator shell.

CLUSTER=cw-rno2a

uv run iris cluster list
uv run iris --cluster="$CLUSTER" cluster status                 # controller and port-forward
uv run iris --cluster="$CLUSTER" rpc controller list-backends  # accelerators, nodes, and availability
uv run iris --cluster="$CLUSTER" cluster dashboard              # blocks until Ctrl+C

CoreWeave has no Iris worker daemon or Iris autoscaler. Use list-backends, not the worker count in cluster status, for the Kubernetes resource view.

For a pending task, inspect Kueue admission and NodePool provisioning without rendering Pod environment values:

CW_KUBECONFIG=~/.kube/coreweave-iris
CW_CONTEXT=<platform.coreweave.kube_context>
CW_NAMESPACE=<kubernetes_provider.namespace>

kubectl --kubeconfig "$CW_KUBECONFIG" --context "$CW_CONTEXT" -n "$CW_NAMESPACE" \
  get pods -l iris.task_id \
  -o custom-columns='POD:.metadata.name,PHASE:.status.phase,SCHEDULING:.status.conditions[?(@.type=="PodScheduled")].reason,NODE:.spec.nodeName'
kubectl --kubeconfig "$CW_KUBECONFIG" --context "$CW_CONTEXT" -n "$CW_NAMESPACE" \
  get workloads.kueue.x-k8s.io \
  -o custom-columns='WORKLOAD:.metadata.name,QUOTA:.status.conditions[?(@.type=="QuotaReserved")].status,ADMITTED:.status.conditions[?(@.type=="Admitted")].status,REASON:.status.conditions[?(@.type=="QuotaReserved")].reason'
kubectl --kubeconfig "$CW_KUBECONFIG" --context "$CW_CONTEXT" \
  get nodepools.compute.coreweave.com \
  -o custom-columns='POOL:.metadata.name,CURRENT:.status.currentNodes,QUEUED:.status.queuedNodes,IN_PROGRESS:.status.inProgressNodes,CONDITION:.status.conditions[*].type,STATUS:.status.conditions[*].status,REASON:.status.conditions[*].reason'

SchedulingGated means Kueue still holds the Pod. QuotaReserved=True means the Workload has reserved quota; check Admitted if the Pod remains gated. NodePool conditions show provisioning failures or delays. These projections omit Pod environment values; do not use kubectl describe pod on task Pods.

If Iris reports that the configured context does not exist, compare the shell override and the canonical CoreWeave kubeconfig before changing either config:

printf 'KUBECONFIG=%s\n' "${KUBECONFIG:-<unset>}"
rg -n 'kubeconfig_path|kube_context|namespace' "lib/iris/config/${CLUSTER}.yaml"
kubectl --kubeconfig ~/.kube/coreweave-iris config get-contexts -o name

env -u KUBECONFIG uv run iris --cluster="$CLUSTER" cluster status

cluster status, list-backends, task describe, task events, logs, and plain Kubernetes get calls are read-only. kubectl describe pod does not mutate the cluster, but it can print literal environment values; avoid it on task Pods. Starting, stopping, or restarting a cluster or controller changes shared infrastructure. Run those commands only with explicit user approval; use the use-iris skill for controller rollouts. Direct Kubernetes changes such as apply, delete, scale, drain, cordon, and uncordon also require explicit approval.

Public LoadBalancer reachability

Federation reaches a CoreWeave controller through the public Traefik LoadBalancer. A direct iris --cluster=<name> cluster status uses a Kubernetes port-forward, so it can succeed while the federation route is unreachable.

Test each layer separately:

KUBECONFIG=~/.kube/coreweave-iris
CONTEXT=<platform.coreweave.kube_context>
NAMESPACE=<platform.coreweave.namespace>
HOST=<provisioning.coreweave.federation_dns.hostname>

# Controller and ingress objects.
kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n "$NAMESPACE" \
  get pods,svc,ingress,certificate -o wide
kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n traefik \
  get pods,svc,endpointslice -o wide

# Controller Service and Traefik route from inside the cluster.
kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n "$NAMESPACE" \
  exec deploy/iris-controller -- sh -c \
  'curl -sS -o /dev/null -w "controller=%{http_code}\n" http://iris-controller-svc:10000/health'
kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n "$NAMESPACE" \
  exec deploy/iris-controller -- sh -c \
  "curl -ksS -o /dev/null -w 'ingress=%{http_code}\n' https://$HOST/health"

# The same public route from outside CoreWeave.
curl -vk --connect-timeout 5 --max-time 10 "https://$HOST/health"

Interpret the result at the first failing layer:

• Controller 200 and ingress 403 mean the backend, Traefik route, and ipAllowList middleware are active. 403 is expected when the test source is not allowlisted.
• An external 403 proves the public route works. If the federation parent is also rejected, compare its observed egress IP with the live iris-federation-ipallowlist Middleware.
• An external 503 reaches Traefik but not a healthy backend. Inspect the Traefik EndpointSlice and controller Service.
• A public VIP that answers inside the cluster but times out from multiple external networks points to LoadBalancer route propagation. Check Cilium's managed BGP session before escalating:

TRAEFIK_NODE=$(kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" \
  -n traefik get pod -l app.kubernetes.io/name=traefik \
  -o jsonpath='{.items[0].spec.nodeName}')
CILIUM_POD=$(kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" \
  -n cw-cilium-system get pod --field-selector "spec.nodeName=$TRAEFIK_NODE" \
  -o jsonpath='{.items[0].metadata.name}')
PEER=$(kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" \
  get ciliumbgpclusterconfig cilium-bgp \
  -o jsonpath='{.spec.bgpInstances[0].peers[0].peerAddress}')

kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n cw-cilium-system \
  exec "$CILIUM_POD" -c cilium -- cilium-dbg bgp peers
kubectl --kubeconfig "$KUBECONFIG" --context "$CONTEXT" -n cw-cilium-system \
  exec "$CILIUM_POD" -c cilium -- \
  cilium-dbg bgp routes advertised ipv4 unicast peer "$PEER"

An established BGP session that advertises the public VIP, combined with an external TCP timeout, requires a CoreWeave route-export escalation. Restarting Iris or Traefik does not repair that layer.

CoreWeave kernel deadlock pending reboot

CoreWeaveNodeKernelDeadlock means CoreWeave reported the structured node condition KernelDeadlock=True. The node lifecycle controller normally cordons the node and moves it toward production-reboot; tenant pods can block that transition even while Ready=True.

Read the canonical kubeconfig and context from the cluster config, then inspect the condition and blockers:

kubectl --kubeconfig <kubeconfig> --context <context> get node <node> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl --kubeconfig <kubeconfig> --context <context> get pods --all-namespaces \
  --field-selector spec.nodeName=<node> -o wide
kubectl --kubeconfig <kubeconfig> --context <context> get pdb --all-namespaces

Do not uncordon a node that CoreWeave cordoned. If PendingPhaseState names production-reboot and CWActive lists tenant Deployments, get operator approval before changing workloads. Record each owner's original replica count and pod template, then choose an action by workload semantics:

Blocker	Reboot-unblocking action
Stateless or leader-elected Deployment	Add one replica, wait for it to become Ready on a healthy node, then delete only the pod bound to the deadlocked node. The current cert-manager and Kueue controllers are in this class.
Singleton controller with persistent state or no concurrency protection	Do not overlap replicas. Confirm that its state is durable, accept the control-plane outage, and scale it to zero. The Iris controller uses a Recreate strategy and a PVC, so it belongs here.
Availability service with a PodDisruptionBudget or hard placement constraints	First provide compatible capacity. If that is unavailable, use an operator-approved temporary placement change and wait for a Ready replacement. Scale to zero only with explicit approval for the resulting outage and after confirming that recovery does not depend on the service. Traefik belongs here.
Running Iris task pod	Preserve the Iris state transition, not the pod: obtain the canonical attempt ID from IRIS_TASK_ID, stop the parent job or mark the attempt preempted, then delete the exact pod normally so Iris can retry it. Node-local task images and caches are disposable.
Completed or failed Iris task pod	Delete the exact pod normally. It does not need replacement capacity.
StatefulSet, local-volume workload, unmanaged pod, or unknown owner	Stop and escalate. Do not infer that another replica is safe.

A PodDisruptionBudget states the desired availability but does not prove that replicas may run concurrently. Scaling an owner to zero also bypasses the eviction protection that operators often expect from a PodDisruptionBudget, so treat it as an explicit outage decision. Change one owner at a time and confirm that CWActive drops the blocker before continuing. Do not delete provider DaemonSets or use kubectl drain --force.

Iris coordinator task PDBs follow the job's priority band. SYSTEM and PRODUCTION use minAvailable: 1 and intentionally block voluntary eviction. INTERACTIVE and BATCH use maxUnavailable: 1; CoreWeave may evict those pods during a drain, and Iris records the disruption as PREEMPTED and retries it within the job's preemption budget. For a SYSTEM or PRODUCTION blocker, follow the running-task procedure in the table above. Do not weaken its live PDB to recover a node without the job owner's approval.

If a replacement remains Pending because no healthy node satisfies its required node affinity or pod anti-affinity, stop before deleting the original pod. Provision compatible capacity when possible. Record and restore any temporary placement change after the reboot. Use the use-iris skill for the exact Iris task retry sequence or when a bound pod does not terminate.

CoreWeave should continue the pending reboot without a separate Iris restart. Before restoring workloads, verify that the provider operation completed, the node boot ID changed, KernelDeadlock=False, Ready=True, CoreWeave returned the node lifecycle state to production, and both the cordon and any node.coreweave.cloud/reserved taint are gone. Restore singleton controllers and original replica counts first, wait for them to become Ready, then remove temporary capacity or placement changes.

CI Workflows

Workflow	Trigger	What
marin-canary-ferry.yaml	Daily 6AM UTC	TPU canary on GCP (marin-dev.yaml)
marin-canary-ferry-coreweave.yaml	Daily 10AM UTC	GPU canary on CW — shares iris-ci controller + H100 nodepool with iris-smoke-coreweave.yaml (concurrency group iris-coreweave-ci-shared)
iris-smoke-coreweave.yaml	PRs touching lib/iris/	CW integration tests (warm cluster)
ops-docker-images.yaml	workflow_dispatch / Sun 02:00 UTC	Rebuilds + pushes SHA-pinned iris-{controller,worker,task} images to GHCR (see Controller Restart)

# Trigger manually
gh workflow run "<workflow name>" -R marin-community/marin --ref main
# View failed run
gh run view <run-id> -R marin-community/marin --log-failed | tail -50