Task States Reference

This document describes the TaskState enum, the state machine governing task lifecycle, retry semantics, and how states appear in the dashboard.

A task is the unit of execution in Iris. Each job expands into one or more tasks (controlled by replicas). Tasks are independently scheduled, retried, and tracked. Job state is derived from task state counts -- there is no independent job state machine.

State Diagram

                          +-----------+
                          |  PENDING  |<-----------------+
                          +-----+-----+                  |
                                |                        |
                          dispatch to worker              |
                                |                        |
                                v                        |
                          +-----------+                  |
                          | ASSIGNED  |                  |
                          +-----+-----+                  |
                                |                        |
                        worker starts task               |
                                |                        |
                                v                        |
                          +-----------+                  |
                          | BUILDING  |                  |
                          +-----+-----+                  |
                                |                        |
                        build completes                  |
                                |                        |
                                v                        |
                          +-----------+                  |
                          |  RUNNING  |                  |
                          +-----+-----+                  |
                                |                        |
            +-------------------+-------------------+    |
            |                   |                   |    |
            v                   v                   v    |
      +-----------+       +-----------+     +------------+
      | SUCCEEDED |       |  FAILED   |---->| retry      |
      +-----------+       +-----------+     +------------+
                                |                  ^
                                | exhausted        |
                                v                  |
                          (terminal)               |
                                                   |
                          +-----------+            |
                          |WORKER_FAIL|------------+
                          +-----------+
                                |
                                | exhausted
                                v
                          (terminal)

                          +-----------+            |
                          | PREEMPTED |            |
                          +-----------+            |
                                ^                  |
                                | exhausted        |
                                |                  |
                          +-----------+            |
                          | preempt   |------------+
                          | (ctrl)    |
                          +-----------+

      Other terminal states: KILLED, UNSCHEDULABLE (never retried)

State Table

State	Proto Value	Terminal	Retriable	Set By	Dashboard Display
UNSPECIFIED	0	--	--	Default zero value; never used in practice	unspecified (grey)
PENDING	1	No	--	Job submission (_on_job_submitted), retry requeue (_requeue_task)	pending (amber)
ASSIGNED	9	No	--	Scheduler dispatch (_on_task_assigned / create_attempt)	assigned (orange)
BUILDING	2	No	--	Worker Reconcile observation; worker sets this during bundle download and dependency sync	building (purple)
RUNNING	3	No	--	Worker Reconcile observation; worker sets this when user command starts	running (blue)
SUCCEEDED	4	Yes	No	Worker Reconcile observation; task exited with code 0	succeeded (green)
FAILED	5	Yes	Yes	Worker Reconcile observation; task exited with non-zero code	failed (red)
KILLED	6	Yes	No	Controller: job cancellation (_on_job_cancelled), job failure cascade (_mark_remaining_tasks_killed), per-task timeout	killed (grey)
WORKER_FAILED	7	Yes	Yes	Controller: worker death cascade (ops.worker.fail)	worker_failed (purple)
UNSCHEDULABLE	8	Yes	No	Controller: scheduling timeout expired (apply_terminal_decisions_batch)	unschedulable (red)
PREEMPTED	10	Yes	Yes	Controller: priority preemption with budget exhausted (apply_terminal_decisions_batch); K8s backend: control-plane disruption of the pod	preempted (orange)
COSCHED_FAILED	11	Yes	No	Controller: coscheduled sibling cascade (_terminate_coscheduled_siblings)	cosched_failed (red)

State Transitions in Detail

PENDING

The initial state for every task. Set in two contexts:

1. Job submission: _on_job_submitted calls expand_job_to_tasks, which creates ControllerTask objects with state=TASK_STATE_PENDING. Tasks are enqueued into the priority-sorted scheduling queue.

2. Retry requeue: _requeue_task resets task.state to TASK_STATE_PENDING and re-inserts the task into the scheduling queue. This happens after a retriable FAILED or WORKER_FAILED when retry budget remains.

ASSIGNED

Set by _on_task_assigned after the scheduler selects a worker and commits resources. create_attempt creates a new ControllerTaskAttempt in TASK_STATE_ASSIGNED state. The task is now bound to a specific worker and consuming its resources.

The worker has not yet acknowledged the task -- it will receive the dispatch in the next controller Reconcile cycle (the DesiredAttempt.run intent carries the AttemptSpec.request payload on this one tick).

BUILDING

Reported by the worker as a Reconcile observation. The worker transitions internally:

• PENDING -> BUILDING when bundle download starts (task_attempt.py:433)
• Later, BUILDING again when dependency sync starts (task_attempt.py:549)

The controller processes this transition in apply_task_updates. Note: if the worker reports PENDING, the controller ignores it to prevent regressing an ASSIGNED task and confusing the building-count backpressure window.

RUNNING

Reported by the worker via Reconcile after the user command starts executing (task_attempt.py:570). The controller records started_at on the attempt.

SUCCEEDED

Reported by the worker via Reconcile when the task process exits with code 0. The controller sets exit_code=0, finished_at, and marks the task terminal. No retry logic applies.

FAILED

Reported by the worker via Reconcile when the task process exits with a non-zero code. Triggers retry evaluation:

1. handle_attempt_result calls _handle_failure, which increments failure_count and compares against max_retries_failure.
2. If failure_count <= max_retries_failure: returns SHOULD_RETRY. The caller (_on_task_state_changed) calls _requeue_task, which resets state to PENDING and re-enqueues the task. Resources are released from the current worker.
3. If failure_count > max_retries_failure: returns EXCEEDED_RETRY_LIMIT. The task remains in FAILED state and is terminal. error and exit_code are recorded.

KILLED

Set by the controller in three scenarios:

1. User cancellation: _on_job_cancelled iterates non-terminal tasks and transitions each to KILLED. Tasks with workers assigned are queued for kill RPCs.

2. Job finalization: _finalize_terminal_job terminates a job's remaining non-terminal tasks with the job's recorded terminal cause.

3. Parent job termination: Parent finalization or explicit cancellation recursively kills live descendant jobs. Descendant jobs and tasks use the error Parent job terminated; task events use the stable reason code ParentJobTerminated. The originating terminal cause remains on the parent.

KILLED is always terminal and never retried.

WORKER_FAILED

Set by the controller when a worker dies. Worker failure (ops.worker.fail) closes all tasks on the dead worker, emitting a TASK_STATE_WORKER_FAILED transition for each non-terminal task.

Retry evaluation uses the preemption budget:

1. _handle_failure increments preemption_count and compares against max_retries_preemption (default: 100).
2. If budget remains: SHOULD_RETRY -- task is requeued to PENDING.
3. If exhausted: EXCEEDED_RETRY_LIMIT -- task stays in WORKER_FAILED and is terminal.

Coscheduled jobs: When a task in a coscheduled (gang-scheduled) job fails terminally, _terminate_coscheduled_siblings transitions all running siblings to COSCHED_FAILED (always terminal — see below). This prevents other hosts from hanging on collective operations.

PREEMPTED

Set by the controller when a higher-priority task evicts a lower-priority running task. The preemption pass (apply_preemptions) selects victims from lower priority bands and submits them to apply_terminal_decisions_batch.

Also reported by the K8s backend when the cluster control plane disrupts an attempt's pod, marked by a DisruptionTarget or Kueue TerminationTarget pod condition (scheduler preemption, Kueue workload eviction, node drain, API eviction). Kueue deletes the pod it evicts, so the condition is readable only while the pod terminates; the backend caches it and reports it as the attempt's terminal_reason once the pod is gone. A pod that vanishes with no disruption ever observed is reported WORKER_FAILED instead — same preemption budget, without claiming a cause. These updates run through apply_one_transition and do cascade coscheduled siblings.

Retry evaluation uses _resolve_task_failure_state with the preemption budget:

1. ASSIGNED tasks: always retry to PENDING regardless of budget (the task never started executing, so preemption is free).
2. BUILDING or RUNNING tasks: preemption_count is incremented and compared against max_retries_preemption.
   • If preemption_count <= max_retries_preemption: task is requeued to PENDING for retry. The current attempt is marked PREEMPTED.
   • If preemption_count > max_retries_preemption: task state is set to PREEMPTED (terminal). Both the attempt and the task are PREEMPTED.

PREEMPTED is in both TERMINAL_TASK_STATES and FAILURE_TASK_STATES. When a coscheduled task becomes terminally PREEMPTED, the job state is recomputed. If all tasks in the job are terminal, the batches.py _finalize_terminal_job orchestrator kills any remaining non-terminal tasks and cascades to child jobs. Note that unlike WORKER_FAILED reported via Reconcile, PREEMPT decisions do not directly cascade coscheduled siblings — the cascade only occurs through job finalization.

UNSCHEDULABLE

Set by the controller's scheduling loop when a task's scheduling deadline expires (via apply_terminal_decisions_batch). The deadline is derived from the job's scheduling_timeout field.

UNSCHEDULABLE is always terminal. If any task becomes unschedulable, the entire job transitions to JOB_STATE_UNSCHEDULABLE and all remaining tasks are killed.

COSCHED_FAILED

Set by the controller in _terminate_coscheduled_siblings when a coscheduled (gang-scheduled) sibling task hits its terminal failure budget. The cascaded task itself was healthy, so the state is distinct from WORKER_FAILED (which implies a real worker death and bumps the preemption counter) and from KILLED (which implies operator-initiated cancellation).

COSCHED_FAILED is always terminal — task_is_finished returns True unconditionally, with no counter math involved. At the job level, it rolls up into JOB_STATE_WORKER_FAILED alongside WORKER_FAILED and PREEMPTED, because the triggering event was a worker-failure pattern on the originating task.

Retry Semantics

Iris maintains two independent retry budgets per task:

Budget	Counter	Limit Field	Default	Trigger States
Failure	failure_count	max_retries_failure	0 (no retries)	FAILED
Preemption	preemption_count	max_retries_preemption	100	WORKER_FAILED, PREEMPTED

Retry flow

1. Worker reports terminal state via Reconcile.
2. handle_attempt_result delegates to _handle_failure.
3. The appropriate counter is incremented.
4. If counter <= limit: TaskTransitionResult.SHOULD_RETRY.
   • _on_task_state_changed calls _requeue_task.
   • Task state is reset to PENDING. A new attempt will be created when the scheduler re-dispatches.
   • Worker resources are released via _cleanup_task_resources.
5. If counter > limit: TaskTransitionResult.EXCEEDED_RETRY_LIMIT.
   • Task remains in its failure state and is terminal.
   • is_finished() returns True.
   • The job's _compute_job_state may trigger a job-level state change (e.g., JOB_STATE_FAILED if max_task_failures is exceeded).

What counts toward job failure

Only TASK_STATE_FAILED counts toward the job's max_task_failures threshold. Worker failures and preemptions do not count. This means a job can survive unlimited preemptions as long as the per-task preemption budget is not exhausted. TASK_STATE_PREEMPTED and TASK_STATE_WORKER_FAILED are grouped together for job state derivation: if all tasks are terminal and any are in one of these states, the job becomes JOB_STATE_WORKER_FAILED.

States that are never retried

• SUCCEEDED: task completed successfully
• KILLED: explicit termination by user or cascade
• UNSCHEDULABLE: scheduling timeout expired
• PREEMPTED: only when preemption budget is exhausted (otherwise retried as PENDING)

Terminal State Summary

A task is considered finished (is_finished() == True) when:

State	Condition
SUCCEEDED	Always finished
KILLED	Always finished
UNSCHEDULABLE	Always finished
FAILED	Finished when failure_count > max_retries_failure
WORKER_FAILED	Finished when preemption_count > max_retries_preemption
PREEMPTED	Finished when preemption_count > max_retries_preemption

The distinction matters: a task in FAILED state with retry budget remaining is in a terminal state at the attempt level but is not finished at the task level. can_be_scheduled() returns True for such tasks.

Dashboard Display

The dashboard uses stateToName() from shared/utils.js to convert proto enum strings (e.g., TASK_STATE_RUNNING) to lowercase display names by stripping the TASK_STATE_ prefix. Each name maps to a CSS class status-{name}:

Display Name	CSS Class	Color
pending	.status-pending	Amber (#9a6700)
assigned	.status-assigned	Orange (#bc4c00)
building	.status-building	Purple (#8250df)
running	.status-running	Blue (#0969da)
succeeded	.status-succeeded	Green (#1a7f37)
failed	.status-failed	Red (#cf222e)
killed	.status-killed	Grey (#57606a)
worker_failed	.status-worker_failed	Purple (#8250df)
unschedulable	.status-unschedulable	Red (#cf222e)
preempted	.status-preempted	Orange (#bc4c00)

The job detail page shows per-task attempt history. Each attempt has its own state badge, and worker failures are annotated with "(worker failure)" in the attempt rows.

Pending tasks display a pending_reason diagnostic below the state badge when the controller can identify why the task cannot be scheduled (e.g., no workers match constraints).

Job State Derivation

Job state is computed from task state counts in _compute_job_state():

1. SUCCEEDED: All tasks are in TASK_STATE_SUCCEEDED.
2. FAILED: Count of TASK_STATE_FAILED tasks exceeds max_task_failures.
3. UNSCHEDULABLE: Any task is TASK_STATE_UNSCHEDULABLE.
4. KILLED: Any task is TASK_STATE_KILLED (and job is not already terminal).
5. RUNNING: Any task is ASSIGNED, BUILDING, or RUNNING.
6. PENDING: Default (no tasks have started).

The ordering matters -- earlier rules take priority. A job with one succeeded task and one failed task (beyond tolerance) is FAILED, not RUNNING.