Proposal: Custom Status / Progress Reporting

Problem Statement

Orchestrations today are opaque while running — clients can only see Running or a terminal state (Completed/Failed). There is no way for an orchestration to report intermediate progress. Users of the durable-copilot-sdk resorted to abusing ActivityCompleted events as a side channel for intermediate content, which is fragile and semantically wrong.

Design

Orchestration API

// Set the custom status (last write per turn wins)
ctx.set_custom_status("Processing item 3 of 10");

// Structured data via JSON
ctx.set_custom_status(serde_json::to_string(&MyProgress { step: 3, total: 10 }).unwrap());

// Clear it
ctx.set_custom_status("");

set_custom_status() is not a history event or an action. It's pure metadata — a field on CtxInner that gets plumbed into ExecutionMetadata at ack time. No impact on determinism, no replay implications.

• Imperative: call it whenever, as many times as you want within a turn
• Last write wins: if called twice in the same turn, only the last value is sent
• Persistent across turns: if you don't call it on a later turn, the provider keeps the previous value
• Not reset on CAN: carries forward as a provider-level column (not in history)

Provider Changes

ExecutionMetadata — new field:

pub struct ExecutionMetadata {
    // ... existing fields ...
    /// User-defined status string for progress reporting.
    /// Updated on every ack when set. Provider preserves the last value
    /// across turns if not set (None = no update).
    pub custom_status: Option<String>,
}

Schema migration — new columns on executions table:

ALTER TABLE executions ADD COLUMN custom_status TEXT;
ALTER TABLE executions ADD COLUMN custom_status_version INTEGER NOT NULL DEFAULT 0;

Provider contract:

• ack_orchestration_item(): if metadata.custom_status is Some(s), write s to the custom_status column and increment custom_status_version. If None, don't touch either column (preserve previous value).
• Reading: return custom_status and custom_status_version alongside status/output when building OrchestrationStatus.

Client API

OrchestrationStatus — custom_status and custom_status_version on all non-NotFound variants:

pub enum OrchestrationStatus {
    /// Instance does not exist
    NotFound,
    /// Instance is currently executing
    Running { custom_status: Option<String>, custom_status_version: u64 },
    /// Instance completed successfully with output
    Completed { output: String, custom_status: Option<String>, custom_status_version: u64 },
    /// Instance failed with structured error details
    Failed { details: ErrorDetails, custom_status: Option<String>, custom_status_version: u64 },
}

Rationale for including on Completed/Failed:

• The last progress update set during execution is useful context even after termination
• Completed: "Processed 10/10 items" — tells you the final progress state alongside the output
• Failed: "Processing item 7/10" — tells you where it failed, complementing the error details
• Without it, the custom status is lost the moment you read the terminal status — you'd have to have been polling at exactly the right moment

Breaking change: Running changes from a unit variant to a struct variant. All existing Running => matches need to become Running { .. } =>. This is a semver-minor bump, and the fix is mechanical.

get_orchestration_status() — reads custom_status from the provider alongside status/output:

match client.get_orchestration_status("order-123").await? {
    OrchestrationStatus::Running { custom_status } => {
        if let Some(status) = custom_status {
            println!("Progress: {status}");
        }
    }
    OrchestrationStatus::Completed { output, custom_status } => {
        println!("Done: {output}");
        if let Some(status) = custom_status {
            println!("Last progress: {status}");
        }
    }
    OrchestrationStatus::Failed { details, custom_status } => {
        eprintln!("Failed: {}", details.display_message());
        if let Some(status) = custom_status {
            eprintln!("Failed at: {status}");
        }
    }
    _ => {}
}

Flow

1. Orchestration calls ctx.set_custom_status("step 3/10")
2. Value stored in CtxInner.custom_status
3. Turn ends → dispatcher reads custom_status from context
4. Plumbed into ExecutionMetadata { custom_status: Some("step 3/10"), .. }
5. ack_orchestration_item() writes it to provider (UPDATE executions SET custom_status = ?)
6. Client calls get_orchestration_status() → provider reads custom_status column → returned on the enum variant

What This Is NOT

• Not a history event — no EventKind::CustomStatusSet. It's metadata, not part of the replay-visible history. This avoids determinism concerns and history growth.
• Not activity progress — activities cannot set custom status. That's a separate feature (Part B: Activity → Orchestration progress reporting) with different execution model challenges.
• Not a full streaming API — wait_for_status_change() provides efficient change notification via long polling with version tracking. Full event streaming (e.g., client.watch_execution()) is a separate, larger feature.

Change Notification API

client.wait_for_status_change() — blocks until the custom status changes or the orchestration reaches a terminal state:

/// Wait until the custom_status_version exceeds `last_seen_version`,
/// the orchestration reaches a terminal state (Completed/Failed),
/// or the timeout is hit.
///
/// Returns the current OrchestrationStatus with the new custom_status and version.
client.wait_for_status_change(
    instance: &str,
    last_seen_version: u64,    // pass 0 on first call to get current state immediately
    poll_interval: Duration,   // how often to check (e.g., 100ms for dashboards, 1s for background)
    timeout: Duration,         // max wait time (Duration::MAX for indefinite)
) -> Result<OrchestrationStatus, ClientError>

Semantics:

• If custom_status_version > last_seen_version, returns immediately with current state
• If the orchestration is already in a terminal state (Completed/Failed), returns immediately
• Otherwise, polls the provider at fixed poll_interval until version changes, terminal state, or timeout
• Timeout returns ClientError::Timeout (not an orchestration error)
• No backoff — this is for live dashboards and interactive UIs where consistent latency matters

Version semantics:

• Version starts at 0 (no custom status ever set)
• Incremented by 1 on every set_custom_status() call that's acked (once per turn, only if set)
• Monotonically increasing within an execution
• Reset to 0 on CAN (new execution)
• Handles the edge case where status is set to the same value twice — version increments, so the client is notified even though the string didn't change

Usage pattern:

let mut last_version = 0;

loop {
    match client.wait_for_status_change("order-123", last_version, Duration::from_millis(200), Duration::from_secs(30)).await? {
        OrchestrationStatus::Running { custom_status, custom_status_version } => {
            last_version = custom_status_version;
            if let Some(status) = custom_status {
                println!("Progress: {status}");
            }
        }
        OrchestrationStatus::Completed { output, .. } => {
            println!("Done: {output}");
            break;
        }
        OrchestrationStatus::Failed { details, custom_status, .. } => {
            eprintln!("Failed: {}", details.display_message());
            if let Some(status) = custom_status {
                eprintln!("Failed at: {status}");
            }
            break;
        }
        _ => break,
    }
}

Implementation:

• wait_for_status_change() lives in the Client
• The client internally calls Provider::get_custom_status(instance, last_seen_version) in a loop at poll_interval
• get_custom_status() returns Some((custom_status, version)) if version > last_seen_version, None otherwise
• When Some is returned, the client does one final get_orchestration_status() to return the full OrchestrationStatus
• Also returns on terminal state or timeout

Provider trait — new method:

/// Lightweight check for custom status changes.
/// Returns Some((custom_status, version)) if version > last_seen_version,
/// None if unchanged.
/// SQL: SELECT custom_status, custom_status_version
///      FROM executions WHERE instance_id = ?
///      AND custom_status_version > ?
async fn get_custom_status(
    &self,
    instance: &str,
    last_seen_version: u64,
) -> Result<Option<(Option<String>, u64)>, ProviderError>;

Flow

1. Orchestration calls ctx.set_custom_status("step 3/10")
2. Value stored in CtxInner.custom_status
3. Turn ends → dispatcher reads custom_status from context
4. Plumbed into ExecutionMetadata { custom_status: Some("step 3/10"), .. }
5. ack_orchestration_item() writes to provider: UPDATE executions SET custom_status = ?, custom_status_version = custom_status_version + 1
6. Client calls wait_for_status_change("order-123", 2, 200ms, 30s) → polls get_custom_status("order-123", 2) every 200ms → returns Some when version > 2 → client calls get_orchestration_status() for full status

Implementation Plan

1. Schema migration — add custom_status TEXT and custom_status_version INTEGER NOT NULL DEFAULT 0 columns to executions
2. ExecutionMetadata — add custom_status: Option<String> field
3. Provider trait — add get_custom_status(instance, last_seen_version) method
4. SQLite provider — write custom_status and increment custom_status_version in ack_orchestration_item(), implement get_custom_status(), read both in status queries
5. OrchestrationStatus — change Running to struct variant, add custom_status and custom_status_version to all non-NotFound variants
6. CtxInner — add custom_status: Option<String> field
7. OrchestrationContext — add set_custom_status() method
8. Orchestration dispatcher — read custom_status from context after turn, include in metadata
9. compute_execution_metadata() — plumb custom_status from context into metadata
10. Client::get_orchestration_status() — read custom_status and custom_status_version from provider response
11. Client::wait_for_status_change() — client-side polling loop using get_custom_status(), final get_orchestration_status() on change
12. Fix all Running => match arms across codebase
13. Provider validation tests — verify custom_status round-trips through ack/read

Test Plan

#	Test	Description
1	custom_status_set_and_read	Orchestration sets custom status, client reads it while running.
2	custom_status_persists_across_turns	Set status in turn 1, don't set in turn 2, verify it's still readable.
3	custom_status_last_write_wins	Set status twice in one turn, verify only the last value is persisted.
4	custom_status_on_completed	Set status, then complete. Verify Completed { custom_status } has the value.
5	custom_status_on_failed	Set status, then fail. Verify Failed { custom_status } has the value.
6	custom_status_with_json	Set structured JSON as custom status, read and deserialize on client.
7	custom_status_none_by_default	New orchestration has custom_status: None, custom_status_version: 0.
8	wait_for_status_change_returns_on_update	Client calls wait_for_status_change with version 0, orchestration sets status → client unblocks with new status and version 1.
9	wait_for_status_change_returns_immediately_if_ahead	Status already at version 3, client calls with last_seen_version: 1 → returns immediately.
10	wait_for_status_change_returns_on_terminal	Client waiting, orchestration completes → client unblocks with Completed status.
11	wait_for_status_change_timeout	No status change within timeout → returns ClientError::Timeout.
12	wait_for_status_change_same_value_increments_version	Set status to "A", then "A" again on next turn → version increments from 1 to 2, client is notified despite same string.
13	custom_status_version_resets_on_can	Set status, CAN, verify version resets to 0 in new execution.