Lifecycle and Cleanup

Durable provisioning ownership

Eligible new Azure Windows creates can be admitted by the versioned durable controller described in Coordinator. Their private operation owns all candidate resource identities until terminal publication or verified cleanup. Legacy interrupted-create recovery, expiry, deferred cleanup and orphan mutation must respect this ownership. Transactional legacy mutation fences prevent a new admission from overtaking cleanup that has already claimed its provider work. An unsuccessful legacy mutation retains its fence and existing cleanup evidence; a new job is never proof that old cleanup debt disappeared.

Cancellation, including ordinary token cancellation with keep=true, stops forward provisioning and fallback. Already dispatched work must settle before the controller deletes verified resources. The journal retains immutable VM, NIC, public-IP and disk identities while deletion progress shrinks separately. An unexplained missing resource, identity replacement or unresolved dispatch remains cleanup debt. In particular, a network PUT with a lost acknowledgement followed by 404 is not permission to repeat allocation or move to another candidate. Completed machines retain their observed identity for the existing owned-deletion path.

Explicit release with { "delete": false } retains a provisioning resource and its private identity records. It stops forward provisioning and active publication, observes any dispatched work to settlement, and then removes the operation from the runnable queue. A later explicit release with delete: true resumes exact owned cleanup without decrypting the retired VM password/bootstrap. Ordinary token cancellation still requires cleanup, including when the original request used keep=true. A retention request cannot undo a deletion request that already owns cleanup.

Healthy queued cleanup reports cleanupStatus: pending through release and GET. Verified final cleanup writes cleanupCompletedAt, clears transient metadata and remote access fields, and reports complete; identity conflicts, missing successful-deletion proof and observed errors report failed. Retention reports retained. These states preserve the existing client rule to keep local credentials until cleanup is verified complete.

Durable Azure cleanup uses the same owned-delete claims, immutable identity and attachment validation as ordinary release. Each bounded tick removes at most one resource or polls one acknowledged deletion operation. Intent alone, a lost DELETE response, or a failed successful-progress write never authorizes removal of the remaining members. An accepted asynchronous DELETE is recorded separately from successful progress; the operation must report success before that progress can be persisted. Retryable in-use errors trigger fresh ownership checks on the next tick. Completed deletion claims remain as exact completion evidence so a lost provisioning-result commit cannot erase cleanup proof.

Nonsecret plans, attempt histories and completed deletion claims currently have the same manual retention lifecycle as lease history; there is no automatic history-pruning job. Understood operations retire sealed replay material in the same transaction that durably disables forward work through cancellation, retention or expiry, even when settlement or cleanup remains unresolved. Pending forward operations still require valid bound material; unsupported/quarantined records are not blindly purged. Successful publication also purges material. A future history pruner must preserve unresolved ownership and consume completed claims together with their operation history, never as orphan cleanup.

Read this when:

• changing how leases are released or expired;
• debugging leaked provider resources (instances, NICs, public IPs, disks, Mac hosts);
• changing direct-provider cleanup behavior.

A lease holds a remote box until it is released or expires. Two independent paths reclaim the underlying resources: the brokered path, owned by the coordinator, and the direct path, owned by the local CLI (and, for GCP, a guest-side guard). Which one applies depends on whether the provider runs through a coordinator.

Brokered lifecycle

When a provider is brokered (only aws, azure, daytona, gcp, and hetzner, and only when a coordinator URL is configured), the coordinator owns the lease record and its lifecycle. The coordinator persists a provisioning reservation before calling the provider (worker/src/types.ts):

provisioning -> active    (readiness completed)
provisioning -> failed    (creation or readiness failed)
active       -> released  (explicit release)
active       -> expired   (TTL or idle cleanup completed)

Release can also end a provisioning lease. It records user intent; it does not cancel an already-dispatched provider request. Failed and released records can still carry unresolved cleanup responsibility. Their local state alone is not proof that the provider resource was deleted.

Interrupted provisioning recovery describes an observed interruption, without attributing it to a deployment. It can reconcile settled calls in the same coordinator generation as well as attempts observed after reconstruction; the existing generation and settlement eligibility checks still apply.

AWS recovery inventory is evidence only for the queried Region and existing tag/state filters. Both the coordinator's instance inventory and private workspace lookup read every DescribeInstances page, retaining duplicate matches for ambiguity checks. A repeated pagination token, more than 100 pages, or a later-page failure makes the inventory incomplete and rejects the entire lookup. Incomplete inventory retains cleanup debt and cannot confirm absence, even after the existing 30-minute absence confirmation window. A complete empty inventory remains subject to that window and the existing ownership checks.

For an exact Azure lease whose provisioning stops before VM creation, ordinary owned-resource release can clean the observed creation prefix: an unattached canonical public IP alone, or the exact canonical public IP and NIC together. A fresh NIC without its public IP is not a valid creation prefix and remains report-only; cleanup can resume past a missing public IP only when its exact durable claim already proves Crabbox deleted that public IP. A managed disk without its complete NIC/public-IP set, a managed-disk VM without that disk, foreign attachments, replacements, and missing immutable identities also fail closed. This narrow exact-lease release path does not relax the separate Azure orphan sweep's complete-set and quarantine requirements.

Heartbeats and expiry

While a command runs, the CLI heartbeats the active lease over the coordinator control WebSocket, falling back to POST /v1/leases/{id}/heartbeat within the same 20-second request budget. An exhausted control request keeps its original failure diagnosis instead of attempting HTTP with an expired context; when both transports fail, the warning retains both causes. A heartbeat bumps lastTouchedAt, recomputes expiresAt, and clears stale cleanup metadata; the HTTP path also refreshes provider SSH access where supported. Heartbeats at or after expiresAt are rejected so they cannot revive a lease once expiry cleanup owns it.

Expiry is the minimum of two clocks (leaseExpiresAt in worker/src/fleet.ts):

• idle expiry — lastTouchedAt + idleTimeout (default idle timeout 1800s);
• max lifetime — createdAt + ttl (default TTL 5400s, capped at 86400s).

A heartbeat can only push idle expiry forward up to the max-lifetime cap, so a busy lease still expires at its TTL regardless of activity.

Release vs expiry

Both release and expiry call the same provider delete path:

• Release (POST /v1/leases/{id}/release, e.g. crabbox stop) records state released and queues provider deletion when the lease is still active. The body defaults delete to !keep; normal release does not synchronously await provider cleanup.
• Expiry is driven by the runtime scheduler. expireLeases deletes the cloud server for every active lease past expiresAt, then sets state expired.

Release and expiry publish provider-cleanup completion only after the provider delete path succeeds and the exact cleanup claim is revalidated. The stored record then clears host, tailscale, sshHostKey, and providerAccessExpiresAt. It retains provider resource IDs, server identity, ownership labels, provider scope, network, exposed ports, and workRoot for orphan recovery, audit, and AWS ingress reconciliation. When provisioning has not dispatched, the reservation owner persists explicit no-resource evidence before publishing the same completion fact. Missing cloud identity without that evidence is a legacy or unresolved state and remains unconfirmed.

keep=true only suppresses the automatic release when a run command exits; it does not exempt a lease from idle or TTL expiry.

After one release mutation is accepted, an explicit CLI stop observes the lease with read-only requests until provider deletion is final or a bounded wait ends. The public cleanupStatus distinguishes normal pending creation or cleanup from an observed failure. Pending work uses that same observation loop and five-minute bound; the CLI does not send another release mutation after acceptance. Existing cleanup diagnostics remain intact for clients that predate this classification. New AWS instance IDs can be temporarily invisible. Cleanup observes visibility within the existing bound before verifying allocation ownership; unconfirmed visibility or termination retains cleanup debt rather than reporting deletion. Managed public AWS release uses the same confirmed termination path as private workspaces: TerminateInstances must acknowledge the exact instance, followed by a terminal terminated read or exact InvalidInstanceID.NotFound. Allocation claims carry the prepared account scope for AWS Mac instances. Storage failures while publishing or checking an allocation preserve its cleanup claim without retrying creation. The CLI removes its local per-lease SSH connection directory only after final cleanup state, cleanupCompletedAt, and hostless public access are observed. Pending or retrying cleanup, observation timeout or cancellation, provider errors, ownership mismatches, and retained resources keep the local claim and credentials available for a safe retry. Acquisition rollback and automatic post-run release only queue cleanup and do not wait for provider deletion; they preserve local state while cleanup is pending. Local cleanup is scoped to <user-config>/crabbox/testboxes/<lease-id> and its private short SSH socket namespace; it never follows configured or shared SSH key paths. Cleanup closes and joins the lease's persistent OpenSSH masters before deleting local artifacts. Local failure preserves the confirmed remote outcome and local claim; a fresh confirmed-deletion lookup retries only local cleanup, not provider deletion.

For public AWS leases, shared security-group writes remain serialized with release state changes and authoritative ingress reconciliation. Image selection, instance creation, network readiness and provider deletion do not hold that fence, so a slow instance or termination does not block another lease's ingress work. Cleanup still waits for its provider result, then reacquires the ingress fence and revalidates its exact claim before publishing success or failure. Every regional ingress attempt rechecks the creating lease and derives access from current lease records before writing.

If a create attempt is canceled or its lease is released during AWS region preparation, the in-flight create returns 409 create_canceled with the reason. It does not continue into another region or report an empty provider error. Cancellation does not confirm deletion; inspect the lease's cleanup status separately.

Cleanup retries

If deleting the cloud server during expiry fails, the lease stays active and the coordinator records cleanupAttempts, cleanupError, cleanupFailedAt, and a cleanupRetryAt set 5 minutes out (leaseCleanupRetryDelayMs). The next alarm is scheduled for the soonest of all active-lease expiry/retry times, so a failed delete is retried automatically. On success the cleanup metadata is cleared and the state becomes expired. You can inspect stuck cleanups with crabbox admin lease-audit.

Brokered native checkpoint retention

New brokered native AWS, Azure, and GCP checkpoints have their own durable lifecycle, independent of the source lease. Retention is manual unless the owner explicitly requests checkpoint create --expire-unused-after <duration> or changes checkpoint policy. Coordinator alarms read a bounded, sorted due index and expire only those explicitly opted-in records; generic provider inventory, old image records, local files, and checkpoint-like names never grant cleanup ownership.

Active renewable fork/shard use claims and AWS/Azure promotion pins prevent deletion. Manual and automatic deletion share the same generation-fenced provider cleanup: AWS confirms the exact AMI and every owned EBS snapshot are gone, Azure confirms its exact canonical managed snapshot, and GCP confirms its exact project-scoped machine image or disk snapshot. Ambiguous or failed provider calls retain ownership and retry with capped backoff; a durable provider-deleted phase makes final metadata cleanup restart-safe. Checkpoint admission charges creating, ready, failed, and deletion-pending records until deletion is confirmed; deleted audit tombstones no longer consume capacity. Expired available fork claims can be replaced, but provisioning claims remain charged until exact lifecycle reconciliation. Each checkpoint retains only its 256 most recent ordered audit events, and eventual tombstone pruning removes that entire retained suffix. Direct, archive, recipe, and historical checkpoints remain operator-managed.

Brokered Hetzner cleanup confirmation

The coordinator records version-1 providerCleanup evidence on the existing lease before dispatching an exact-server DELETE. It saves the validated delete action ID before waiting on /servers/actions/{id}. Completion requires action success followed by exact /servers/{id} GET absence. Hetzner can remove the server from account visibility while deletion is still running; GET 404 alone does not complete a known action. Observation is bounded to 60 seconds per attempt, with each request and body read bounded to 10 seconds or the remaining observation budget.

An exact DELETE rejection with HTTP 400, 401, 403, 405, 409, 410, 412, 423, or 429 retires only that rejected attempt's dispatch marker under the cleanup owner fence. These statuses describe rejected requests in Hetzner's error contract. The rejection remains a cleanup failure and uses the existing broker backoff. The next attempt freshly reads the exact server and revalidates ownership, then durably records a new dispatch before DELETE. A failed rejection write retains uncertainty. HTTP 408, 422 (which also covers service errors), 5xx, timeouts, malformed actions, and lost acknowledgements never authorize retiring that marker.

An exact server already missing before any recorded dispatch has the distinct already-absent basis. DELETE 404 requires a subsequent exact GET 404 and has its own basis. Neither can replace an unresolved action or lost acknowledgement. Action errors, malformed or missing actions, uncertain dispatches, conflicting identity or ownership, and exhausted waits retain cleanup debt. An unknown numeric server identity is unresolved unless the recorded failure proves that only a newly created owned SSH key needs cleanup; an empty inventory is not deletion proof. Modern ownership requires canonical lease, provider, slug, and owner labels. The supported pre-slug contract still requires the canonical server name and legacy labels, and rejects an explicitly conflicting owner label.

Evidence writes and final publication are fenced to the same cleanup claim, lease incarnation, and resource identity. Retries resume recorded actions without another DELETE. For provisioned servers, owned SSH-key cleanup follows durable server confirmation; key failures retain that evidence for retry, and shared keys remain retained. Definite key-only failure records retain provisioningResourceMayExist=false and the exact owned-created key ID, with no server identity, server cleanup journal, or outstanding provisioning request. Cleanup rechecks the current claim before deleting that key and creates no server-absence receipt. Explicit no-delete retention preserves this evidence for later deletion. Historical records that lost the definite no-resource flag require resolution; absence of request markers alone cannot authorize the key-only path. Ordinary authenticated GET and crabbox inspect --json expose the nonsecret journal. Its confirmation timestamp records the provider API contract observed then, not physical hardware inspection or a fresh probe. cleanupStatus still governs finality. Historical released leases without evidence are not passively backfilled.

For recovery, the ordinary owner should inspect the exact lease with crabbox inspect --id <lease-id> --json and retain local credentials and recorded evidence. Historical host fields, released state, or missing flags do not prove absence. An explicit stop of a historical managed lease with stored provider identity re-enters the same exact-owner cleanup path and establishes cleanupCompletedAt only after provider deletion succeeds. A recorded action with a known ID resumes through the same lease owner and broker cleanup path. Missing no-resource evidence or dispatch acknowledgement authority requires operator reconciliation of the exact resource in its original provider project. Never manually clear cleanup flags or fabricate a receipt. There is no supported automatic population sweep.

Managed Daytona cleanup

New brokered Daytona sandboxes receive native wall-clock TTL in the original allocation request: the requested lease TTL rounded up to whole minutes, with a one-minute minimum. This clock starts at provider creation, not coordinator reservation, and applies even when the sandbox is kept or explicitly retained. Heartbeats do not extend it. A TTL-capable Daytona API is required; Crabbox never retries allocation without TTL. Before publishing access, a ready sandbox must report a future native destruction deadline no later than one requested TTL from the start of readiness observation. This bound does not move during polling or assume that the provider's creation and TTL timestamps share an exact clock tick. An API that silently ignores TTL is not accepted as ready. Organization, region, and sandbox-class lifespan limits still apply. This is a provider lifetime contract, not a teardown-latency SLA during provider failures.

Before dispatch, the lease stores a nonsecret fingerprint of the normalized API URL, configured organization, and credential context. A returned sandbox UUID is recorded before readiness waits. Cleanup uses that exact UUID and original context, verifies current ownership before mutation, and observes a terminal provider state or exact-resource absence before reporting completion. Accepted DELETE requests alone are not completion.

If no authoritative UUID arrives, cleanup remains explicitly unresolved. The coordinator preserves its original dispatch evidence, failureError, and cleanupError; it neither adopts name/label matches nor declares deletion after an empty inventory read or elapsed TTL. It does not repeatedly schedule a lookup that cannot establish identity. Inspect these records with crabbox admin lease-audit and resolve the resource in its original provider account. A release acknowledgment does not clear this responsibility.

Missing historical scope or a changed API URL, organization, or API key blocks automatic cleanup and access refresh rather than adopting the current context. This deliberately includes same-organization key rotation: the fingerprint proves identical credential context, not upstream account continuity. Finish known-ID cleanup before rotating credentials, or restore the original context and repeat explicit stop. Legacy records without scope require operator resolution; never backfill them from current configuration.

AWS orphan sweep

Independent of per-lease expiry, the Worker can report AWS resources that no longer map to an active lease. Delete mode terminates instances or releases idle Mac dedicated hosts only when retained coordinator state binds the exact resource; tag-only and legacy candidates stay report-only. It runs from the same alarm/cron, gated by CRABBOX_AWS_ORPHAN_SWEEP_* environment variables.

Azure orphan sweep

The coordinator's Azure sweep uses a reconciliation-specific inventory of the configured resource group across canonical per-lease VMs, NICs, public IPs, and managed OS disks. The ordinary VM list/pool path remains VM-only. The sweep can group a complete exact-owned NIC, public IP, and managed-disk set even when an interrupted provisioning attempt left no VM.

An exact group must retain consistent Crabbox ownership tags, location, canonical topology, and stable Azure resource identities. Mixed or incomplete groups remain report-only. VMs with any data disk are also report-only because VM deletion can cascade through data-disk delete options. VM managed-disk, VM-to-NIC, and NIC-to-public-IP cascade-delete options are likewise rejected and fingerprinted. A change to the member set, ownership labels, location, topology, or stable identity changes the reconciliation fingerprint and resets the grace quarantine. Ordinary release still supports a verified VM/NIC/public-IP set whose OS disk is explicitly ephemeral (diffDiskSettings.option=Local); the managed-disk orphan sweep keeps that non-managed shape report-only.

Delete mode releases only a group bound to an exact retained coordinator lease after the same complete resource identity has survived the grace period and two successful authoritative inventories. The normal owned-resource delete path performs a fresh preflight before each deletion. Shared VNets, subnets, NSGs, and resource groups are not part of the inventory or deletion path. With all four Azure broker credentials configured, the sweep is enabled unless CRABBOX_AZURE_ORPHAN_SWEEP_ENABLED=0; set CRABBOX_AZURE_ORPHAN_SWEEP_DELETE=1 to allow deletion. Its interval and grace default to 3600 and 900 seconds and are controlled by CRABBOX_AZURE_ORPHAN_SWEEP_INTERVAL_SECONDS and CRABBOX_AZURE_ORPHAN_SWEEP_GRACE_SECONDS.

Durable owned-delete claims persist each successful member deletion with that member's stable resource identity before cleanup advances. A missing member is accepted on retry only when the same claim contains that exact ordered progress; an external disappearance or failed progress write stops the remaining cleanup. Progress updates transactionally merge the longest verified prefix so concurrent cleanup attempts cannot overwrite newer evidence. Fresh preflight also compares the quarantined ownership labels, including keep and expiry, on every survivor. New cleanup starts with a transactional version-2 preparing reservation, then binds the complete inspected identity before the first deletion. Every later claim transition and completed-claim clear is transactional, so a delayed preparation, replacement, or empty-set observer cannot overwrite or erase newer progress. Older workers reject version-2 claims rather than replaying them with version-1 semantics. Legacy version-1 claims still replay through the same scope and topology checks, but a legacy partial claim with an unexplained missing member fails closed because it cannot prove which cleanup deleted that member. A legacy claim can establish a new stable baseline only while the VM, NIC, public IP, and managed disk are all still present; an already-empty legacy claim can be cleared without mutation.

Automatic cleanup does not relax that rule. For the specific expired, disk-only case with recorded VM/NIC deletion and an absent public IP, an owner or admin may use audited cleanup recovery to explicitly accept original-scope public-IP absence. The resulting version-3 claim retains its original baseline and actual DELETE receipts, records the operator acknowledgement separately, and requires the exact original owned disk to remain detached. Older workers reject this claim version. Normal release rechecks survivors before deleting the disk; it does not infer a historical public-IP DELETE receipt, and its separate audit survives claim cleanup.

Azure polling distinguishes Azure-AsyncOperation status documents from Location completion responses. Location HTTP 202 remains pending; terminal 200/204 completes only without an explicit pending/failure state. An empty Azure-AsyncOperation response never establishes success.

Direct-provider lifecycle

Without a coordinator, the CLI talks to the provider API directly and owns cleanup itself. Releasing a direct lease (crabbox stop / crabbox release) deletes the backing machine immediately.

Ordinary direct AWS, Azure, GCP, and Hetzner acquisition can retry a bootstrap timeout with a fresh lease. If rollback reports a cleanup failure, acquisition stops instead: the original failure and cleanup diagnostics remain available, and no second allocation is attempted. This consumes each adapter's existing cleanup result; it does not give all providers the same deletion-confirmation guarantee. AWS termination acceptance and Hetzner DELETE success still differ from GCP operation completion and Azure's dependent-resource absence checks. GCP also stops retrying if its selected-scope cleanup client cannot be rebuilt, even if the existing best-effort fallback client returns success. That fallback does not establish that the selected project and zone were cleaned.

RunPod and Hetzner bind readiness to the immutable resource ID and expected name returned or requested at creation, before using the SSH endpoint. Hetzner also checks the new allocation's lease, provider ownership, slug, and selected-key labels. A contradictory readiness response cannot replace the original rollback target or its key labels. If the creation response itself cannot be bound to the request, Hetzner refuses to delete the returned server and reports the unresolved allocation; only an independently created attempt key can be cleaned up. These checks do not establish generation-fenced deletion for providers addressed by reusable names. Existing provider-specific failed-acquisition keep and cleanup policies remain unchanged.

crabbox cleanup (alias crabbox machine cleanup) sweeps expired direct-provider machines and stale local state. It refuses to run when a coordinator is configured, because sweeping provider resources can race live brokered leases:

crabbox cleanup --provider hetzner --dry-run
crabbox cleanup --provider hetzner

Use --dry-run to print what would be deleted without touching anything. The sweep is conservative; for each candidate machine shouldCleanupServer (internal/cli/pool.go) decides from the machine's Crabbox labels:

• skip machines with no labels, or labeled keep=true;
• running / provisioning: delete only when stale — past expires_at plus a 12-hour safety window;
• leased / ready / active: delete once past expires_at;
• failed / released / expired: delete;
• otherwise: delete once past expires_at, skip if expires_at is missing or still in the future.

For this to work, every direct-provider machine must carry Crabbox labels/tags (at least crabbox, state, and expires_at) so the sweep can identify owned resources without touching unrelated infrastructure.

GCP guest-side expiry guard

A direct GCP lease can outlive the local CLI that created it — if cleanup never runs, the VM would leak. To guard against this, direct GCP leases install a self-deleting guard (cloudInitGCPExpiryGuardFiles in internal/cli/bootstrap.go): a systemd timer runs every 2 minutes, reads the instance's own labels via the GCP metadata server, and deletes the instance when it is clearly expired. It applies the same conservative logic as the CLI sweep:

• exits unless crabbox=true and keep != true;
• failed / released / expired: delete;
• running / provisioning: delete only past expires_at plus 12 hours;
• leased / ready / active (and unlabeled state): delete once past expires_at.

So an expired GCP box can reclaim itself even if the operator's machine is gone.

Claims and --reclaim

Independent of provider cleanup, the CLI keeps a local claim file per lease so repo-local wrappers do not need their own ledger. Commands that reuse a lease validate that the current repo matches the claim; deleting a lease removes its claim. Move a claim to a different repo deliberately with --reclaim. See Identifiers for the claim file format and location.

Providers may durably publish an exact-resource claim with the generic state=provisioning label before post-create readiness completes. Such a claim is recovery authority, not proof that the lease is ready: inventory and status must keep it non-ready until a fenced claim update records state=ready. Provider adapters own the immutable resource identity and routing scope needed to inspect or delete that pending resource. Cleanup must compare the unchanged claim under its lifecycle fence before mutation so an old readiness or cleanup attempt cannot overwrite or delete a newer claim.

Related docs

• stop command
• cleanup command
• status command
• inspect command
• Identifiers
• Security

Direct Azure acquisition identity

After successful VM creation, direct Azure acquisition binds the returned native VM ID to the requested VM name and ownership tags. A final readiness observation must retain that binding before bootstrap, ready tagging, or claim publication. An incomplete creation result or changed readiness identity reports unresolved cleanup and withholds VM deletion; a later matching read does not erase the contradiction.

Ordinary failures after a valid creation use the same prepared owned-cleanup operations as release: re-read the original VM, capture companion identities, revalidate them, delete the VM first, then its surviving companions. Refusal or cleanup failure preserves the original acquisition error and prevents a fresh bootstrap retry. This in-process rollback has no persisted recovery claim and does not provide crash-resumable companion cleanup.

These are observed-identity checks, not an atomic generation-conditional Azure DELETE. Inner create-failure rollback and Windows setup performed during create, intermediate hidden readiness polls, and replacement races after the final validation remain separate boundaries. This does not change GCP cleanup or make name-addressed resources equivalent to immutable-ID deletion targets.