Capacity And Fallback

September 4, 2026 · View on GitHub

Read this when:

  • adding or changing machine classes;
  • debugging "why did Crabbox pick this instance type?";
  • working on AWS or Azure Spot/on-demand failover;
  • configuring multi-region or multi-AZ capacity for AWS or Azure.

Crabbox treats capacity in three layers:

  1. Class fallback — the ordered list of provider instance types that a class request expands into.
  2. Market fallback — AWS/Azure Spot-to-On-Demand failover within a class.
  3. Region/AZ routing — where the brokered AWS or Azure path may launch when one region or zone is configured against several.

Hetzner only exercises class fallback. AWS exercises all three; Azure exercises class, market, and region fallback for VM leases. Static SSH and the delegated runners (Blacksmith, Daytona, Islo, E2B, Modal, and the rest) have no capacity fallback because the operator or the external service owns the underlying resource.

Classes

Class names are provider-neutral intent labels:

tiny      smoke checks and minimal repositories
small     small repositories and light CI lanes
standard  typical CI lane
fast      more cores for parallel-friendly suites
large     memory-heavy or many-process workloads
beast     maximum capacity within the provider's family

Each provider maps the six class names to an ordered list of concrete instance types. That list is the fallback chain: try the first; if it is rejected, try the next; and so on until one succeeds or the chain is exhausted.

The full per-provider class tables (including AWS Windows, Windows WSL2, and macOS) live in Providers.

When Class Fallback Triggers

Hetzner falls back when:

  • the requested server type is unavailable in the configured location;
  • the project quota rejects the request;
  • the API returns a transient capacity error.

AWS and Azure fall back when a candidate is rejected by:

  • insufficient instance capacity (InsufficientInstanceCapacity);
  • a vCPU or Spot quota limit (VcpuLimitExceeded, MaxSpotInstanceCountExceeded);
  • an unsupported type or account policy (e.g. Free-Tier restrictions).

Crabbox classifies each provider rejection (capacity, quota, unsupported, policy, region) rather than guessing, then advances to the next candidate type. Azure VM leases also bound slow long-running create operations so requests can advance through the candidate chain instead of waiting on a stalled allocation. Non-retryable failures stop the chain immediately so a genuine misconfiguration fails fast instead of churning through every type.

When the chain is exhausted the command fails. On the brokered (Worker) path the failure carries provisioningAttempts — one record per type tried, with its region, market, failure category, and message — and the same metadata is attached to the failed lease record on the coordinator, so an operator can see what went wrong without rerunning the workflow:

crabbox history --lease cbx_...

Explicit Type Override

--type c7a.16xlarge (and the matching type: config key) pins one exact instance type and skips the class fallback chain. The contract is "give me this type, not a substitute": if the provider rejects it, Crabbox fails loudly rather than silently choosing a different type.

Use --type when:

  • you want deterministic capacity for a benchmark;
  • you are pinning a specific generation to work around a known issue;
  • you are debugging the capacity layer itself.

For everything else, prefer a class so the fallback chain absorbs transient rejections without operator intervention.

AWS And Azure Market Fallback

AWS and Azure support two markets: spot and on-demand.

capacity:
  market: spot
  fallback: on-demand-after-120s

capacity.market: spot (the default) requests Spot capacity first. capacity.fallback controls whether a rejected Spot request retries the same class on On-Demand. Any value that starts with on-demand (the shipped default is on-demand-after-120s) enables the On-Demand retry pass; set it to none (or leave it empty) to never fall back.

The On-Demand pass runs after every Spot candidate in the class chain has been tried and rejected. AWS retries only the candidates whose individual failures are Spot-recoverable, including Spot capacity or quota errors and request errors that explicitly identify Spot as unsupported. Market-independent request or image errors, such as an invalid block-device mapping, are not retried on On-Demand. Azure also treats a slow Spot VM provisioning operation as a capacity miss after the configured on-demand-after-* duration, or after the default 120 seconds when on-demand fallback is disabled with spot-only or none. Slow Azure On-Demand creates are bounded too, so the coordinator can keep trying the class chain before the CLI lease wait expires.

Per-command overrides:

crabbox warmup --market spot
crabbox run --market on-demand -- pnpm test

--market overrides capacity.market for one lease without editing repo config. It must be spot or on-demand. Reach for it when an account is temporarily out of Spot quota or when Spot interruption rates spike. The same value is also available as the CRABBOX_CAPACITY_MARKET environment variable.

Tencent Cloud Market Selection

Tencent Cloud also honors explicit spot and on-demand choices, mapping them to CVM SPOTPAID and POSTPAID_BY_HOUR. This is selection only: Tencent does not perform market, instance-type, or region fallback, and capacity.fallback does not change that behavior. Spot requests use Tencent's default discounted bid without a Crabbox maximum-price override.

An unconfigured Tencent lease keeps its historical hourly billing, independent of the generic spot default. After upgrading, an explicitly configured capacity.market: spot, environment override, CLI flag, or job market now requests interruptible Spot capacity; select on-demand to keep hourly billing. See the Tencent Cloud provider guide.

AWS Capacity Hints

Successfully provisioned brokered AWS, Azure, and GCP leases retain ordered provisioningAttempts, including failed candidates from earlier markets and regions or zones. Each record describes the actual attempted type, market, location, and provider-classified failure; AWS quota preflight rejections are recorded even when no instance launch was needed. These diagnostics do not authorize retry or cleanup: each provider still owns those decisions and its durable allocation claims. Failed brokered leases still report their terminal failure separately; this history is not a durable journal of unresolved provisioning or cleanup work.

The brokered AWS path can preflight large requests against Service Quotas:

capacity:
  hints: true

When hints is enabled (the shipped default), the broker checks the applied Spot or On-Demand vCPU quota for each candidate type before launching it. Candidates that exceed the quota are recorded as a quota attempt and skipped, so the chain spends launch attempts only on types the account can actually run. Brokered failures also emit advisory CapacityHint records (for example, when a large class is under capacity pressure) alongside provisioningAttempts.

The set of classes treated as high-pressure for hint purposes is controlled on the broker by the CRABBOX_CAPACITY_LARGE_CLASSES environment variable (comma-separated, default beast); it is not a repo config key.

Hints apply only on the brokered path. Direct AWS mode still walks the class chain but runs no quota preflight. To preflight quota before the first warmup in either mode, run:

crabbox doctor --provider aws

It reports the applied EC2 vCPU quota for the relevant market(s) and warns when the default class needs more vCPUs than the account is allowed.

Region And Availability Zone Routing

capacity:
  regions:
    - eu-west-1
    - us-east-1
  availabilityZones:
    - eu-west-1a
    - eu-west-1b

regions is the ordered list of brokered AWS regions the coordinator considers. Single-region setups use aws.region and leave capacity.regions empty; multi-region setups list every region the broker may launch into. Regions are tried in order: when every candidate type in a region is rejected for a retryable reason, Crabbox advances to the next region.

Brokered Azure uses CRABBOX_AZURE_REGIONS on the coordinator for its Azure-specific region list, so AWS and Azure do not accidentally share incompatible region names. The CLI-level capacity.regions list is still included in Azure lease requests for direct or explicitly shared setups.

availabilityZones narrows zone selection within a region (used today mainly for EC2 Mac Dedicated Host allocation). Zones that do not belong to the active region are ignored.

Both keys also accept comma-separated values from the CRABBOX_CAPACITY_REGIONS and CRABBOX_CAPACITY_AVAILABILITY_ZONES environment variables.

Capacity Strategy

capacity:
  strategy: most-available

capacity.strategy is accepted and persisted (it is one of most-available [default], price-capacity-optimized, capacity-optimized, or sequential), and it shows up in crabbox config show. It does not yet change provisioning behavior: both direct and brokered AWS provisioning currently follow the class chain in order. Treat the field as forward-looking configuration rather than a behavior switch, and prefer ordering the class chain itself when you need a specific preference today.

Direct Mode Differences

Direct provider mode (no coordinator) supports class fallback and the spot-to-on-demand failover above, but has no Service Quotas preflight, no provisioningAttempts metadata, and no central run history. If a direct AWS run fails to provision, rerun it once through the broker to capture structured provisioningAttempts evidence, then return to direct mode for the rest of the iteration loop.

Failure Surface

crabbox returns 0 on success. Crabbox-internal failures — including capacity exhaustion, provisioning that never reached SSH, and a lease that expired during a long warmup — are reported before the remote command runs and exit non-zero; when the remote command does run, its own exit code is passed through verbatim. There is no fixed numeric enum for the internal categories (see the CLI exit codes reference); branch on 0 versus non-zero and read stderr or --json for the reason.

The error message names the class chain, the markets that were tried, and — for brokered runs — the provisioningAttempts you can inspect with crabbox history --lease cbx_....

Related docs: