Runner Bootstrap

September 8, 2026 · View on GitHub

Read this when:

  • you are changing cloud-init or the per-OS bootstrap scripts;
  • you are debugging a managed box that never becomes SSH-ready;
  • you are changing the minimal runner contract or the crabbox-ready checks.

Bootstrap is what turns a freshly provisioned cloud instance into a box Crabbox can reach over SSH and run commands on. It is generated by the CLI for direct-provider leases and by the coordinator for brokered leases; both surfaces share common script fragments while retaining their own provider and runtime composition. Code lives in internal/cli/bootstrap.go and worker/src/bootstrap.ts; shared sources live in recipes/bootstrap/v1/.

Bootstrapped boxes carry no coordinator credentials. The box never calls the broker; the CLI connects to it directly over SSH.

Managed WSL2 distributions additionally install Node/npm through the bundled --node-only entrypoint of scripts/install-linux-developer-tools.sh. The bootstrap generator reads that canonical script into both runtimes; regenerate with node scripts/generate-bootstrap.mjs after changing it, and refresh its input SHA-256 in recipes/devtools/v1/linux-x86_64.json. The default amd64 Node version and archive verification match Linux developer images. WSL2's ready check requires Node and npm, but its setup omits the full developer image's Docker, Go, browser tools, pnpm activation, and offline pnpm archive set.

The minimal Linux contract

Brokered and direct cloud Linux runners are Ubuntu machines configured by cloud-init. The bootstrap deliberately does not run apt upgrade (package_upgrade: false), so a box comes up fast and predictable rather than chasing the latest package set.

Bootstrap creates:

  • the crabbox SSH user, in the sudo group with passwordless sudo;
  • key-only SSH (PasswordAuthentication no), authorizing the per-lease public key;
  • SSH listening on the primary port (default 2222) plus the configured fallback ports (default 22);
  • the work root (/work/crabbox by default), owned by crabbox;
  • shared package cache directories /var/cache/crabbox/pnpm and /var/cache/crabbox/npm.

After writing the SSH port configuration, bootstrap reloads systemd and restarts the active ssh.socket, or the SSH service on images without socket activation. This also runs on prepared images that skip package installation: restarting only the service would keep the socket's previous listening ports.

Bootstrap installs only a small base set with --no-install-recommends:

  • ca-certificates
  • curl
  • git
  • jq
  • openssh-server
  • rsync
  • tmux
  • util-linux

apt-get runs are wrapped in a retry loop (8 attempts, increasing backoff) so a transient mirror failure does not fail the whole boot.

The minimal bootstrap's APT refresh skips translation, AppStream DEP-11, and command-not-found indexes through command-local APT options. These auxiliary indexes are not needed to install the baseline packages. Package indexes, repository signature verification, and installation errors are unchanged; slow package mirrors can still delay boot. Later operator updates and separate developer-tools or project setup refreshes keep their existing behavior.

Managed Debian and Ubuntu images describe this baseline in the canonical /var/lib/crabbox-readiness/linux.json manifest. The dedicated readiness directory is root-owned and mode 0755, independent of the runtime-user-owned /var/lib/crabbox state directory. The strict crabbox-linux-readiness/v1 object contains exactly profile, recipeDigest, and schema; its SHA-256 recipe digest covers executable package, probe, and inheritance requirements, not cosmetic recipe descriptions. The shared recipes in recipes/linux/v1/ generate identical CLI and coordinator bootstrap fragments plus the standalone scripts/linux-readiness.generated.sh producer.

The linux-minimal profile proves the executable SSH daemon, nonempty system CA bundle, and working curl, git, rsync, jq, tmux, and flock commands. The optional linux-builder profile inherits every minimal requirement and additionally proves the generic build-essential, git-lfs, pkg-config, python3, and python3-venv capabilities. The virtual-environment probe creates a disposable, pip-enabled environment, runs its Python and pip, and cleans the temporary directory on success or failure. Cold bootstrap installs and claims only linux-minimal; an image producer claims linux-builder only after every minimal and builder probe passes.

Bootstrap skips baseline APT only when the exact canonical manifest bytes, root-owned non-symlink path, root group, 0644 file mode, bounded file size, non-writable parent directories, and every declared profile probe are verified. Named probe commands use only the sanitized system PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin and bypass shell functions, so workspace or user-provided commands cannot satisfy the contract. Strict canonical-byte comparison uses sha256sum from Debian/Ubuntu's essential coreutils package, so validation does not require Python or permissive JSON stream parsing. An existing invalid manifest always takes the normal install path. The root-owned /var/lib/crabbox/image-ready file remains a non-authoritative compatibility hint in a directory the runtime user may own; its exact bytes, owner, group, and mode are checked, but its parent is not the manifest trust boundary. Migration is permitted only when no manifest exists and every minimal probe independently succeeds, with zero APT or dpkg calls. The compatibility marker is prepared inside the root-owned readiness directory and renamed into the legacy directory without following the destination. If the legacy directory does not exist yet, the marker writer first verifies its ancestors and creates it root-owned with mode 0755; existing safe directories owned by the runtime user remain valid. Missing tools are installed normally or fail visibly if installation is unavailable. Optional browser, desktop, and Tailscale installation remains independent.

The manifest is local image capability evidence. It is not package-version attestation, signed provenance, tenant isolation, or a ready-pool identity. Per-lease identity, SSH, work-root, optional capability, and readiness steps still run for every lease.

Readiness: crabbox-ready

Bootstrap writes /usr/local/bin/crabbox-ready and runs it at the end of boot. A box does not count as ready until this script exits 0 over SSH. The base Linux script checks:

  • git, rsync, curl, jq, tmux, and flock are present and runnable;
  • the marker file /var/lib/crabbox/bootstrapped exists;
  • the work root is writable.

Optional lease capabilities (below) and Tailscale extend crabbox-ready with additional checks, so readiness always reflects the full requested capability set.

What bootstrap does not install

Bootstrap intentionally omits project language runtimes and services — Go, Node, pnpm, Docker, databases, and the like. Those are repository-owned setup and belong in Actions hydration, devcontainers, Nix, mise/asdf, or repository setup scripts. Keeping bootstrap minimal keeps boot fast and keeps the contract stable enough that snapshots and prebaked images can replace cloud-init. See Prebaked runner images.

Optional lease capabilities

Interactive tooling is opt-in per lease, never part of the minimal bootstrap. Each requested capability appends both its install steps and its crabbox-ready checks, and is gated by the provider's declared feature set.

  • --desktop — installs a headless desktop. The default managed XFCE environment adds resize-capable TigerVNC on display :99 and an XFCE session, with VNC on 127.0.0.1:5900. With --desktop-env wayland (labwc) or --desktop-env gnome it installs a Wayland session driven by wayvnc instead. Readiness checks the relevant systemd units and that something is listening on 127.0.0.1:5900.
  • --browser — installs Google Chrome stable, falling back to Chromium, plus the native-addon build helpers (build-essential, python3) that browser-channel QA often needs during dependency fallback installs. It writes a crabbox-browser wrapper and /var/lib/crabbox/browser.env; readiness verifies the wrapper runs --version.
  • --code — installs code-server (managed Linux only) for the authenticated portal editor; readiness verifies code-server --version.

Crabbox owns these machine capabilities; scenario systems still own browser automation and proof artifacts. For slow QA lanes, bake these capabilities into a provider image while keeping secrets, browser profiles, repository checkouts, and built artifacts out of the image. See Interactive desktop and VNC and Prebaked runner images.

Tailscale

--tailscale on a managed Linux lease is also optional. The default package mode installs Tailscale from its signed APT repository with a pinned keyring; the opt-in pinned mode verifies a versioned static archive by SHA-256. Bootstrap then brings the box up on the configured tailnet, writes non-secret metadata under /var/lib/crabbox (such as tailscale-ipv4, tailscale-hostname, and exit-node details), and extends crabbox-ready with a bounded check that a 100.x address has appeared.

The auth key is piped to tailscale up through stdin and is not persisted or placed in process arguments. Brokered leases receive a one-off key minted by the coordinator; direct-provider leases read it from CRABBOX_TAILSCALE_AUTH_KEY. Set TS_CONTROL_URL on the operator shell to register the box against a self-hosted control plane (Headscale and similar) instead of the default Tailscale control plane.

When a lease also carries a --pond label, the Tailscale bootstrap installs a 30-second timer that rewrites /etc/hosts.cbx and a managed block in /etc/hosts so pond peers resolve as <slug>.cbx. Peers are discovered from the box-local tailscale status filtered by the pond ACL tag, so the broker never sees a Tailscale credential. See Tailscale and the pond notes in Provider Reference.

Other targets

macOS and Windows (managed)

aws (Windows and Mac instances) and azure (Windows) bootstrap non-Linux boxes with per-OS scripts rather than cloud-init:

  • macOS — a shell script creates SSH access for the lease user, enables Remote Login on the configured ports, enables Screen Sharing, and writes a crabbox-ready that checks rsync/curl, a writable work root, an open SSH port, and the VNC port 5900. Shell tracing stays disabled so the generated account password is not copied into user-data logs; command errors remain visible. The password file is private from creation and feeds the account tool over stdin rather than process arguments.
  • Windows — a PowerShell script installs OpenSSH, configures key-only access for administrators, enables the internal-sftp subsystem before restarting sshd, opens firewall rules on the SSH ports, and installs Git for Windows so git and tar are on the machine PATH. Crabbox verifies pinned SHA-256 values before extracting or executing those downloads; desktop TightVNC and the versioned WSL rootfs use the same fail-closed check. --windows-mode wsl2 additionally enables the WSL feature set, imports an Ubuntu rootfs, and runs the minimal Linux base inside WSL; --desktop adds TightVNC. The work root defaults to C:\crabbox (native) or the Linux default inside WSL.

Existing OpenSSH services are reused. Both runtimes resolve ssh-keygen.exe from C:\Program Files\OpenSSH, then %WINDIR%\System32\OpenSSH, then PATH, and fail if none is present. This supports images with Windows' built-in OpenSSH as well as the pinned archive installation.

Managed native Windows bootstrap (--windows-mode normal, the default) also provides the native Visual C++ v14 runtime before OpenSSH/Git installation and before writing setup-complete. The shared recipe covers Azure extension bootstrap, Azure snapshot rehydration, AWS's post-EC2Launch bootstrap, and Parallels. It checks actual DLL loadability in a fresh native PowerShell child process on every run: vcruntime140.dll, msvcp140.dll, and (AMD64 only) vcruntime140_1.dll. A healthy runtime needs no download or installation. Architecture comes from Windows IsWow64Process2, independently of the controller's requested architecture. WOW64, 32-bit, and emulated PowerShell fail with instructions to rerun in native AMD64 or ARM64 PowerShell matching the OS. The ARM64 check covers native ARM64 applications; it does not attest x86 or emulated x64 application prerequisites, or make other bootstrap downloads ARM64-native.

WSL2 uses a distinct Linux runtime flow. Its generated bootstrap omits the native VC++ helper definitions, installer URLs and pins, and runtime readiness gate, including elevation, download, pending-boot, and stale-readiness handling. WSL2 retains its existing Windows baseline, Linux setup, and finalization/reboot behavior.

Missing or unloadable runtime DLLs require elevated managed bootstrap. The installer uses immutable Microsoft URLs and SHA-256 pins in artifacts.json (retrieved 2026-09-02). It downloads into a unique directory restricted to Administrators and SYSTEM, verifies the ACL, then requires both the pinned hash and Valid Authenticode with the exact Microsoft Corporation publisher identity before execution. Publisher validation compares every expected subject field, not a substring or signature status alone. Only downloads are retried, at most three times. The installer runs once with /install /quiet /norestart; exit 0 requires a fresh loadability check. Exit 3010 blocks readiness until an operator reboots outside bootstrap and retries; 1641 and other nonzero exits fail explicitly. The Azure extension never reboots for this prerequisite.

A pending installation boot identifier is stored under HKLM:\SOFTWARE\Crabbox\Bootstrap\WindowsRuntime before installer execution. It prevents an interrupted install or a reboot-required result from becoming false readiness on a same-boot retry, even when the DLLs already load. A failed installation or postcondition requires investigation and an external reboot before retrying. After reboot the helper probes again, installs only if still needed, and clears its pending state after success. Bootstrap removes stale setup-complete before checking the runtime; an existing desktop setup retains its prior first-setup reboot decision. Temporary cleanup removes only the helper's own staging directory.

This is a machine prerequisite, not repository setup: Node versions, pinned package managers, and their existing checks remain repository-owned. Hyper-V and scripts/install-windows-developer-tools.ps1 retain their separately owned provisioning paths; this change does not rewrite those paths.

Static / BYO SSH hosts

provider=ssh (aliases static, static-ssh) targets are not bootstrapped by Crabbox. They are assumed to be operator-managed and must already provide:

  • macOS targets: SSH, bash, git, rsync, and tar;
  • Windows WSL2 targets: Windows OpenSSH with Subsystem sftp internal-sftp, WSL, bash, git, rsync, and tar;
  • native Windows targets: OpenSSH, PowerShell, git, tar, and the Visual C++ runtime matching each native application's architecture;
  • static.workRoot pointing at a writable directory for that target mode.

For native Windows, install Git before the Crabbox check or restart OpenSSH Server afterward, so new non-interactive SSH sessions inherit git and tar on PATH.

BYO operators must install or repair the required VC++ runtime themselves; Crabbox does not install it on static hosts. An executable exiting -1073741515 (0xC0000135, STATUS_DLL_NOT_FOUND) has an unresolved DLL dependency. Missing VC++ runtime DLLs are one cause, not the only cause; a working Node executable or an executable found on PATH does not establish that a native package manager can load its dependencies. Preserve the failing exit code and repository package-manager checks when diagnosing this failure.

For Windows WSL2, run crabbox doctor --provider ssh --target windows --windows-mode wsl2 --static-host <host> --doctor-probe-ssh after changing OpenSSH or WSL. Doctor verifies the SFTP handshake without staging a workload; only a conclusive OpenSSH subsystem rejection is diagnosed as missing SFTP. Staged commands upload one digest-bound envelope without a content readback; the native exclusive-handle verifier remains mandatory before execution or deletion. Windows PowerShell 5.1 is supported, including its stdin encoding preamble. The launcher uses existing distribution tools without installing a Linux loader or requiring Windows-drive automount.

SSH port fallback

The CLI prefers the configured SSH port and falls back through ssh.fallbackPorts during early bootstrap or when operator-network egress restricts ports. The default is primary 2222 with a 22 fallback. Disable the fallback with ssh.fallbackPorts: [] in config or CRABBOX_SSH_FALLBACK_PORTS=none in the environment.

Once the bootstrap contract is stable, snapshots or provider images can replace slow cloud-init while preserving the same readiness contract.

Editing shared bootstrap sources

Edit recipes/bootstrap/v1/ rather than the generated Go or TypeScript files:

  • os-catalog.json defines the default portable OS, supported-selector order, aliases after case/separator normalization, and image metadata. Provider adapters still choose images and architecture-specific behavior; the catalog does not choose providers or change fallback policy. For example, Ubuntu 26.04 retains its existing Hetzner 24.04 mapping.
  • artifacts.json defines pinned URLs, versions, SHA-256 digests, and the browser signing-key fingerprint. The generator produces constants for both runtimes, including the existing Tailscale default exports. Runtime override lookup and precedence remain outside the generator.
  • fragments.json declares each script asset and its typed parameters. The .ps1 and .sh files own the Windows header/core, native/desktop preludes, desktop setup and finalization, the installed GNOME theme helper, macOS bootstrap, Linux code-server and Tailscale installers, and WSL TruffleHog installer.

The Windows header composes windowsRuntime.ps1 definitions followed by windowsRuntimeGate.ps1 only for native mode, before the managed core and readiness. The gate owns stale-readiness invalidation and the helper call; windowsCore.ps1 remains shared with WSL2 and contains no runtime gate.

Run node scripts/generate-bootstrap.mjs after edits, then node scripts/generate-bootstrap.mjs --check. Generation requires Node and gofmt; consumers use compiled constants and functions, with no runtime recipe loading or new dependency. CI checks for stale output and runs node --test scripts/generate-bootstrap.test.mjs. The generator validates exact metadata fields, hashes, URLs, aliases, parameter types, and template tokens; duplicate JSON keys, unknown tokens, and unused parameters fail generation.

A fragment marked "literal": true must have no parameters and is emitted byte-for-byte, without placeholder parsing. Use this for standalone scripts such as the GNOME theme helper, where nested shell expansions contain template-like braces. Omitting the flag retains strict template validation.

Templates use {{parameter}} for typed runtime inputs and {{ps:constant}} or {{sh:constant}} for quoted artifact metadata. Runtime strings and string lists are emitted through PowerShell or Bash quoting, never through raw replacement or an evaluated template. Bash newline escapes keep injected values from ending the macOS readiness here-document. Pinned Linux installers pass download arguments directly to curl, without a second shell evaluation.

The shared fixture suite compares compiled Go output with generated TypeScript output exactly, including quotes, backslashes, shell metacharacters, Unicode, and multiline values. Composition tests cover Linux optional capabilities, native Windows, WSL2, and macOS. Architecture fixtures exercise amd64 and arm64 installer selection. When PowerShell is available, local filesystem tests exercise the OpenSSH resolver, parser, and checksum rejection without installing services or downloading packages; these are not Windows machine readiness proof.

node --test scripts/windows-runtime.test.mjs adds runtime-specific local assertions and runs the native behavioral harness when on Windows. On a native Windows runner, run the harness directly with no downloads or installation:

powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File scripts/test-windows-runtime.ps1

Use native ARM64 PowerShell on ARM64 Windows. The harness parses the helper, exercises native architecture/DLL APIs and a fresh child probe, then mocks OS, download, signature, staging, and process boundaries to check idempotence, security rejection, architecture errors, retry bounds, installer exit codes, postconditions, reboot retry, and readiness failure. It does not validate the downloaded Microsoft publisher certificate or a real installation.

For separately authorized native installation proof, the generator also emits scripts/windows-runtime.generated.ps1. Dot-source it in elevated native PowerShell, then call its entry point:

. ./scripts/windows-runtime.generated.ps1
Ensure-CrabboxWindowsRuntime

This file contains the same parameter-free fragment as Go's sharedWindowsRuntime() and TypeScript's exported sharedWindowsRuntime(); dot-sourcing alone does not install anything. Regenerate it with node scripts/generate-bootstrap.mjs and check it with --check. Edit only recipes/bootstrap/v1/windowsRuntime.ps1 and the artifact recipe, never the generated copy. Native publisher and installation proof remains separate from the mocked harness.

Whole bootstrap outputs intentionally need not match. EC2Launch wrapping, compression, YAML indentation, coordinator SSH host-key injection, private AWS SSM-only bootstrap, CLI desktop launcher installation, Azure snapshot credential and host-key rotation, and provider additions remain explicit in their current composition code. The CLI still supplies a default macOS work root when called with an empty value; the Worker receives its resolved config value.

Remaining handwritten common areas include Linux desktop/browser setup, the outer cloud-init/readiness wrappers, and WSL provisioning/readiness. WSL retains its pre-existing differences: the CLI clears the setup marker and checks Python and wslpath; the Worker repairs and checks WSLInterop and installs tmux. Those contracts need a separate behavior decision before further consolidation. Shared Linux readiness generation in recipes/linux/v1/ is unchanged.