Provisioned Linux offline doctor lifecycle gate v1

September 21, 2026 · View on GitHub

Audience: release engineers and security reviewers who can supply one disposable, trusted Linux x86-64 host, or dispatch this gate against a GitHub-hosted runner.

Status: executed, and still failing, but narrowly, most recently 2026-09-20 against 4496d1a0 (run 35477758519). Preconditions and settlement pass. Both admitted suites report 12 passed, 1 failed of 13. The remaining real-distributions fixture shows Clang successfully, while Node terminates with SIGSEGV (signal 11) and rustc exits with status 127. This is a hosted failure, not promotion evidence. See Executions.

Owning contract: Linux production offline doctor provisioner v1. This document adds the executable gate that contract's distribution and evidence section requires; it changes no admission rule, activates no ordinary CLI route, and promotes no completion row.

Executions

Four runs, all failed, all on a GitHub-hosted ubuntu-24.04 runner: 34040867346 at 758388e2 (evidence bundle c5be8289f0c22d40895c7065669266e93dc0720601f24cd436db9b2f0ee17de8), 34041757908 at 7b5dfd65, 34043466045 at 0f7f7639, and 34047743589 at c53386a9. Each selected test harness exited 101; the collector suite ran 3 passed, 10 failed every time. Two earlier dispatches in this range, 34038624936 and 34039884083, are excluded from this count on purpose: both failed in workflow packaging steps ("Assemble the real distribution carrier" and "Package and independently unpack the signed release", respectively) before the gate script ever started, so neither carries any gate evidence.

What the runs establish. Zero precondition failures: all twelve required kernel features observed present on 6.17.0-1022-azure x86_64; the delegated scope reported delegated: true, populated: 0, procs: []; the signed release was unpacked outside the checkout at the checked-out commit under the test-only anchor with production_signing_material_present: false; the checkout was clean; every image was static ELF with interpreter: None. Settlement passed every time — the final cgroup still existed, populated 0, with no surviving members, so nothing leaked despite the failure.

The third run added more, because Observation::describe started carrying the collector's own output into the status assertion instead of discarding it. The confined worker materializes into tmpfs, pivots, applies seccomp, reaches execve, and emits a well-formed canonical report — correct schema and target, five of six checks ok, and an empty stderr — which the collector delivers to the fixture. Exit 1 is the correct response to the sixth check.

Through the third run, every tool check reported offline tool terminated unsuccessfully: clang in every failing fixture, and node and rust as well wherever all three roles are exercised. Nine fixtures run the collector-fixture profile's sentinel image and one runs real-distributions; both failed identically. The detail renders DoctorProbeError::Exit, which is any nonzero child status, so through the third run alone the evidence could not say which of two causes it was: the tool itself exiting nonzero (exit 7 for the fixture image, taken iff write(1, payload, len) != len), or the worker's own child setup rejecting something and never reaching the tool (fail_stop is _exit(126), and child.rs::enter reaches it on a failed dup2, materialize, chroot, close_range, guard.install, or execve, among others). The report carries no exit code, which is correct for a report and is why it cannot distinguish them.

The fourth run resolves that. Through the third run, the collector suite's fixtures assert on a report's exit code and so stop before the underlying reply-frame bytes are compared, and the only suite whose fixtures assert on those bytes directly — platform-sys-lib — had never run, because execute() broke after the first failing suite. c53386a9 changed that: every admitted suite now runs (failure selection stays sticky regardless, since Settlement.selected is the first recorded reason and nothing reorders the list), and the fourth run is the first time platform-sys-lib executed at all. Verified directly from the run's own log: 6 passed, 7 failed of its 13 fixtures, alongside the same 3 passed / 10 failed collector result.

The decisive failure among those seven is doctor::offline_worker::tests::lifecycle::post_exec_capabilities_and_supervisor_death_are_observed_externally. Its fixture image (fixture::machine_code with spin = true) is a bare jmp self — two bytes, 0xeb 0xfe — which issues no syscall of any kind and cannot exit, write, or fail a comparison on its own; the test drives it to prove supervisor-death observation while the child spins forever. It failed anyway. A process incapable of exiting by its own code that nonetheless produces a failing (non-ok) observation did not run its own code at all — so, at least for this fixture, the rejection is inside child.rs::enter, at or before execve, which eliminates the two post-execve explanations this document previously left open (a denied syscall — the spin image makes none — and a short write — it has no write). The fixture image and a full clang have almost nothing in common and fail identically, which is better explained by one shared cause before execve than by independent post-execve failures in each.

What is still not established. The confinement boundary's properties. Mechanics execute (materialize, pivot, seccomp, execve is reached by the report-emitting path, settlement), but every fixture that would demonstrate a property — including the negative ones, such as rejecting image defects, digest drift, a missing loader, or a non-child pidfd — fails on this defect before reaching its own assertion. The carrier ceiling recorded below was a real defect, fixed in 7b5dfd65; it was never why a fixture failed.

Which rejection inside child.rs::enter fires first is also still unestablished. ProbeError::Exit collapses every nonzero child status into one wire value (offline_worker/wire.rs), and Operations::reap_owned — the trait seam it crosses — discards the numeric status before any point a fixture can observe. Naming the exact step therefore needs one of: a fixture-only capture behind #[cfg(test)], a signature change to Operations::reap_owned (touching its Linux implementation and every mock), or a versioned reply-frame change (a protocol surface this gate exists to protect) — or, more cheaply, bisecting child.rs::enter's enumerated rejections directly on a provisioned host, which needs no code change. No run has done any of these yet.

Runs five through thirty-eight, and what they overturned

The four runs above are no longer the record. gh run list --workflow doctor-provisioned-linux.yml reports 38 dispatches — 30 failure, 8 cancelled — the most recent on 2026-09-17 against bfb1da19 (run 35274228390). The gate has been iterated on continuously rather than left where the fourth run stopped, and the commits doing it are on main: raising the worker's rlimits (NOFILE 64 → 256, AS 2G → 4G), admitting O_NOATIME and O_PATH in the worker's open filter, tolerating EINVAL in the signal reset, and admitting the event-loop syscalls a real node --version issues.

The pre-execve conclusion above is now falsified at the current head, by this gate's own output. Read directly from run 35274228390's log, the platform-sys-lib suite reports:

test result: FAILED. 12 passed; 1 failed; 0 ignored; 0 measured; 188 filtered out

and the collector-provisioned suite reports the same shape:

test result: FAILED. 12 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out

Two things follow, and both contradict the fourth-run reading:

  1. doctor::offline_worker::tests::lifecycle::post_exec_capabilities_and_supervisor_death_are_observed_externally — the zero-syscall jmp self sentinel whose failure was the whole basis for "the child never reaches its own code" — now passes. Whatever rejected it inside child.rs::enter has been fixed.
  2. The confined child demonstrably reaches and runs real tool code. In the one remaining failing fixture, the emitted report carries "id":"clang" ... "status":"ok" with the detail clang version 17.0.6, read out of the real provisioned distribution. A confined process that never reached execve cannot report a version string it had to execute a compiler to obtain.

In run 35274228390, the remaining failure was therefore narrow and specific, not structural: clang succeeds while node and rust both report offline tool terminated unsuccessfully, leaving the overall report exit 1 where the fixture asserts exit 0. The two failing test names are doctor::offline_worker::tests::provisioned_real_clang_node_rust_distributions (panicking at offline_worker/tests.rs:404 with "real selected tool must complete under confinement: Exit") and real_launched_handoff::production_launcher_reports_all_roles_from_provisioned_real_distributions.

What this does not license. Two of thirteen fixtures still fail, so WP-05 stays unpromoted and no confinement property is claimed as established on the strength of this section. The evidence is a GitHub-hosted ubuntu-24.04 runner only; nothing here is evidence for macOS, Windows, or a physical device. The cause of the node/rust failure is not established — only that it is downstream of execve rather than before it, which is where the four earlier runs placed it.

Latest execution: run 35568902945

The latest dispatch ran commit ce38c335 on a GitHub-hosted ubuntu-24.04 runner. All host, release, image, and cgroup preconditions passed, and the final delegated cgroup was empty. The diagnostic reply trailer and all-role failure collection identify both failing roles in the platform-sys suite:

role 2 failed under confinement: Exit: the tool was killed by signal 11 (SIGSEGV)
role 4 failed under confinement: Exit: the tool's own process exited with status 127

The collector suite's canonical report agrees on the role split: Clang is ok, Node and Rust are failed, and the collector exits one as required. The platform-sys and collector suites each report 12 passed and 1 failed. The first Node-only RLIMIT_AS increase to exactly 2 TiB did not change its termination. V8 can transiently map a 2 TiB candidate to obtain a 1 TiB-aligned sandbox while loader and ordinary mappings are already charged, so the next tranche gives Node a finite 4 TiB ceiling while retaining the fixed 4 GiB physical- memory cgroup limit. Rust's status 127 remains separately unexplained. The workflow now prints a strace -f -c census and a path-free fcntl command trace for the exact staged Node and rustc bytes before confinement; this is diagnostic evidence only and grants the worker nothing. The observed fcntl operation supports only the role-local x86 F_GETFD/F_GETFL query rule; it does not admit descriptor mutation or creation. The command trace must settle the actual requests before any broader policy change is considered.

What is true today

The private Linux doctor boundary has substantial implementation and source-layout coverage. It now also has runtime evidence that its mechanics execute on a real host, and — unlike when the first four runs were recorded — the property fixtures themselves now execute and pass: 12 of 13 in each admitted suite, including the hostile ones that reject image defects, digest drift, a missing loader, and a non-child pidfd. See Executions. What remains unestablished is narrower than "properties do not hold": one real-distributions fixture in each suite still fails on node and rust, so the boundary is not yet demonstrated against the full real toolchain, and WP-05 stays unpromoted on that basis.

  • tests/doctor_production_provisioner_v1.rs reads the provisioner's source text: it pins the fixed descriptor inventory, the capsule parser, the clone flags, the pivoted read-only tmpfs root, and the absence of an ordinary CLI activation path, and it proves those tripwires reject representative widening mutations. It is a textual gate. It cannot observe a namespace, a cgroup, a syscall, or a settled descendant.
  • Twenty-six lifecycle tests are #[ignore]d because they need a host nobody has provisioned. They are listed under test selection below, and each #[ignore] reason names the missing prerequisite rather than disabling the case.
  • docs/DOCTOR-PRODUCTION-PROVISIONER-V1.md already states that physical distribution evidence and ordinary CLI activation remain unrun and unpromoted, and that missing namespace, cgroup, sealing, or kernel prerequisites fail rather than skip. Nothing here weakens that.
  • docs/COMPLETION-MATRIX.md keeps WP-05 doctor unpromoted, and this document does not change it. A status change requires a run.

The gate

scripts/doctor-provisioned-linux-gate.py is the whole gate. Each evidence path is create-new, regular and owner-only: a later invocation refuses an existing target or link rather than overwriting prior evidence or following a redirected path. .github/workflows/doctor-provisioned-linux.yml is one dispatch-only job that invokes it on a GitHub-hosted ephemeral ubuntu-24.04 runner, provisioning that runner in-job so nothing survives the VM.

The workflow is deliberately outside ci.yml. Issue #61 scopes the run to a disposable trusted environment and does not authorize a privileged workflow over arbitrary fork input, so the gate is never triggered by push or pull_request and never sees a fork's contents.

Failing closed is the point

The single behaviour this gate exists to guarantee is that absent provisioning is a failure, not a skip. GitHub treats a skipped check as a satisfied required status check, so a gate that skips when its host is unprovisioned reports the same green as a gate that proved confinement. That is the exact confusion the owning contract forbids.

Accordingly:

  • the job carries no if: condition, no continue-on-error, and no conditional execution step;
  • precondition_failures returns a nonempty list for every unmet precondition, and an unmet precondition exits 1 before any test runs;
  • a precondition the probe could not observe is treated exactly like one observed to be false — the contract calls an unobservable disagreement a violated trusted-launch precondition, not a reason to continue;
  • libtest_failures rejects a harness outcome in which the named tests did not actually run: a nonzero ignored count, a 0 passed summary, an unmatched filter, a narrowed selection, a missing test result: line, or a test that ran outside the selection are each a failure at exit code 0.

--self-test drives those functions with synthetic inputs and needs no Linux host. It proves rejection of a missing kernel feature, absent cgroup-v2 delegation, a wrong architecture, a missing or dynamically linked binary, unprovisioned context and fixture variables, a release built at another commit, a production trust anchor, an unsettled cgroup, and a skipped-rather-than-run harness outcome. It is evidence that the gate refuses. It is not, and must never be reported as, evidence that the gate passed.

Host provisioning

The gate does not create the environment. It asserts one. The trusted host owner supplies a wrapper that establishes the end state below and then execs the script; the contract already makes the caller, not the provisioner, the owner of the clean fixed-inventory handoff.

Required end state when the script starts:

RequirementWhy
Linux x86-64, 64-bit little-endianThe gate admits exactly one target. AArch64, Windows and macOS stay separately tracked and are never generalized from this run.
A disposable, dedicated hostThe harness is a trusted provisioner outside both offline guarantees; it must not run beside other workloads or over untrusted input.
Kernel features: clone3, CLONE_INTO_CGROUP, close_range, MFD_EXEC, memfd sealing, no_new_privs, openat2, pidfd_open, pidfd_send_signal, pivot_root, seccomp filters, unprivileged user namespacesEach is used by the provisioner, the worker policy, or the sealed-memfd surrogates.
A private mapped user + mount namespaceSEMAPRAX_DOCTOR_ROOT_TEST_CONTEXT and SEMAPRAX_DOCTOR_WORKER_TEST_CONTEXT acknowledge it; the acknowledgement is not attestation.
An empty delegated cgroup-v2 scope, path in SEMAPRAX_DOCTOR_GATE_CGROUPMust expose cgroup.controllers, cgroup.events, cgroup.kill, cgroup.procs, cgroup.subtree_control, cpu.max, memory.max, pids.max; carry the cpu, memory and pids controllers; accept the fixed writes; and report populated 0 with no members.
Immutable current-head SEMAPRAX_DOCTOR_LAUNCHER, SEMAPRAX_DOCTOR_WORKER, SEMAPRAX_DOCTOR_COLLECTORAbsolute physical regular files, executable, native static ELF without PT_INTERP. A dynamic loader cannot be reopened from the ambient root during held-image execution.
A real bundle in SEMAPRAX_DOCTOR_REAL_BUNDLE with its exact SEMAPRAX_DOCTOR_REAL_SELECTOR, plus independent SEMAPRAX_DOCTOR_EXPECTED_{CLANG,NODE,RUST}_DETAILThe real-distribution fixture must not manufacture its own oracle from observed output.
A signed release archive unpacked outside the checkout, identified by SEMAPRAX_DOCTOR_GATE_RELEASE_{ROOT,ARCHIVE,MANIFEST,CAPSULE,COMMIT}Target-directory binaries and synthetic ELF fixtures are explicitly insufficient.
SEMAPRAX_DOCTOR_GATE_TRUST_ANCHOR=test-only, no production signing material in the environmentThe gate signs disposable fixtures only. No production signing material belongs in test output.
A clean checkout at the same commit the release archive was built fromThe evidence binds exact source bytes.
SEMAPRAX_DOCTOR_GATE_DISPOSABLE=yesAn explicit operator declaration that this host is disposable and trusted. It is a declaration, not a proof, and every other check still applies.

The gate derives most kernel-feature answers from the reported kernel release and records that basis in the evidence under kernel_feature_basis. A version is a necessary condition, not a runtime capability proof: a distribution kernel can carry the version with a feature compiled out or policy-blocked. The fixtures themselves still fail rather than skip when a syscall is unavailable, so the probe rejects early without claiming to be the authority.

scripts/doctor-provisioned-linux-provision.sh is that wrapper. It refuses unless the host is Linux on x86-64, unprivileged user namespaces are permitted, the cgroup-v2 unified hierarchy exposes the cpu, memory and pids controllers, and the release directory lies outside the checkout and carries every image; it then delegates the controllers, creates an empty scope exporting SEMAPRAX_DOCTOR_GATE_CGROUP, exports the two context acknowledgements and the three image paths, and execs the gate inside unshare --user --map-root-user --mount --net --ipc --uts. It removes the scope on exit unless --keep is given.

The wrapper has never run on Linux. It was authored and self-tested on macOS/arm64, where every path refuses: argument handling, the Linux-only refusal, and the x86-64 refusal were exercised directly, and the remaining refusal chain was reached with a spoofed uname. Its namespace, cgroup delegation and image immutability behaviour on a real host is unproven, and the operator still supplies SEMAPRAX_DOCTOR_REAL_SELECTOR, the three SEMAPRAX_DOCTOR_EXPECTED_*_DETAIL values and SEMAPRAX_DOCTOR_GATE_DISPOSABLE=yes, which the wrapper deliberately does not invent. Proving those properties on a disposable host remains the maintainer's work; the gate independently rechecks every one of them and fails closed if the wrapper is wrong.

Test selection

The gate runs two suites serially, in this order, with an exact --exact --ignored --test-threads=1 selection and no wildcard. --plan prints the exact commands.

cargo test --locked --offline -p semaprax-doctor-collector --test provisioned

TestCase the issue enumerates
actual_worker_materializes_executes_and_settles_before_canonical_reportHealthy materialize/execute/settle
literal_reply_surrogates_reject_cross_binding_and_malformed_framesForged, cross-bound and truncated replies
complete_literal_frame_followed_by_nonzero_exit_never_becomes_a_reportA complete frame is not settlement
complete_frame_and_capture_eof_each_still_require_worker_exitDeadlines; two real 60-second budgets
created_handoff::production_created_native_and_all_files_reach_worker_and_reject_digest_driftSealed carrier handoff and digest drift
prepared_handoff::prepared_native_and_all_role_handoffs_preserve_literal_wire_and_reject_transport_driftImmutable request/bundle reacquisition
launched_handoff::production_launcher_reports_native_and_all_from_literal_transport_filesReal launcher, native and all roles
launched_handoff::production_launcher_rejects_both_image_defects_and_digest_driftImage swap and mutation
launched_handoff::production_launcher_rejects_structural_collector_with_missing_loaderFailure during bootstrap; loader omission
physical_reports::all_three_roles_settle_and_tool_failure_is_an_ordinary_exit_one_reportOrdinary failed-check report at exit one
physical_reports::closed_report_sink_fails_after_collection_without_successful_deliveryOutput overflow and closed report sink
nonchild::nonchild_pidfd_rejects_without_killing_or_stopping_the_owned_sentinelDescendant survival; no foreign signal
real_launched_handoff::production_launcher_reports_all_roles_from_provisioned_real_distributionsReal packaged Clang, Node and Rust roles

cargo test --locked --offline -p semaprax-native-rust-interop-platform-sys --lib

TestCase the issue enumerates
doctor::offline_worker::tests::provisioned_materializer_exec_and_socket_denialForbidden process and network routes
doctor::offline_worker::tests::provisioned_overflow_and_timeout_publish_only_settled_failureOutput overflow and timeout settlement
doctor::offline_worker::tests::provisioned_missing_role_bad_hash_and_invalid_request_emit_no_frameRole swap, wrong digest, invalid request
doctor::offline_worker::tests::provisioned_real_clang_node_rust_distributionsReal tool distributions through the worker
doctor::offline_worker::tests::hostile::provisioned_capability_operations_and_process_creation_are_deniedCapability and process-creation denial
doctor::offline_worker::tests::hostile::provisioned_stdin_is_eof_and_nonstandard_descriptors_are_closedDescriptor closure
doctor::offline_worker::tests::hostile::provisioned_root_hides_real_outside_file_and_rejects_write_opensFilesystem confinement
doctor::offline_worker::tests::lifecycle::post_exec_capabilities_and_supervisor_death_are_observed_externallySupervisor/child lifecycle and cancellation
doctor::offline_root::linux::tests::provisioned_detached_root_bytes_modes_and_read_onlyDetached read-only root materialization
doctor::offline_root::linux::tests::provisioned_wrong_page_cost_stops_before_tree_writesPre-effect bounds
doctor::offline_root::linux::tests::provisioned_setup_and_exact_write_failures_return_no_rootFailure during bootstrap
doctor::offline_root::linux::tests::provisioned_metadata_mismatches_feed_actual_admissionMetadata disagreement
doctor::offline_root::linux::tests::provisioned_close_uncertainty_is_fail_stopClose uncertainty is fail-stop

These are the existing hostile fixtures. The gate substitutes no weaker smoke test for any of them, and adding a case here means adding it to the owning harness first.

selection_drift parses the #[ignore] inventory of each owning file and fails when the selection and the inventory disagree in either direction, so a new ignored lifecycle case that nobody adds here is a failure rather than a silently narrower gate. --self-test runs that comparison against the real tree and separately proves a synthetic narrowing is detected, so the check cannot pass vacuously against an inventory the parser never found.

Two #[ignore]d functions in the doctor tree are deliberately excluded: doctor::offline_input::create::tests and its executable-fault sibling are private subprocess helpers selected by their own parent tests, not gates. So are the Windows revision-store and owned_npm symlink fixtures, which belong to separately tracked hosts.

selection_drift also walks each suite's owning directory (crates/semaprax-native-rust-interop-platform-sys/src/doctor and crates/semaprax-doctor-collector/tests) for any .rs file containing an #[ignore]d test that is not already named in the checked-in list above or in EXCLUDED_IGNORED_FILES. This closes what was previously a residual gap: the per-file comparison alone only ever looks at files the list already names, so a brand-new file of ignored lifecycle tests would not have been noticed. --self-test proves this walk is clean against the real tree and separately proves it detects a file the list stops naming. The walk still does not guess a new file's module path or suite membership — a human adds it to the list above once it exists, exactly as before.

Evidence

A run writes one JSON document, semaprax.doctor.provisioned-linux-gate.v1, binding:

  • probe.revision: the checked-out 40-digit commit and whether the tree was clean;
  • probe.host: system, architecture, pointer width, byte order, kernel release and version;
  • probe.kernel_features and probe.kernel_feature_basis: each required feature and how it was decided;
  • probe.cgroup: the delegated scope's path, filesystem, controllers, control files, writability, populated value and membership before the run;
  • probe.images: each held image's absolute path, size, SHA-256, ELF machine, static/PT_INTERP status;
  • probe.release: the unpacked release root, whether it lies outside the checkout, archive/manifest/capsule digests, the build commit, and the trust anchor;
  • probe.fixture: the real bundle's path, SHA-256 and selector;
  • probe.tools: exact cargo -Vv and rustc -vV output and resolved paths;
  • suites[]: the exact argv, the expected test list, exit code, elapsed time, and a bounded capture of stdout and stderr with byte count, SHA-256 of the full stream, and an explicit truncated flag;
  • final_cgroup: the same scope reread after the run, proving populated 0 and no surviving members;
  • failures[] and selected_failure: every reason in order, with the first one selected.

Failure selection is sticky. Cleanup and settlement observations are appended after execution and can never replace the first selected failure, so a run that failed while executing and also failed to prove an empty cgroup still reports the execution failure as its verdict.

What a maintainer must do to execute this gate

  1. Provision one disposable, trusted Linux x86-64 host satisfying every row of host provisioning. Run scripts/doctor-provisioned-linux-provision.sh, which establishes the private namespace and the empty delegated cgroup-v2 scope, and confirm its unproven Linux behaviour on that host.
  2. Build the current-head launcher, worker, collector and provisioner with the test-only SEMAPRAX_DOCTOR_RELEASE_PUBLIC_KEY_HEX anchor, package them with scripts/package-doctor-release.sh, and unpack the archive outside the checkout.
  3. Acquire dependencies once with cargo fetch --locked; both suites then run --locked --offline, so the gate performs no build-time network access.
  4. Or let .github/workflows/doctor-provisioned-linux.yml do all of the above in-job on a GitHub-hosted ephemeral ubuntu-24.04 runner, which is what the recorded runs use. A self-hosted runner is not required, and a runner label is not attestation in either case.
  5. Dispatch .github/workflows/doctor-provisioned-linux.yml, or run python3 scripts/doctor-provisioned-linux-gate.py --evidence <path> under the wrapper directly.
  6. Record the run link, commit and evidence digest in DOCTOR-PRODUCTION-PROVISIONER-V1.md and only then consider a WP-05 status change in COMPLETION-MATRIX.md.

Nonclaims

This gate does not activate ordinary semaprax doctor --profile selection, wire the private provisioner into any CLI route, prove host-wide network silence, trust the kernel, LSM or VM, authenticate an arbitrary build host, or support AArch64, Windows or macOS. It grants no promotion by existing. Until a run is recorded, every claim about the private Linux doctor boundary's runtime behaviour stays exactly where the owning contract left it: unrun and unpromoted.