NemoClaw E2E CI

August 9, 2026 · View on GitHub

Direct E2E coverage runs through Vitest.

Interactive TUI targets require expect. The unified workflow installs it before those targets run; local runners must provide it themselves.

  • .github/workflows/e2e.yaml selects the default workflow E2E jobs on each push to main and supports trusted manual dispatches for specific PR head commits. Push runs skip the Jetson nvmap and DGX Spark llama.cpp jobs because their required workflow dispatch flags cannot be set by a push event.
  • .github/workflows/hosted-runner-recovery.yaml evaluates first-attempt failures from approved main workflows and requests one full rerun only when every non-passing job has authenticated GitHub-hosted runner-loss evidence.
  • .github/workflows/e2e-main-retry.yaml evaluates eligible E2E main push attempts, requests at most two failed-job reruns, and uploads attempt evidence.
  • The staging-brev-launchable job in .github/workflows/e2e.yaml validates the baked candidate without installing or copying NemoClaw source.
  • .github/workflows/macos-e2e.yaml, .github/workflows/wsl-e2e.yaml, and .github/workflows/sandbox-images-and-e2e.yaml call focused E2E targets directly. .github/workflows/e2e.yaml selects free-standing jobs, including whatsapp-qr-compact and ollama-auth-proxy.

CI execution shape

Candidate CLI Artifact

The candidate CLI comes from the source commit that an E2E run tests. The generate-matrix job builds it once. The job publishes root dist/ and nemoclaw/dist/shared/ in one content-addressed artifact. The boundary validator derives artifact consumers from jobs that use the pinned preparation action. It excludes generate-matrix and the no-build and trusted-build jobs in E2E_JOB_POLICY. Each selected consumer restores the artifact instead of running npm run build:cli. Each consumer runs the pinned preparation action with build-cli: "false" to install Node.js and project dependencies. The managed-image-protected-runtime qualification does not use this artifact. It builds the CLI from the trusted workflow checkout and never executes or restores the candidate CLI.

Artifact Identity

For a pull request (PR) run, checkout_sha identifies the candidate source commit. The trusted workflow runs from github.workflow_sha. A push or manual run uses github.sha when checkout_sha is empty.

The artifact manifest records these values:

  • The candidate repository and commit SHA.
  • The trusted workflow SHA, run ID, and attempt.
  • The source tree and lockfile digests.
  • The Node.js and npm versions, runner platform, and build command.
  • The payload digest.

The artifact name contains the candidate commit SHA and payload SHA-256 digest. The generate-matrix job emits one nemoclaw-e2e-cli-provenance-v1 JSON object through its cli_artifact_provenance output. Each artifact-using job passes that object as the restore action's only provenance-json input.

Each artifact-using job invokes the repository-owned restore-e2e-cli-artifact composite action at a full commit SHA. The workflow does not load the action implementation from the candidate checkout. Before download, the action rejects extra or missing provenance fields. The action requires the candidate checkout SHA, repository, workflow SHA, and run ID to match the provenance object. The producer attempt must not be newer than the consumer attempt. The action downloads the artifact by immutable ID and sets digest mismatch handling to error.

Before the action restores root dist/ and nemoclaw/dist/shared/ into the workspace, it verifies these conditions:

  • The upload digest is present and well formed.
  • The candidate SHA matches the expected commit.
  • The manifest matches the source, workflow run, toolchain contract, and payload.
  • The archive contains no path traversal, links, special files, or files outside root dist/ and nemoclaw/dist/shared/.
  • Neither root dist/ nor nemoclaw/dist/ already exists, including as a dangling symbolic link.
  • The candidate checkout's nemoclaw/ path is a directory and is not a symbolic link.
  • The CLI entry point and required shared modules are nonempty regular files.
  • The staged dist/build-identity.json names the candidate commit SHA.

If a pre-restore check fails, the action stops before it adds either directory to the workspace. After the checks pass, the action restores root dist/ and nemoclaw/dist/shared/, then runs bin/nemoclaw.js --version. If the version command fails, the action stops before the live test runs. This boundary keeps candidate source separate from the trusted workflow implementation.

Timing Baseline

The pre-change baseline uses GitHub Actions Build CLI step timings from these workflow runs:

Workflow runJobTested candidateBuild CLI duration
30574154335cloud-inference385f59818.740 seconds
30574154335cloud-onboard385f59818.793 seconds
30503498077Shared E2E (vllm-docker-storage)d52d45918.756 seconds

The three observed build steps have a median duration of 18.756 seconds. This baseline measures only the replaced build step. Artifact upload, download, validation, and the dependency on generate-matrix add runtime and can affect the workflow critical path. Do not use the build-step median to claim savings in runner time or workflow elapsed time.

A manual PR E2E run tests candidate code but executes .github/workflows/e2e.yaml from main. The PR run cannot measure this workflow change before merge. After merge, use a passing main run and complete these steps:

  1. Match the job selection, runner labels, and first attempt to the baseline.
  2. Record durations for the candidate build, artifact upload, artifact download, combined verification and restore step, job, and workflow.
  3. Sum affected step durations for runner-time comparison.
  4. Compare matched job and workflow elapsed times.
  5. Identify each result by workflow run, tested commit SHA, trusted workflow SHA, and attempt.

Do not substitute a theoretical value for post-change CI evidence.

Historical Fixtures

The historical fixtures retain these version boundaries:

FixtureRequired boundary
openshell-gateway-upgradeRetain the historical installer commit and SHA-256 digest, sandbox image digest, and reviewed OpenClaw npm URL and SHA-512 integrity. Install the historical package before testing the candidate upgrade path.
rebuild-openclawRetain the reviewed old-base build in the target. Build and create the old sandbox before testing the candidate rebuild path.

These targets may restore the shared artifact for the candidate CLI. They must not replace a historical installer, package, image, or version boundary with that artifact. The gateway fixture already binds its remote historical inputs to immutable commits and cryptographic digests. The workflow does not republish those inputs as artifacts.

Hermes Sandbox Image Artifact

The sandbox image workflow builds the Hermes production image in the dedicated 30-minute build-hermes-sandbox-image job. It uses full-SHA-pinned Buildx actions and a GitHub Actions cache scoped to the runner OS and architecture. The producer adds a bounded 32 GiB swap file and validates the guarded production build arguments before the build. It loads the image locally with registry writes disabled. After the build, it scans the completed image for node-tar and verifies the sandbox-readable installed files. It then uploads the compressed image as the one-day hermes-isolation-image artifact.

The 90-minute test-hermes-sandbox-image job and the state-dir-guard-metadata job download and load that artifact instead of rebuilding the image. Within the Hermes test job, the secret-boundary and root-entrypoint steps have 45- and 30-minute budgets respectively.

The former top-level test/e2e/test-*.sh suite has been removed. Keep real shell, installer, process, Docker, OpenShell, /proc, and sandbox boundaries in E2E tests when those boundaries are the behavior under test.

Platform Vitest main watch

.github/workflows/platform-vitest-main.yaml runs the full Vitest suite in four independent shards on each of macOS and WSL, with fail-fast disabled. Each macOS shard has a 30-minute budget and each WSL shard has a 90-minute budget. The additional root-required WSL contracts run only on shard 1.

Retired Brev source-install coverage

Issue #7490 retired the generic Brev source-install lane. The unified workflow and exact-staging Launchable job own its product coverage:

Legacy suiteDispositionCurrent owner
fullLaunchable E2Estaging-brev-launchable runs full-e2e in preinstalled mode against the exact baked candidate.
credential-sanitizationUnified E2Ecredential-sanitization
telegram-injectionUnified E2Etelegram-injection
messaging-providersUnified E2Emessaging-providers
messaging-compatible-endpointUnified E2Emessaging-compatible-endpoint
dashboard-remote-bindUnified E2Edashboard-remote-bind owns install, onboard, artifacts, and terminal cleanup.
gpuUnified E2Egpu-e2e runs on the dedicated GPU runner.
allRetiredThe selector only duplicated credential-sanitization and telegram-injection.

The retired nightly caller no longer runs. Each push to main starts a workflow that selects the default workflow E2E jobs. Manual GPU validation must use gpu-e2e. It must not provision a generic Brev VM.

Credential-free tests

Credential-free tests that can use the standard Ubuntu runner, CLI build, and artifact policy opt into the shared E2E job with a tag beside the test:

// @module-tag e2e/credential-free

Discovery reads tagged files from the e2e-live and integration Vitest projects. It derives each test ID from the filename and supplies only the ID, repository-relative file, and Vitest project to the test matrix. Keep the filename stem unique and lowercase kebab-case. Do not add the test to a separate catalog or manually maintained workflow matrix.

The E2E workflow owns the shared job's runner, timeout, setup, permissions, secrets, and artifact handling. Keep a dedicated workflow job when a test needs different capabilities, such as credentials, a custom runner, additional setup, or a different timeout.

Both jobs and targets selectors continue to accept the test ID. Run the discovery command locally to inspect the generated test matrix:

npx tsx tools/e2e/credential-free-tests.mts

Inactive Windows MXC OpenClaw qualification

windows-mxc-openclaw-process-container.test.ts is an explicit local qualification target for epic #8178. It exercises an operator-supplied native Windows OpenShell package and a staged OpenClaw artifact through the OpenShell process_container driver. It does not register MXC, call wxc-exec.exe directly, or establish Windows support. The generated driver configuration requests the stricter less-privileged AppContainer mode and records that choice in the receipt.

The target requires a Windows x64 host that passes the minimum MXC candidate check. It rejects a dirty NemoClaw checkout and requires exact expected identities for that checkout, the OpenShell CLI and gateway, the OpenShell-supplied wxc-exec.exe, the complete OpenClaw artifact tree, Node.js, and the OpenClaw entrypoint. Compute the canonical artifact-tree digest after staging:

npx tsx tools/e2e/windows-mxc-openclaw-artifact-tree.mts $env:NEMOCLAW_WINDOWS_MXC_OPENCLAW_ROOT

Set the following environment variables to paths or exact lowercase identity values. Do not put credentials in them.

VariableMeaning
E2E_ARTIFACT_DIRExisting directory for the secret-free qualification receipt
NEMOCLAW_E2E_EXPECTED_SHAExact 40-character NemoClaw checkout revision
NEMOCLAW_WINDOWS_MXC_OPENSHELL_CLIExtracted openshell.exe path
NEMOCLAW_WINDOWS_MXC_OPENSHELL_GATEWAYExtracted openshell-gateway.exe path
NEMOCLAW_WINDOWS_MXC_WXC_EXECwxc-exec.exe supplied for that OpenShell package
NEMOCLAW_WINDOWS_MXC_OPENSHELL_VERSIONExact OpenShell package version
NEMOCLAW_WINDOWS_MXC_OPENSHELL_REVISIONExact 40-character OpenShell source revision
NEMOCLAW_WINDOWS_MXC_OPENSHELL_CLI_SHA256Expected OpenShell CLI SHA-256
NEMOCLAW_WINDOWS_MXC_OPENSHELL_GATEWAY_SHA256Expected OpenShell gateway SHA-256
NEMOCLAW_WINDOWS_MXC_WXC_EXEC_SHA256Expected wxc-exec.exe SHA-256
NEMOCLAW_WINDOWS_MXC_OPENCLAW_ROOTStaged native OpenClaw artifact root
NEMOCLAW_WINDOWS_MXC_NODENode.js executable beneath the artifact root
NEMOCLAW_WINDOWS_MXC_OPENCLAW_ENTRYOpenClaw entrypoint beneath the artifact root
NEMOCLAW_WINDOWS_MXC_OPENCLAW_VERSIONExpected OpenClaw version
NEMOCLAW_WINDOWS_MXC_OPENCLAW_ARTIFACT_TREE_SHA256Expected canonical artifact-tree SHA-256
NEMOCLAW_WINDOWS_MXC_NODE_SHA256Expected Node.js SHA-256
NEMOCLAW_WINDOWS_MXC_OPENCLAW_ENTRY_SHA256Expected OpenClaw entrypoint SHA-256

The target creates a random OpenClaw gateway token for readiness checks. It passes that token through the MXC agent environment; current OpenShell process_container packaging can therefore expose its encoded configuration, including the token, to privileged host process inspection while wxc-exec.exe starts the sandbox. The token is never written to the receipt or supplied in the OpenClaw command arguments, is not reused, and is useful only for the temporary loopback OpenClaw gateway. Cleanup attempts sandbox deletion, stops the recorded OpenClaw process, clears the in-memory environment value, and removes the runtime home, state, configuration, and gateway logs. A direct process-tree termination is an emergency cleanup fallback only. The host-side OpenShell processes receive an allowlist of Windows runtime variables rather than the complete caller environment. Before using a termination fallback, the host binds the process ID to the expected executable, command arguments, and creation time. For OpenClaw, it also validates the probe-parent ancestry. The host rejects a mismatched or reused PID. The fallback uses the taskkill.exe beneath the validated Windows system root. If either the OpenClaw process or OpenShell gateway needs that fallback, the qualification fails. The delete retry and process-termination paths are failure containment, not compatibility workarounds that permit a passing result; their presence does not assume a specific upstream defect. Remove them only when failed or partial OpenShell lifecycle operations can still guarantee teardown without host-side cleanup.

Run only the explicit target:

$env:NEMOCLAW_RUN_LIVE_E2E = "1"
$env:NEMOCLAW_RUN_WINDOWS_MXC_OPENCLAW_E2E = "1"
npx vitest run --project e2e-live test/e2e/live/windows-mxc-openclaw-process-container.test.ts

The target verifies OpenClaw startup and in-sandbox health, read-write and denied filesystem behavior, registry cleanup, and termination of the recorded OpenClaw process on sandbox delete. After preflight and local setup succeed, it writes a secret-free receipt for either verdict and records whether sensitive runtime artifacts were removed. When that cleanup succeeds, a failed run retains only non-sensitive probe files for diagnosis. Gateway mTLS, governed egress, managed inference, gateway-restart recovery, and production activation remain outside this target.

The retired hermes-dashboard selector remains a compatibility alias for hermes-e2e in both selector inputs. Reports use the canonical hermes-e2e name. That lane always enables dashboard coverage while preserving the manually selected mock, internal-nvidia, or public-nvidia inference mode.

Current OpenClaw plugin EXDEV lifecycle

The openclaw-plugin-runtime-exdev job keeps one current-version lifecycle:

  1. Onboard the custom weather plugin as v1.
  2. Restart the gateway and verify v1.
  3. Recreate the sandbox with the plugin changed to v2.
  4. Run the cross-device runtime-dependency replacement probe.

The recreation remains the replacement boundary. It verifies the v2 plugin with runtime inspection, tools.catalog, and tools.invoke, and it preserves the workspace marker. The job also keeps the test-only tmpfs mount, unchanged stock policy-source bytes, and the distinct-device and source-side EXDEV checks. The duplicate v3 rebuild is removed from this job. The rebuild-openclaw job remains the canonical live rebuild coverage.

The runtime target for openclaw-plugin-runtime-exdev is 16–17 minutes. Push-run timing for the reduced lifecycle has not yet been measured.

Larger-runner routing

The larger-runner experiment is inactive while the configuration variable E2E_LARGER_RUNNER_LABEL is unset. In that state, every eligible lane continues to use ubuntu-latest. The trusted generate-matrix job builds one runner map before checking out test code, and it consumes the variable only when the workflow repository is NVIDIA/NemoClaw, the ref is refs/heads/main, and no alternate checkout SHA is requested. Manual PR E2E dispatches therefore remain on standard runners even though they use the trusted workflow definition from main.

Manual PR E2E dispatches and direct push or manual main runs use a bounded swap fallback for eligible hosted Hermes image-building lanes. The fallback does not change runner routing. The trusted workflow provisions the fallback as the first job step, before checking out or executing the selected revision. Manual PR E2E requires a maintainer-supplied lowercase 40-character checkout SHA plus matching trusted workflow and dispatch revisions. Direct-main mode rejects alternate checkout and workflow revisions and requires the workflow source to match the run revision. Both modes require an ephemeral GitHub-hosted Linux x64 runner. Candidate code cannot supply the program or arguments passed to sudo.

The trusted step requires at least 32 GiB (34,359,738,368 bytes) of usable swap. It reuses active swap that meets this requirement. Otherwise, it preserves at least 16 GiB of available disk capacity under /mnt, creates a root-owned mode-0700 directory, and creates an exclusive randomized mode-0600 file. The file allocation is 32 GiB plus 4,096 bytes (34,359,742,464 bytes). The additional 4,096 bytes keep the usable swap capacity at or above 32 GiB after formatting. Setup failure stops before candidate checkout and removes partial state only after proving the file inactive or successfully disabling it. After swapon succeeds, the trusted step makes up to five activation observations, one second apart. If visibility remains stale, cleanup treats the file as active. Cleanup removes it only after swapoff succeeds. Successful state is discarded with the ephemeral runner.

The fallback covers agent-turn latency, Hermes inference switch and shields, the Hermes stable MCP shard, the Hermes common-egress and channel stop/start shards, the dashboard-bearing hermes-e2e lane, hermes-discord, and Hermes security-posture tests. Rebuild lanes with workflow-managed swap, dedicated-runner lanes, mcp-bridge-dev, and non-Hermes shards do not use it. Candidate-authored workflow definitions and fork-owned runs cannot reach it.

The fallback exists because the alternate-checkout trust boundary deliberately keeps PR-authored code from selecting the administrator-managed larger-runner label; changing the PR checkout cannot safely grant itself that capacity. Remove the fallback only after trusted main and manual PR E2E runs use ephemeral GitHub-hosted runners with at least 32 GB RAM without weakening the source guards, and five consecutive runs of every protected lane complete without runner loss while runner-pressure telemetry reports less than 1 GiB of swap used.

The eligible set is limited to the measured or repeatedly interrupted heavy lanes:

  • common-egress-agent;
  • hermes-e2e, including dashboard coverage, and hermes-discord;
  • the Anthropic-compatible hermes-inference-switch mode;
  • hermes-shields-config;
  • the Hermes shards of security-posture and channels-stop-start;
  • rebuild-hermes;
  • rebuild-hermes-stale-base;
  • the hermes and deepagents shards of mcp-bridge.

The OpenClaw shards of the matrix jobs, the openclaw MCP shard, mcp-bridge-dev, and openshell-credential-generation-window remain on ubuntu-latest; unrelated jobs retain their existing runner assignments. The credential-generation window runs as an independent fresh-runner job in parallel with the stable MCP agent matrix. Empty-selector dispatches and explicit mcp-bridge selections run both jobs, while the credential-window job keeps its own exact-release provenance, secret scan, and artifact. Before setting the variable, an organization owner must:

  1. Create a GitHub-hosted Ubuntu x64 larger runner with 8 vCPU, 32 GB RAM, and 300 GB SSD in a dedicated runner group.
  2. Set the group maximum concurrency to 4 and restrict repository access to NVIDIA/NemoClaw and workflow access to NVIDIA/NemoClaw/.github/workflows/e2e.yaml@refs/heads/main.
  3. Record at least five standard-runner samples for each eligible lane, including queue time, execution time, peak CPU, memory and disk use, infrastructure failures, and estimated cost.
  4. Copy the larger runner's workflow label into the repository variable, then repeat the same measurements for at least five representative executions per migrated lane.

Clearing E2E_LARGER_RUNNER_LABEL is the rollback. It sends the eligible lanes back to ubuntu-latest without changing selectors, test setup, or test semantics. Do not replace this experiment with a persistent self-hosted runner; that requires a separate decision.

Push operations

The consolidated workflow keeps its operational reporting in the same job graph as the live targets:

  • GitHub Actions run history is the authoritative record for push and manual E2E results.
  • Automated issue routing and the workflow's issues: write capability are retired. Any future issue escalation should use a separately reviewed exceptional threshold, such as the same lane failing twice consecutively or remaining broken for 24 hours, rather than posting on every failed schedule.
  • scorecard writes the push/manual result summary and posts it to the daily or full-run Slack route. The summary:
    • separates queue time from execution time for the ten jobs with the longest combined duration;
    • reports the runner class as standard, larger, or unknown without exposing runner labels;
    • adds this run's semantic phase runtime table;
    • compares each of the ten slowest current tests with up to ten prior completed push runs; and
    • compares the trusted cloud-onboard timing summary with the latest prior-release e2e.yaml run.
  • The push comparison reads only validated e2e-runtime-summary.json artifacts retained for 14 days. Manual runs can display the comparison but never enter its baseline. The table reports the current outcome, prior median and p95, prior pass/fail/skip counts and rates, the failure streak including the current run, and the most common failed phase across the compared runs. It marks a total or phase regression only when the current duration exceeds the prior median by both at least 20% and at least 30 seconds.
  • A separate flake watch shows at most five current tests that both passed and failed across the current run and up to ten prior completed push runs. It ranks them by pass/fail flips and then failure count. It reports pass/fail/skip counts, failure rate, pass/fail flips, current failure streak, and the most common failed phase. The failure-rate denominator and pass/fail-flip count exclude skips.
  • Selective dispatches remain silent unless they run on main with post_to_slack=true, which uses the preview Slack route. Branch-dispatched runs never receive Slack webhook secrets.

A manual run with jobs=staging-brev-launchable runs only Exact staging Brev Launchable. Each push run also selects this job as part of the complete main run.

A manual run with include_staging_brev_launchable=true and empty jobs and targets selectors runs the default workflow E2E selection plus the Launchable E2E job. This is the full run required for pre-tag evidence. Each full dispatch uses github.run_id in its workflow concurrency identity, so another full dispatch cannot supersede it while it waits. The trusted main workflow dispatch verifies that the dispatching and rerunning actors have repository maintain or admin permission before the Launchable path's source checkout. That automatic role check authorizes staging-brev-launchable; the job does not use GitHub environment approval. The job uses the non-cancelling staging-brev-launchable-cpu group with queue: max, so pending Launchable E2E runs remain queued instead of replacing one another.

The Jetson nvmap and DGX Spark llama.cpp jobs remain excluded from ordinary and full runs unless their independent runner-queue flags are true. A user permitted to dispatch this workflow may set either flag, but only after a repository administrator confirms the corresponding runner is online in the authoritative repository runner inventory. Set allow_jetson_runner_queue=true to select jetson-nvmap-gpu. Set allow_dgx_spark_runner_queue=true to select both llama-cpp-dgx-spark-plan and llama-cpp-dgx-spark-qualification. GitHub can pause the qualification job for the approve-dgx-spark-image-qualification environment before it reaches the DGX Spark runner. Pre-tag evidence requires both runner-queue flags to remain false. Results from opt-in hardware runs do not enter the required pre-tag E2E denominator.

Hosted-Runner Recovery

Hosted Runner Recovery can request one full rerun for eligible WSL, macOS, and platform-watch push runs. It does not handle E2E main. The complete non-passing job listing must contain only authenticated hosted-runner-loss evidence for the workflow's approved runner labels. An ordinary assertion failure, mixed failure set, incomplete listing, custom or self-hosted label, changed evidence, or ambiguous pagination prevents recovery.

For eligible E2E main push runs, E2E / Main Retry asks GitHub Actions to rerun failed jobs and their dependent jobs. A successful CLI artifact producer is not rerun. The workflow retains its CLI artifact for 3 days. During that period, consumers can reuse the immutable, content-addressed artifact from an earlier producer attempt in the same workflow run. If the artifact is unavailable when a consumer downloads it, restoration fails because the failed-job rerun does not rerun the successful producer. Restore validation binds the producer provenance to the workflow run, workflow SHA, and candidate checkout. It downloads by immutable artifact ID and verifies the manifest and the payload digest. It rejects a producer attempt newer than the consumer attempt. The controller can request two reruns. It does not verify that GitHub schedules a different runner, so do not treat a rerun as evidence of a fresh host. It ignores manual runs and source runs superseded by a newer main push. The controller checks out only trusted default-branch code and receives no repository secrets.

The runner-allocation and internal-error failures handled by Hosted Runner Recovery originate in GitHub Actions, outside repository-controlled workflow code. Hosted Runner Recovery contains these failures without claiming to repair their source. Remove .github/workflows/hosted-runner-recovery.yaml and its controller only after the three platform workflows record 30 consecutive days with no first-attempt failure accepted by the recovery classifier, or after those workflows stop using GitHub-hosted runners. Each accepted Hosted Runner Recovery request resets that observation window.

Runner comparison telemetry

Trusted main runs without an alternate checkout SHA record runner-comparison telemetry for 12 routed workflow lane identities / 14 concrete job executions.

  • agent-turn-latency, spanning its sequential OpenClaw and Hermes setup
  • common-egress-agent with the openclaw-balanced-weather, openclaw-open-reference, and hermes-open-reference shards
  • rebuild-hermes
  • rebuild-hermes-stale-base
  • mcp-bridge with the hermes shard
  • mcp-bridge with the deepagents shard
  • channels-stop-start with the hermes shard
  • hermes-discord
  • hermes-e2e, including dashboard coverage
  • hermes-inference-switch with the anthropic mode
  • hermes-shields-config
  • security-posture with the hermes shard

The two extra executions come from common-egress-agent, which runs three scenario shards. The OpenClaw matrix entries for mcp-bridge, channels-stop-start, and security-posture are not instrumented. The #7145 standard-versus-larger-runner cohort compares the same lane and equivalent workload while varying the runner class. The newly instrumented agent-turn-latency extends diagnostic coverage; this does not route it to a larger runner.

Each execution writes one bounded, ordered v2 time series to the canonical runner-comparison.jsonl ledger. It contains:

  • an initialize endpoint after exact-commit artifact restoration; the rebuild jobs initialize after their fixed-capacity swap;
  • a distinct scenario-start for every test handled by the execution;
  • a periodic sample on an approximately 15-second fixed cadence for rebuild-hermes and rebuild-hermes-stale-base, and an approximately 60-second fixed cadence for every other execution;
  • a phase sample before each semantic phase transition and when the final phase stops; and
  • a finalize endpoint from an always() step immediately before artifact checking and upload.

The progress pulse owns both stall reporting and periodic comparison sampling, so it never creates a second timer. Phase samples that cross a periodic deadline consume that slot, and delayed probes skip missed slots instead of producing a catch-up burst. Each successful append also prints one bounded E2E_RUNNER_COMPARISON_SAMPLE line in the job log.

The v2 ledger accepts at most 256 samples. Ordinary sampling stops once 255 records exist to reserve the last slot for finalize. A missing, historical-v1, already-finalized, full, or invalid ledger permanently disables comparison sampling for that test progress instance. The two Hermes rebuild lanes use their shorter cadence to improve Docker/BuildKit peak-RSS evidence without changing the ledger bound, schema, privacy contract, or reserved final slot. In rebuild-hermes and rebuild-hermes-stale-base, where legacy phase resource evidence is configured, the workflow establishes its 32 GiB swap before initialize so the ledger sees one stable swap capacity. If canonical sampling becomes unavailable, the existing five-minute full snapshot becomes the best-effort fallback. That full profile may run ps, docker stats, and docker system df sequentially with a 15-second timeout each, or 45 seconds in the worst case; canonical sampling suppresses this heavier collection while it remains active. Other lanes stop canonical sampling without creating a second evidence stream. Historical v1 ledgers and summaries remain readable, but a v1 ledger cannot be extended or mixed with v2 samples.

Probe cost depends on the sample kind. initialize and finalize read only kernel and filesystem sources and launch no child process. periodic adds one one-second ps probe and does not call Docker. scenario-start and phase samples add the same bounded process probe plus two-second docker stats and docker system df probes. The emitted schema contains only numeric fields, fixed process classes (docker-buildkit, openshell, or other), and fixed sample metadata, including the explicit target and shard labels. It never records process or container names, command lines, child output, or arbitrary environment and secret values. Docker memory evidence is reduced to the largest retained container value; maximum Docker CPU considers every row in the bounded command output. When the globally largest process is in the Docker/BuildKit class, the collector also reads that process's VmRSS, RssAnon, RssFile, RssShmem, and optional VmSwap values from procfs. PID and exact process identity remain private to the collector, and the breakdown is null if the process exits, its identity changes, procfs denies access, or the resident components are incomplete or inconsistent. The outer rssKb is the ps selection and ranking observation; breakdown.vmRssKb is the immediately following procfs observation and may differ when a live process changes memory.

The finalizer validates the complete ledger before writing runner-comparison-summary.json. The v2 summary reports the sampled window from initialize until immediately before artifact scanning or upload. Initialization follows artifact restoration and any required rebuild swap. For the Hermes security-posture shard, the window includes OpenShell installation and installer-backed NemoClaw setup, but not workspace preparation or artifact restoration. The summary reports CPU average and busiest interval; one-minute load; available, cached, reclaimable, swap, root-cgroup current/peak/limit, and endpoint OOM-counter evidence; memory and I/O pressure; workspace bytes and inodes; Docker image, container, and build-cache usage; largest container memory and CPU; and the largest fixed process class by RSS. Extrema include the semantic phase where they were observed when attribution is sound. CPU intervals ending at a scenario-start remain unattributed because they can span two tests, and extrema whose selected observation is initialize have a null phase. OOM deltas are also null unless both endpoint counters are available. Unsupported or unreadable measurements are null.

The largest-process summary carries the breakdown from the same sample that provided the maximum total RSS; it does not combine per-component maxima from different samples. Treat this as an approximate breakdown of one process's RSS, not as a Docker/BuildKit process-tree working set: file-backed RSS can count shared mappings in more than one process, and this evidence excludes host page cache and sibling processes.

The root-cgroup peak is a lifetime counter that includes Docker siblings but can also include host activity before the measured window. Compare it only across runs with the same runner setup. Canonical v2 memory.availableKb comes only from /proc/meminfo MemAvailable and is null when that field is unavailable. Separately, the adjacent progress/stall resource line falls back to the portable free-memory value and labels that value as memory free.

The comparison time series is diagnostic-only and is not an input to terminal classification or retry policy. Runner-comparison telemetry does not affect E2E / Main Retry decisions. Hosted Runner Recovery remains limited to authenticated runner-loss evidence for its three platform workflows.

Treat a missing summary as unavailable evidence, not as low utilization. A hard runner loss can prevent finalization or artifact upload. When you compare standard and larger runners, use runs with the same commit SHA, workflow inputs, target, and shard. Pair the artifact with the GitHub Actions runner label, queue time, result, and usage or cost metadata. The ledger is a time series for one execution only; this telemetry does not maintain cross-run rolling history or write to the GitHub Actions step summary. Both output files are private regular files on the runner (0600) with strict per-line and total size limits.

Raw cloud-onboard traces stay under the runner temporary directory. Before artifact upload, scripts/e2e/sanitize-trace-timing.py reduces them to the allowlisted cloud-onboard-trace-timing-summary.json timing schema and deletes the raw directory. Aggregation ratchets require report-to-pr and scorecard to wait for the same execution-job set.

Registry-driven Vitest targets also enable onboard trace collection. Each live matrix target writes raw traces under the runner temporary directory, sanitizes them before upload, deletes the raw trace directory, and uploads only e2e-artifacts/live/<target>/cloud-onboard-trace-timing-summary.json with the target artifact. These per-target summaries are artifact evidence only; the Slack/GitHub scorecard comparison remains tied to the dedicated cloud-onboard artifact so baseline aggregation stays stable. Older issue references to Vitest target artifacts under e2e-artifacts/vitest/ map to this consolidated e2e-artifacts/live/ registry-target artifact layout.

Every e2e-live test and every credential-free integration test selected by the shared E2E workflow planner declares an ordered semantic phase plan in meta.e2ePhases and uses its automatic progress fixture. Normal E2E output identifies the workflow target and test scenario, then shows immediate phase start and completion lines with both phase and total elapsed time. A transition looks like:

[e2e target="cloud-onboard" scenario="onboards a hosted sandbox"] [phase 2/4] completed: onboard the sandbox — passed in 2m 14s (total 2m 21s)
[e2e target="cloud-onboard" scenario="onboards a hosted sandbox"] [phase 3/4] started: verify hosted inference (total 2m 21s; phase 0s)

For e2e-live, the stateful fixture appends release registered E2E resources after the test-declared plan, so the displayed phase count includes that terminal phase. Registered cleanup duration, failures, and stall diagnostics are attributed there. Workflow-selected integration tests instead declare and enter their own final release phase. Soft assertion failures remain attributed to the semantic phase in which they occurred rather than being reassigned to resource release.

If one phase remains active for five minutes, a content-free diagnostic adds the target/scenario identity, total and phase duration, age of the last child output, current redacted command or cleanup activity, and runner resources. It repeats every ten minutes while that same phase remains active. Automatic child output observation forwards only a timestamp and stream name, never contents. Operations with bounded retries may emit immediate content-free progress.event(...) lines for a timeout, cleanup, backoff, or retry; event labels are explicitly logged and must never contain child output, request data, credentials, or tokens.

During fixture teardown, the fixture writes test-progress.json into each test's existing artifact directory for passing and failing tests. The summary keeps the test identity and overall timestamps, plus each recorded phase's timestamps, duration, outcome, child-output event count, and last-output timestamp. It records the target from E2E_TARGET_ID, falling back to the Actions GITHUB_JOB identity, and records NEMOCLAW_E2E_SHARD when set. Compare extracted artifacts from multiple runs with:

npm run test:runtime-audit -- path/to/run-1 path/to/run-2

The audit groups each test by target and optional shard, ranks the groups by p95 runtime, and reports variability plus the slowest observed phase's duration and outcome. Push and ordinary manual runs include the same table for that run in the GitHub Actions scorecard summary. Their push trend uses only the bounded timing and outcome summary rather than downloading historical raw test artifacts. Keep phase labels specific to test behavior, call progress.phase("literal phase label") at the declared boundaries in order, and transition through the final test-declared phase on every passing path. Both fixtures reject a passing test that never reaches that phase; only the stateful live fixture enters its resource-release phase automatically. Validate phase coverage without executing test bodies with:

npm run test:e2e-phases:check

The checker preserves coverage for every file under test/e2e/live/ and adds workflow-selected integration files from the authoritative shared-job planner. Live modules import fixtures/e2e-test.ts; selected integration modules import fixtures/workflow-e2e-test.ts and declare their final release phase explicitly. It also follows shared E2E runtime helpers. Run child processes through ShellProbe or an existing audited progress-aware boundary; new direct async process boundaries fail the check. Synchronous calls require both a positive timeout shorter than the first heartbeat and killSignal: "SIGKILL". Keep child contents in redacted artifacts and report only timestamp-based output activity to the console. Pass the fixture-provided frozen, canonical progress capability unchanged to an audited subprocess boundary; do not replace it with a custom, copied, or no-op adapter.

Push and Manual PR E2E

E2E does not run automatically for pull requests. Pull requests retain deterministic CI, including the e2e-support Vitest project. Each push to main selects the default workflow E2E jobs. The central workflow skips the Jetson nvmap and DGX Spark llama.cpp jobs on push. The central workflow has no scheduled trigger.

The main-push selection includes:

  • the staging Brev Launchable journey;
  • the OpenShell gateway authentication contract;
  • the development MCP bridge;
  • managed-image startup on AMD64 and ARM64;
  • managed-image GPU, Ollama, NVIDIA NIM, and vLLM behavior;
  • Hermes GPU startup.

These jobs retain their runner, credential, evidence, and cleanup boundaries. A main push can queue repository-owned GPU runners and can create Brev resources. The retry workflow reruns failed jobs at most twice.

Each trusted push to main selects Exact staging Brev Launchable. The job reads these credentials from repository Actions secrets:

  • BREV_API_KEY authenticates the Brev CLI for workspace operations in the organization identified by BREV_ORG_ID.
  • NEMOCLAW_IMAGE_DISPATCH_TOKEN is exposed as GH_TOKEN only to the trusted host script. It grants Actions read/write access to brevdev/nemoclaw-image, which the script uses to dispatch the image workflow, inspect its run, and download its handoff artifact.
  • NVIDIA_INFERENCE_API_KEY is exported into the Brev guest for the full E2E process. Code in the baked candidate checkout can read and use it.

These credentials remain valid until they expire or an administrator revokes them in their issuing services. If cleanup fails, remove the recorded Brev workspace. Rotate or revoke each credential to remove later access.

When an eligible E2E main push workflow concludes with failure, E2E / Main Retry asks GitHub Actions to rerun failed jobs and their dependent jobs. The controller permits two reruns but does not verify that GitHub schedules a different runner. After evaluation succeeds, it uploads an artifact named for the current attempt. The artifact contains one attempts summary for each source attempt through the current attempt. The totalRunnerMinutes field contains the cumulative runner time for those summaries. A later successful attempt sets action to passed-after-retry and flaky to true. The controller does not retry manual PR runs or a run superseded by a newer main push.

For a PR revision run, a repository maintainer or administrator leaves jobs and targets empty. The run selects:

  • every default-selected free-standing workflow E2E except Exact staging Brev Launchable;
  • every shared credential-free test; and
  • these controller-selected registry targets: ubuntu-policy-custom-missing-presets-negative, ubuntu-repo-cloud-langchain-deepagents-code, ubuntu-repo-cloud-openclaw, and ubuntu-repo-docker-post-reboot-recovery.

The run skips jetson-nvmap-gpu, llama-cpp-dgx-spark-plan, and llama-cpp-dgx-spark-qualification unless their separate runner-queue flags are true. The trusted workflow definition remains on main and binds the candidate head to the current PR base SHA. It does not run GitHub's synthetic merge commit.

PR Review Advisor maps changes to either of these shared journaled-recreation handlers to recommended E2E coverage:

  • src/lib/onboard/machine/handlers/sandbox-resume.ts.
  • src/lib/onboard/machine/handlers/sandbox.ts.

The risk plan selects the openshell-gateway-upgrade job and the ubuntu-repo-cloud-langchain-deepagents-code typed target. The job covers the installer-driven OpenShell gateway upgrade handoff. The target covers the LangChain Deep Agents Code sandbox recreation path.

An empty-selector manual run exposes these values to candidate-controlled job processes:

  • Long-lived API keys from repository secrets: NVIDIA_INFERENCE_API_KEY, NVIDIA_API_KEY, and BRAVE_API_KEY.
  • Long-lived messaging credentials from repository secrets: TELEGRAM_BOT_TOKEN_REAL, DISCORD_BOT_TOKEN_REAL, SLACK_BOT_TOKEN_REAL, and SLACK_APP_TOKEN_REAL.
  • The job-scoped GITHUB_TOKEN in the token-rotation and openshell-gateway-upgrade jobs. It has checks: read, contents: read, and pull-requests: read access. Candidate code can use it while either job runs. GitHub Actions invalidates it after the job.
  • Messaging account and channel identifiers from repository secrets: TELEGRAM_ALLOWED_IDS, TELEGRAM_AUTHORIZED_CHAT_IDS, TELEGRAM_CHAT_ID, TELEGRAM_CHAT_ID_E2E, DISCORD_CHANNEL_ID_E2E, and SLACK_CHANNEL_ID_E2E.

The workflow does not rotate or revoke these API keys or messaging credentials. To remove later access, rotate or revoke every listed credential in the external service that issued it. The workflow cannot erase identifiers copied by candidate code. Review the complete candidate diff before dispatch. Live targets can create external resources. After a failure, inspect the workflow artifacts and remove resources that target cleanup did not remove.

For managed-image-protected-runtime, the workflow supplies the long-lived NVIDIA_API_KEY repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as NGC_API_KEY and NIM_NGC_API_KEY to the temporary NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. The live fixture attempts to stop and remove nemoclaw-managed-image-nim-e2e, but Docker stop or removal errors do not fail the test. A surviving container can retain the API key until runner teardown. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Rotate or revoke it in the issuing NVIDIA service to remove later access.

For a manual PR run, provide the current PR number, lowercase 40-character head SHA, head repository, lowercase 40-character base SHA, trusted main workflow SHA, and a review reason containing 10 to 500 printable characters. Leave jobs and targets empty and keep include_staging_brev_launchable=false to use this PR revision selection. Keep allow_jetson_runner_queue=false and allow_dgx_spark_runner_queue=false for the default PR revision selection. If allow_dgx_spark_runner_queue=true, GitHub can pause the qualification job for the approve-dgx-spark-image-qualification environment. An authorized environment reviewer must approve it before qualification starts. To select the protected managed-image runtime qualification, set jobs=managed-image-protected-runtime. Leave targets empty. Keep include_staging_brev_launchable=false. The exact candidate must contain ci/protected-managed-image-multiarch-activation-v1.json and ci/protected-managed-image-runtime-activation-v1.json. The trusted pre-checkout step requires current maintain or admin permission and validates the exact open PR and selected mode before candidate code runs. A second validation after checkout rejects a changed head, base, or repository before preparation.

The Actions run is advisory for the pull request and is not a required merge context. Treat it as passing evidence only when the E2E workflow concludes with success for the recorded PR number, head repository, head SHA, base SHA, and trusted workflow SHA. A changed head repository, head SHA, or base SHA invalidates the evidence and requires a new manual run.

The macOS and WSL workflows also run on configured pushes to main. The portable-profile workflow runs on main when one of its configured paths changes. Their manual dispatch paths remain available for branch diagnosis.

Onboard performance budget

The push/manual scorecard evaluates the trusted cloud-onboard timing summary against ci/onboard-performance-budget.json. The budget covers the warm-system path and is advisory: exceeding the total-duration cap or a regression threshold emits a GitHub Actions warning and adds details to the run summary, but does not fail the scorecard job.

The config separates the absolute total-duration budget from total and phase regression thresholds. Phase regressions are diagnostic and are only compared when the current run and prior-release baseline contain the same known onboard phase names. Cold image pulls, first-time model downloads, provider outages, and runner or network incidents can still affect the signal, so maintainers should inspect the timing table before acting on a warning.

For PRs, the unified PR Review Advisor builds and renders guidance from the deterministic risk plan for the PR SHA and changed-file set. It recommends jobs for known regression families and includes cloud-onboard when changes affect onboard behavior, trace timing, scorecard analysis, budget configuration, or the unified E2E workflow. Compatibility schema fields may classify that guidance as required, but rendered advisor guidance remains non-authoritative. Model advice is additive and cannot downgrade the deterministic floor. PR Review Advisor recommendations remain advisory. A maintainer decides whether to dispatch this trusted selection for the current PR revision. No PR E2E controller dispatches the risk plan.

The full-e2e target enforces a separate hard acceptance contract for the first fresh onboarding path in that job. It measures from the onboard root span (a conservative anchor before wizard step [1/8]) through the first non-empty agent response, requires the local BuildKit prebuild for the NemoClaw-generated context without a gateway-builder fallback, enforces the calibrated root and phase limits in the budget file, and limits the longest onboard output gap to 60 seconds. A violation fails full-e2e, and the target writes its evidence to onboard-progress-budget.json. The artifact records the first-turn command wall clock and OpenClaw's internal agent duration separately. Older or malformed OpenClaw output records an explicit unavailable reason instead of fabricating a duration. The artifact also identifies the model, provider, inference mode, and prompt contract. When every deterministic cold-onboard budget passes and the real first turn exits successfully with the expected sentinel, a sole root-end-to-first-turn overage is recorded as a structured, non-blocking hosted-latency anomaly rather than a PR regression. The same overage remains blocking when accompanied by a root-start or phase-budget failure.

The trusted push scorecard stores the current eligible sample in the e2e-runtime-summary artifact. The scorecard compares only samples with the same agent, provider, model, inference mode, and prompt contract. The recurrence window contains the 12 most recent eligible samples from push main runs. The current anomaly fails the scorecard when the window is full and contains at least one earlier anomaly. A current sample without an anomaly does not fail because of an earlier anomaly. Missing, malformed, or functionally unsuccessful samples do not enter the window. The scorecard waits for 12 eligible samples when retained history is incomplete. The canonical E2E uploader retains each push summary for 14 days.

When changed base-image inputs require the authoritative local OpenClaw base build, the target applies the separately calibrated 90-second allowance only to the root-start and sandbox-phase limits. The installer must emit the exact local base-build reason before the allowance applies. Published-image runs retain the normal limits, and output silence, first-turn, and all other phase requirements remain unchanged.

The two Hermes rebuild jobs and both reusable-workflow Hermes image exporters add a bounded 32 GiB swap file on their ephemeral hosted runners before the memory-heavy image build. The rebuild fixture verifies that floor and provisions the same swap file on GitHub Actions when a trusted control-plane run uses the workflow definition from main. Those paths build large Hermes image layers and can otherwise exhaust the runner's default memory and swap during Docker layer export. Apart from those rebuild and export paths, E2E jobs add swap only through the trusted Hermes main-workflow fallback described in Larger-runner routing.

These assertions run inside the existing full-e2e lifecycle instead of a second standalone onboarding run. This keeps the measurement on the job's first sandbox build, avoids warming Docker layers before a duplicate performance test, and makes full-e2e the source of truth for the hard cold-path contract.