Test Results

September 2, 2026 ยท View on GitHub

Read this when you are:

  • adding or extending a result format;
  • changing how failed cases are summarized or capped;
  • debugging why crabbox results shows no data.

Crabbox can attach a parsed JUnit XML summary to a recorded run so a failed run can answer "which tests failed?" without scraping a large raw log. The CLI reads the remote JUnit files after your command exits, parses them locally, and sends only the compact summary to the coordinator. Raw XML is never uploaded or stored.

This page is the conceptual companion to the command reference; for invocation, flags, and example output see results.

Pointing a run at result files

Results attach only when crabbox run knows where to find remote JUnit XML in the workdir. There are three ways to tell it.

Per run, on the command line:

crabbox run --id cbx_... --junit junit.xml -- go test ./...
crabbox run --id cbx_... --junit junit.xml,reports/junit.xml -- go test ./...
crabbox run --id cbx_... --results-auto -- go test ./...
crabbox run --id cbx_... --junit junit.xml --fail-on-test-failures -- ./test-wrapper

Per repo, in any config file:

results:
  auto: true
  failOnFailures: true
  junit:
    - junit.xml
    - reports/junit.xml

Or through the environment: CRABBOX_RESULTS_JUNIT (comma-separated paths), CRABBOX_RESULTS_AUTO (boolean), and CRABBOX_RESULTS_FAIL_ON_FAILURES (boolean). The --junit path also flows through crabbox job run (from a job's junit: list) and crabbox capsule replay.

In layered YAML, omitting results.junit inherits the lower layer's paths, results: {junit: []} clears them, and a nonempty list replaces them. Clearing the list removes old explicit collection paths but does not disable independent results.auto: true discovery. Set auto: false too to stop both forms of collection. A higher CRABBOX_RESULTS_JUNIT override or explicit --junit can select paths again after a YAML clear.

What happens after the command exits

crabbox run collects results only when --results-auto is set or at least one explicit --junit path is configured.

For explicit --junit paths, the CLI resolves each listed file and reads it only when its final target remains inside the workdir, then parses it. Relative paths, absolute paths within the workdir, and symlinks that stay within the workdir remain supported. Windows-native targets apply the same rule while resolving directory junctions and symbolic links. Collection validates the opened file and reads from the same descriptor or stream, so a background process cannot swap a checked path before the read.

Auto discovery (--results-auto) is freshness-aware so it never reports stale reports from an earlier run:

  1. Before the command runs, the CLI writes a results-start marker. In a Git checkout it lives under the repo's Git dir (so the worktree stays clean); otherwise it falls back to .crabbox/results-start in the workdir.
  2. After the command, it walks the workdir for junit*.xml, TEST-*.xml, and results.xml, pruning node_modules and .git.
  3. Each candidate must be newer than the marker, must sniff as JUnit XML (the leading bytes contain <testsuite/<testsuites), and reports that contain failures or errors are prioritized over passing ones.
  4. Collection considers at most 50 files, accepts reports up to 16 MiB each, and transfers at most 64 MiB total. Reports outside those bounds are skipped with a warning naming the file; accepted reports are never truncated.

Explicit --junit files and auto-discovered files are merged (de-duplicated by normalized workdir-relative path), so a multi-report setup still produces one result record. Aliases within the explicit list, such as junit.xml, ./junit.xml, and its absolute path inside the workdir, are counted once; the first readable explicit spelling is retained. Native Windows normalizes path separators and workdir-prefix casing while preserving filename case. Distinct lexical paths stay separate even when report contents match or symlinks point to the same file.

A malformed, partial, or oversized report emits a named warning without discarding summaries parsed from other valid files. The CLI prints a one-line summary to stderr and includes every valid parsed summary in the run's finish payload.

Result collection warnings remain non-fatal. To make parsed test failures affect the run status, opt in with --fail-on-test-failures, results.failOnFailures: true, or CRABBOX_RESULTS_FAIL_ON_FAILURES=true. When the wrapped command exits zero and a valid report contains failures or errors, Crabbox records and exits with code 1 after collecting requested artifacts. An existing non-zero command exit remains authoritative.

The parsed summary

Parsing produces a TestResultSummary (internal/cli/results_parse.go): the format (junit), the source file list, aggregate counters (suites, tests, failures, errors, skipped, total time in seconds), and a failed list of individual failing cases. Each TestFailure records its suite, test name, optional classname/file, the first failure or error message, the JUnit type, and a kind of failure or error. The parser accepts both <testsuites> and a bare <testsuite> root, including nested <testsuite> children. Failed cases retain the name of their owning suite, not an enclosing aggregate suite. The suite count includes every visited <testsuite> element, including empty and aggregate suites; the <testsuites> wrapper itself does not count as a suite.

Each suite combines its direct cases and child-suite summaries before applying its own counters. A nonzero reported test count is retained; a missing or zero count is derived. Failures, errors, and skips are at least the corresponding direct-case plus child-summary totals, so a reported zero cannot hide a failed case. Parent aggregate counters are not added again to their descendants. As with flat reports, a positive suite time is used when the suite reports a positive counter; otherwise time is derived from direct cases and child summaries, never added on top of them. A <testsuites> wrapper uses its child summaries when present, or its own aggregate attributes when it has no suites.

Coordinator storage limits

The coordinator bounds the stored record so a huge report cannot blow past Durable Object storage or slow the run and lease detail pages (worker/src/fleet.ts, boundedTestResults):

  • aggregate counters are kept verbatim;
  • the failed-case list is capped at 100 entries;
  • the file list is capped at 50 paths;
  • every stored string (file, suite, test, message, type) is truncated to 4096 bytes.

Reading results back

crabbox history --lease cbx_...
crabbox results run_...

crabbox results reads the stored summary from the recorded run, so a coordinator must be configured. It is distinct from crabbox logs: results is the structured pass/fail summary, logs is the retained command output.

Human result lines, shard failure summaries, and failure digests visibly escape terminal controls and Unicode formatting characters in stored failure fields. Machine-readable JSON preserves the stored values unchanged.

Supported formats

  • JUnit XML.

Possible future additions tracked for this feature: Vitest JSON, Go test2json, flaky-test history across runs, and changed-file correlation.

Source

  • CLI command and output: internal/cli/results.go
  • JUnit parsing: internal/cli/results_parse.go
  • Remote collection, auto discovery, freshness marker: internal/cli/results_remote.go
  • Run integration: internal/cli/run.go
  • Config keys and env overrides: internal/cli/config.go
  • Summary types: internal/cli/coordinator.go, worker/src/types.ts
  • Storage bounds: worker/src/fleet.ts