Testing Matrix

June 18, 2026 · View on GitHub

Where each feature is tested, and how to run the manual gates.

See issue #215 for the original audit. See CLAUDE.md for the CI philosophy (remote CI is deliberately lean; deep gates run on laptops and nightly).

Looking for coverage by Dora capability rather than by test tier? See testing-capabilities.md (#1633).

Looking for how much validation a given change needs (Class A / B / C)? See agentic-qa-policy.md (#1634).

Looking for coverage by CLI command source file (binaries/cli/src/command/*)? See cli-command-coverage.md (#1980).

Tier 0 — Remote CI (every push and PR)

Fast gates. Must pass for every PR.

Area	Where	How
Format	`.github/workflows/ci.yml` `fmt`	`cargo fmt --all -- --check`
Clippy	`.github/workflows/ci.yml` `clippy`	`cargo clippy --all -- -D warnings`
Core tests (Linux)	`.github/workflows/ci.yml` `test`	`cargo test --all` (Python crates excluded) on ubuntu-latest
CLI smoke (argparse, `validate`, `expand`, `graph`)	`.github/workflows/ci.yml` `test`	Per-subcommand `--help` + validate/expand/graph on representative YAMLs (ubuntu-latest)
Core tests (macOS + Windows)	`.github/workflows/nightly.yml` `test-cross-platform`	same steps as PR `test`, nightly-gated (#1716)
Example dataflows (Rust/C/C++/Python)	`.github/workflows/nightly.yml` `examples`	`cargo run --example ...` on all 3 platforms, nightly-gated (#1716)
CLI template + basic Python examples	`.github/workflows/nightly.yml` `cli-tests`	`dora new` + `dora run` on all 3 platforms, nightly-gated (#1716)
E2E WS control plane	`.github/workflows/ci.yml` `e2e`	`cargo test --test ws-cli-e2e`
Fault tolerance E2E	`.github/workflows/ci.yml` `e2e`	`cargo test --test fault-tolerance-e2e`
Semantic contract tests (service, action, streaming, validated-pipeline)	`.github/workflows/ci.yml` `contract-tests`	`cargo test -p dora-examples --test example-smoke contract_`
Supply chain (cargo-audit / cargo-deny)	`.github/workflows/ci.yml` `audit`	`scripts/qa/audit.sh`
Unwrap budget	`.github/workflows/ci.yml` `unwrap-budget`	`scripts/qa/unwrap-budget.sh`
Benchmark regression	`.github/workflows/ci.yml` `benchmark-regression`
Typos	`.github/workflows/ci.yml` `typos`	`crate-ci/typos`

Tier 1 — Nightly (`.github/workflows/nightly.yml`)

Runs daily at 06:00 UTC and via workflow_dispatch. Failures file issues with the nightly-regression label but do not block PRs.

The liveness-oriented smoke_* tests in tests/example-smoke.rs stay nightly-only — they catch crash-class regressions and, for the four capabilities with a contract_* sibling (service, action, streaming, validated-pipeline), the contract tests in Tier 0 already provide stronger per-PR coverage. See "Tier policy" below for the promotion criteria used in #1632.

Area	Job
Full smoke suite (`tests/example-smoke.rs`, 45 `smoke_*` tests)	`smoke-suite`
Log-sink examples (file, alert, tcp)	`log-sinks`
Service / Action patterns (Rust-only)	`service-action`
Streaming example (Python nodes)	`streaming`
Record / replay round-trip	`record-replay`
Cluster lifecycle (`cluster status`, `cluster down`)	`cluster-smoke`
Inspection commands (`top --once`, `topic list/info/pub`)	`topic-and-top-smoke`
`cpu_affinity` end-to-end (mask actually applied)	`cpu-affinity-smoke` (Linux only)
redb coordinator store survives restart	`redb-backend-smoke`
Daemon auto-reconnect (SIGSTOP past heartbeat → SIGCONT → re-register)	`daemon-reconnect-smoke` (Linux only)
State reconstruction (partial: store hydrate + Running → Recovering)	`state-reconstruction-smoke`

Run locally:

# Smoke suite
cargo test -p dora-examples --test example-smoke -- --test-threads=1

# Rust-only examples (no Python setup needed)
dora run examples/service-example/dataflow.yml --stop-after 15s
dora run examples/action-example/dataflow.yml --stop-after 20s

# Python-backed examples — need uv venv + local dora install first:
#   uv venv --seed -p 3.12
#   source .venv/bin/activate
#   uv pip install pyarrow
#   uv pip install -e apis/python/node
dora run examples/log-sink-file/dataflow.yml --uv --stop-after 15s
dora run examples/streaming-example/dataflow.yml --uv --stop-after 15s

Tier 2 — Laptop / manual

Not automated — too heavy, platform-specific, or requires dedicated infra. Run before releases or when touching the relevant subsystem.

Deep QA gates

Documented in CLAUDE.md. Invoke via make:

Gate	Command	When
Full QA (fast + tests + coverage)	`make qa-full`	before significant push
Target Tier 1 gate (qa-full + mutants + semver)	`make qa-deep` (alias: `make qa-tier1`)	stronger than today's CI — laptop-only extras (coverage, adversarial, mutants, semver)
Full parity with `.github/workflows/nightly.yml` after #1716 (qa-deep + proptest@1000 + miri + example-smoke + hub-smoke + ci-nightly-jobs)	`make qa-nightly`	overnight runs on a powerful machine; ~3-4 hours. Covers all 19 nightly test jobs: example-smoke = 4 (smoke-suite, log-sinks, service-action, streaming); hub-smoke = 1; ci-nightly-jobs.sh = 14 (record-replay, cluster-smoke, topic-and-top, cpu-affinity [Linux], redb-backend, daemon-reconnect [Linux], state-reconstruction, test-cross-platform, examples, cli-tests, bench-example, cross-check, ros2-bridge [Linux+ROS2], msrv). Platform-aware: green local run on platform X predicts green CI nightly for platform X's jobs
Full-repo mutation audit (~10-18 hrs)	`make qa-mutation-audit`	deliberate test-quality audit, NOT every nightly
All smoke-eligible example dataflows end-to-end (~15-20 min)	`make qa-examples`	orthogonal to ladder; wraps `scripts/smoke-all.sh`. Skips CUDA/ROS2/webcam/C++/interactive; qa-fast/full/deep `--exclude dora-examples` by design
Tier 3 automatable (qa-deep + semver)	`make qa-release-gate`	before tagging a release
Coverage (lcov report)	`make qa-coverage`	when investigating coverage
Mutation testing	`make qa-mutants`	when auditing test quality
Semver check	`make qa-semver`	before publishing to crates.io
Adversarial LLM review	`make qa-adversarial`	before merging AI-authored code (requires codex or claude CLI)

Cluster mode (SSH-backed `cluster up`)

# cluster.yml describes coordinator + remote machines
dora cluster up cluster.yml
dora cluster status
dora start examples/multiple-daemons/dataflow.yml
dora cluster down

Not automated: dora cluster up SSH-es into each machine in cluster.yml, which requires real remote hosts or a configured localhost SSH setup that's not available on stock GHA runners.

What is automated (Tier 1 nightly cluster-smoke job): dora cluster status and dora cluster down against a local dora up coordinator. Covers the coordinator-side wire protocol for those subcommands. The SSH-dependent cluster up path needs dedicated infra.

Soft real-time

sudo dora daemon --rt       # requires CAP_SYS_NICE or root
# With cpu_affinity + SCHED_FIFO in the dataflow YAML

Not automated: requires specific kernel config (CONFIG_RT_GROUP_SCHED) and privileged execution, which GHA runners do not provide reliably.

Maintainer validation runbook lives in docs/realtime-tuning.md. Run it before release if the --rt / mlock / SCHED_FIFO code paths were touched in the cycle. Tracked as #256.

Coordinator HA / distributed

# Terminal 1
dora coordinator --backend redb --store-path /tmp/dora-coord
# Terminal 2
dora daemon --machine-id A --coordinator-addr 127.0.0.1
# Terminal 3 — kill coordinator, restart, verify daemon auto-reconnects
pkill dora-coordinator
dora coordinator --backend redb --store-path /tmp/dora-coord

Not automated: failure-injection timing is flaky on cloud runners. Planned: dedicated self-hosted runner for HA scenarios.

Cross-machine Zenoh data plane

# Machine A
dora coordinator --bind 0.0.0.0
dora daemon --machine-id A --coordinator-addr <machine-a-ip>
# Machine B
dora daemon --machine-id B --coordinator-addr <machine-a-ip>

Not automated: requires two networked hosts.

What's NOT covered anywhere yet

Tracked in issue #215. Selected items:

dora self update (destructive path — binary swap; --check-only is covered in nightly but the actual upgrade requires a sandbox harness)
dora top interactive mode (non---once)

Most of these are interactive TUI commands that need either non-interactive modes or an expect-style harness. Opening separate issues as we pick them up.

Already covered (despite earlier audit)

These were on the gap list but now have nightly coverage in topic-and-top-smoke:

dora top --once (JSON snapshot)
dora topic list --format json (NDJSON parse)
dora topic info --duration N (asserts Total messages >= 10 on 10Hz fixture — regression guard for #236)
dora topic echo --count N (asserts N frames of correct topic name)
dora topic hz --duration N (asserts samples >= 10 on 10Hz fixture)
dora topic pub --count N
dora trace list (empty-state message)
dora trace view <absent-uuid> (empty-spans message, exit 0)
dora trace view <bad-prefix> (clear error message, exit non-zero)
dora self update --check-only (read-only path against live GitHub releases; tolerates API rate limits)

Platform parity

C/C++ template CI coverage is intentionally scoped to Linux (see issue #230):

Test	Linux	macOS	Windows
`dora new --lang rust/python` scaffolding	Tier 0 (`cli` job)	Tier 0 (`cli` job)	Tier 0 (`cli` job)
`dora new --lang c/cxx` scaffolding + CMake build + `dora run`	Tier 0 (`cli` job)	not covered	not covered
`cxx-dataflow`, `c-dataflow` examples	Tier 0 (`examples` job)	Tier 0	Tier 0
`cxx-arrow-dataflow` (needs Arrow C++ lib)	Tier 0	Tier 0 (homebrew `apache-arrow`)	not covered
`cmake-dataflow` example	Tier 0	not covered	not covered

Rationale. Production C/C++ node users are Linux-heavy. macOS/Windows C/C++ CI would need vcpkg or homebrew-formula pinning per runner, and the cost/benefit is poor for the number of real-world users on those paths. Rust/Python coverage stays cross-platform because those are the primary audience. If a macOS or Windows C/C++ regression is reported, the fix is to lift the if: runner.os == 'Linux' gate on the relevant step and add platform-specific install logic — not to keep shipping broken.

Tier policy (promotion / demotion criteria)

See issue #1632 for the policy discussion; this section is the authoritative summary.

Promote to Tier 0 (PR CI) when a test is:

Deterministic — no "either outcome is fine" fallbacks, no retries on flakiness.
Fast — adds a reasonable slice of per-PR runtime. The contract-tests job runs in ~2 min; that's roughly the ceiling for anything else promoted without a compelling reason.
High-signal — catches user-visible feature regressions, not just "did not crash".
Hosted-runner-friendly — no privileged execution, no multi-host infra, no per-runner timing hacks.

Keep in Tier 1 (nightly) when a test is useful but slow, mildly flaky on shared runners, timing-sensitive, or only catches a lower-frequency regression class.

Keep in Tier 2 (laptop / manual) when a test needs SSH-backed cluster deployment, real multi-host networking, privileged RT / mlock paths, or destructive operations (for example, an actual self update binary swap).

Demotion

If a Tier 0 gate flakes > 10% false-positive over a month, demote it to Tier 1 and open an issue to harden it before re-promoting.

Reactive promotion

If a Tier 1 failure reproduces for 3 consecutive nightly runs, promote the corresponding test to Tier 0. Keep lean-CI-first: promote only gates that catch real regressions, not flaky infra.