Dora Testing Guide

This guide covers how to run, write, and troubleshoot tests across the Dora workspace.

Looking for which tests cover which capability? See testing-capabilities.md (#1633) — the capability-oriented source-of-truth companion to this guide.

Prerequisites

• Rust toolchain — MSRV is the workspace rust-version in Cargo.toml (currently 1.88.0)
• Python 3 with numpy and pyarrow installed (pip install numpy pyarrow) — required for Python smoke tests

Quick Start (5-minute validation)

Run these three commands to validate that the workspace is healthy:

# 1. Format check (~5s)
cargo fmt --all -- --check

# 2. Lint (~60s first run, cached after)
cargo clippy --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python \
  -- -D warnings

# 3. Unit + integration tests (~90s first run)
cargo test --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

All three must pass before opening a PR. Python packages are excluded because they require maturin.

Test Tiers

Tier	What it covers	Command	Speed
Format	Code style	cargo fmt --all -- --check	~5s
Lint	Warnings, correctness	cargo clippy --all ...	~60s
Unit	Individual functions	cargo test --all ...	~90s
CLI	Command parsing, validation	cargo test -p dora-cli	~5s
Integration	Node I/O via env vars	cargo test --test example-tests	~30s
Smoke	Full CLI lifecycle	cargo test --test example-smoke -- --test-threads=1	~3min
E2E	Multi-dataflow scenarios	cargo test --test ws-cli-e2e -- --ignored --test-threads=1	~2min
Fault tolerance	Restart policies, timeouts	cargo test --test fault-tolerance-e2e	~45s
Typos	Spelling	Install typos-cli, then typos	~2s

Tier Details

Unit Tests

Unit tests live alongside the code they test using #[cfg(test)] modules. Key crates with tests:

Crate	Test count	What's tested
dora-arrow-convert	~26	Round-trip Arrow type conversions
dora-cli	~96	Command parsing, value parsers, log grep/filtering, JSON parsing, WebSocket client, cluster config
dora-coordinator	~24	WS control/daemon plane, health check, concurrent requests, artifact store, rate limiter, error sanitization
dora-coordinator-store	~10	In-memory and redb CRUD, schema versioning, persistence
dora-core	~8	Dataflow descriptor validation
dora-daemon	~2	Shlex argument parsing
dora-node-api	~10	Input tracking, service/action helpers (ID generation, send_service_request/response)
dora-log-utils	~11	Log parsing utilities
dora-message	~36	Common types, WS protocol, node/data IDs, metadata, auth tokens
ros2-bridge	~30	ROS2 message/service/action parsing

Run a single crate's tests:

cargo test -p dora-cli
cargo test -p dora-core
cargo test -p dora-arrow-convert

CLI Tests

CLI tests verify command parsing, argument validation, and value parsers without running any commands. They live in #[cfg(test)] modules inside the CLI crate.

What's tested:

• Clap schema validation (Args::command().debug_assert())
• Parsing of every subcommand (run, up, down, start, stop, list, logs, build, graph, new, status, inspect top, topic list/hz/echo, node list)
• Rejection of unknown subcommands
• --help and --version exit codes
• Value parsers: parse_store_spec (coordinator store backend), parse_window (topic hz window)
• Utility functions: parse_version_from_pip_show

How to run:

cargo test -p dora-cli

How to add new tests:

When adding a new CLI subcommand or value parser, add a corresponding test in the #[cfg(test)] module of the same file. For subcommand parsing, add a parse_ok call in binaries/cli/src/command/mod.rs. For value parsers, add tests in the file that defines the parser function.

Integration Tests (Node I/O)

File: tests/example-tests.rs

These tests run compiled node executables with pre-recorded inputs and compare outputs against expected baselines. No coordinator or daemon is needed.

cargo test --test example-tests

How it works:

1. Builds and runs a node crate (e.g., rust-dataflow-example-node)
2. Sets DORA_TEST_WITH_INPUTS to a JSON file with timed events
3. Sets DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 for deterministic output
4. Compares JSONL output against tests/sample-inputs/expected-outputs-*.jsonl

Sample input/output files live in tests/sample-inputs/.

Smoke Tests

File: tests/example-smoke.rs

Two execution modes are tested for each applicable example:

• Networked (dora up + dora start --detach + poll + dora stop + dora down): exercises the full coordinator/daemon WS control plane.
• Local (dora run --stop-after): runs everything in-process, testing the single-process dataflow path.

# Must run single-threaded (shared coordinator port)
cargo test -p dora-examples --test example-smoke -- --test-threads=1

# Run only networked or local tests
cargo test -p dora-examples --test example-smoke smoke_rust -- --test-threads=1
cargo test -p dora-examples --test example-smoke smoke_local -- --test-threads=1

A bash script is also available for quick local validation:

./scripts/smoke-all.sh              # all examples
./scripts/smoke-all.sh --rust-only  # Rust examples only
./scripts/smoke-all.sh --python-only # Python examples only

The suite currently exercises 45 smoke scenarios split across:

• Rust dataflows and benchmark examples
• Python dataflows in both networked and local modes
• module expansion via a runnable module dataflow
• service and action examples in both modes
• streaming, typed-dataflow, and log aggregation examples
• cross-language Rust↔Python examples in both modes
• deterministic validated-pipeline checks
• queue/timeout regressions, including the Rust queue-latest receiver

Representative coverage includes:

Category	Examples
Rust/networked	rust-dataflow, rust-dataflow_dynamic, rust-dataflow-url, benchmark
Python/networked	python-dataflow, python-async, python-echo, python-drain, python-log, python-logging, python-multiple-arrays, python-concurrent-rw, python-recv-async
Local-mode	Python examples above plus service-example, action-example, module-dataflow, streaming-example, typed-dataflow, log-aggregator, validated-pipeline, queue regressions
Cross-language	cross-language/rust-to-python.yml, cross-language/python-to-rust.yml
Feature-focused	service-example, action-example, streaming-example, validated-pipeline

Examples requiring special dependencies (webcam, CUDA, ROS2, C/C++ toolchain, multi-machine deploy) are not included in smoke tests.

E2E Tests (WebSocket CLI)

File: tests/ws-cli-e2e.rs

Two groups:

Non-ignored (fast): Start an in-process coordinator and test WsSession directly:

cargo test --test ws-cli-e2e

• cli_list_empty -- empty dataflow listing
• cli_status_no_daemon -- daemon connectivity check
• cli_stop_nonexistent -- error for missing dataflows
• cli_multiple_requests_same_session -- session reuse

Ignored (full stack): Use dora up with real nodes:

cargo test --test ws-cli-e2e -- --ignored --test-threads=1

• e2e_start_list_stop -- start, list, stop lifecycle
• e2e_sequential_dataflows -- two dataflows in sequence

Fault Tolerance Tests

File: tests/fault-tolerance-e2e.rs

These test restart policies and input timeouts using Daemon::run_dataflow directly (no CLI needed).

cargo test --test fault-tolerance-e2e

Tests:

• restart_recovers_from_failure -- node with restart_policy: on-failure survives panics (15s)
• max_restarts_limit_reached -- node exhausts max_restarts: 2 budget (15s)
• input_timeout_delivers_input_closed_to_downstream -- input_timeout: 0.5s fires after a silent upstream, delivering InputClosed to the observer (5s)

Dataflow YAMLs for these tests live in tests/dataflows/.

Coordinator Integration Tests

Files: binaries/coordinator/tests/ws_control_tests.rs, binaries/coordinator/tests/ws_daemon_tests.rs

These start an in-process coordinator and test the WebSocket control/daemon planes.

cargo test -p dora-coordinator

Topics covered: health check, list/stop/destroy requests, invalid JSON/params, concurrent requests, ping/pong, daemon registration, disconnect cleanup, error sanitization (no internal chain leaks), artifact store cleanup on drop.

CI Pipeline

Two workflows split by cadence (#1716):

• .github/workflows/ci.yml — runs on every PR and push to main. Linux-only. Blocks merge. Target ~30-45 min critical path.
• .github/workflows/nightly.yml — daily 06:40 UTC cron + manual dispatch. Cross-platform. Does NOT block PRs; auto-files nightly-regression issue on failure. ~3-4 hours wall-clock.

PR CI (ci.yml) — fast Linux-only gate

Job	Runner	What runs
fmt	ubuntu-latest	cargo fmt --all -- --check
clippy	ubuntu-latest	cargo clippy --all ... -- -D warnings
test	ubuntu-latest	cargo check, cargo build, cargo test --all ... (excluding Python crates and dora-examples) plus fast CLI smoke/semantic checks
e2e	ubuntu-latest	ws-cli-e2e and fault-tolerance-e2e
contract-tests	ubuntu-latest	tests/example-smoke.rs::contract_* behavior contracts
bench	ubuntu-latest	criterion benchmark regression check
typos	ubuntu-latest	crate-ci/typos@master
audit	ubuntu-latest	cargo-audit + cargo-deny via make qa-audit
unwrap-budget	ubuntu-latest	.unwrap() / .expect() budget check
check-license	ubuntu-latest	license validation

Nightly CI (nightly.yml) — cross-platform breadth

Job	Runner	What runs
smoke-suite	ubuntu-latest	Full tests/example-smoke.rs (52 tests, all Python examples)
test-cross-platform	macOS + Windows	Same steps as PR test, on the non-Linux matrix
examples	ubuntu/macOS/windows	Rust, Rust Git, Multiple Daemons, C, C++, C++ Arrow, CMake
cli-tests	ubuntu/macOS/windows	dora new template projects, Python dynamic node, queue latency, error-event
bench-example	ubuntu/macOS/windows	cargo run --example benchmark --release
cross-check	matrix (8 targets)	compile-only across x86_64/aarch64 linux-gnu + musl + pc-windows-gnu + apple-darwin
ros2-bridge	ubuntu-22.04	ROS2 Humble bridge tests + examples
msrv	ubuntu-latest	cargo-hack check --rust-version
log-sinks / service-action / streaming / record-replay	ubuntu-latest	Integration smoke against individual dataflow surfaces
cluster-smoke / topic-and-top-smoke / cpu-affinity-smoke / redb-backend-smoke / daemon-reconnect-smoke / state-reconstruction-smoke	ubuntu-latest	Specialized lifecycle / regression smokes
file-issue-on-failure	ubuntu-latest	Auto-files issue on any failure

The full smoke suite in tests/example-smoke.rs runs in nightly (smoke-suite), not in PR CI. Developers who need cross-platform or broader coverage before merge run make qa-test / make qa-examples / make qa-nightly locally — see qa-runbook.md.

Writing New Tests

Unit tests

Add a #[cfg(test)] module in the same file as the code under test:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parses_valid_input() {
        let result = parse("valid");
        assert_eq!(result, expected);
    }
}

Integration tests for nodes

Use the integration testing framework in dora-node-api. Three approaches:

1. setup_integration_testing (recommended)

Call before the node's main function to inject inputs and capture outputs:

#[test]
fn test_main_function() -> eyre::Result<()> {
    let events = vec![
        TimedIncomingEvent {
            time_offset_secs: 0.01,
            event: IncomingEvent::Input {
                id: "tick".into(),
                metadata: None,
                data: None,
            },
        },
        TimedIncomingEvent {
            time_offset_secs: 0.055,
            event: IncomingEvent::Stop,
        },
    ];
    let inputs = TestingInput::Input(
        IntegrationTestInput::new("node_id".parse().unwrap(), events),
    );
    let (tx, rx) = flume::unbounded();
    let outputs = TestingOutput::ToChannel(tx);
    let options = TestingOptions { skip_output_time_offsets: true };

    integration_testing::setup_integration_testing(inputs, outputs, options);
    crate::main()?;

    let outputs = rx.try_iter().collect::<Vec<_>>();
    assert_eq!(outputs, expected_outputs);
    Ok(())
}

2. Environment variable mode

Test the compiled executable directly, closest to production behavior:

DORA_TEST_WITH_INPUTS=path/to/inputs.json \
DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 \
DORA_TEST_WRITE_OUTPUTS_TO=/tmp/out.jsonl \
cargo run -p my-node

3. DoraNode::init_testing

For testing node logic without going through main:

let (node, events) = DoraNode::init_testing(inputs, outputs, Default::default())?;

Generating test input files

Record real dataflow events by setting DORA_WRITE_EVENTS_TO:

DORA_WRITE_EVENTS_TO=/tmp/recorded-events dora run examples/rust-dataflow/dataflow.yml

This writes inputs-{node_id}.json files that can be used directly with DORA_TEST_WITH_INPUTS.

Workspace-level integration tests

Add new test files in the tests/ directory. For tests that need the full CLI stack, follow the patterns in tests/example-smoke.rs:

Networked pattern (exercises coordinator + daemon):

1. Build nodes with Once guards (avoid rebuilding per test)
2. Clean up stale processes with dora down
3. Start cluster with dora up
4. Run dataflow with dora start --detach
5. Poll dora list --json for completion
6. Clean up with dora stop --all and dora down

Local pattern (single-process, in-process coordinator):

1. Build CLI with Once guard
2. Run dora run <yaml> --stop-after <duration>
3. Assert exit code is success

Conventions

• Use assert2::assert! for better error messages (available as dev-dependency)
• Use tempfile::NamedTempFile for temporary output files
• E2E tests that need exclusive port access should be #[ignore] and run with --test-threads=1
• Async tests use #[tokio::test(flavor = "multi_thread")]
• Fault tolerance test dataflows go in tests/dataflows/
• Sample input/output baselines go in tests/sample-inputs/

Troubleshooting

cargo test fails to compile Python packages

Always exclude Python packages:

cargo test --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

Smoke/E2E tests fail with "address already in use"

A stale coordinator or daemon is still running. Clean up:

dora down
# or kill processes manually:
pkill -f dora-coordinator
pkill -f dora-daemon

Smoke tests hang or timeout

• Increase the timeout in the test if your machine is slow (look for Duration::from_secs(...))
• Check that example nodes build successfully:

  cargo build -p rust-dataflow-example-node -p rust-dataflow-example-status-node \
    -p rust-dataflow-example-sink -p rust-dataflow-example-sink-dynamic
  cargo build -p log-sink-file -p log-sink-alert -p log-sink-tcp
  cargo build --release -p benchmark-example-node -p benchmark-example-sink
  

• For Python smoke tests, ensure pyarrow and numpy are installed

E2E tests fail when run in parallel

Smoke and ignored E2E tests must run single-threaded:

cargo test --test example-smoke -- --test-threads=1
cargo test --test ws-cli-e2e -- --ignored --test-threads=1

Integration test output doesn't match expected

1. Check that DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 is set (time offsets vary per machine)
2. Regenerate baselines if the node's behavior intentionally changed:

   DORA_TEST_WITH_INPUTS=tests/sample-inputs/inputs-rust-node.json \
   DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 \
   DORA_TEST_WRITE_OUTPUTS_TO=tests/sample-inputs/expected-outputs-rust-node.jsonl \
   cargo run -p rust-dataflow-example-node
   

Typos check fails

The typos config is in _typos.toml. To add a false-positive exclusion:

[default.extend-identifiers]
MyCustomIdent = "MyCustomIdent"

Tests pass locally but fail in CI

• CI runs on Ubuntu; check for platform-specific assumptions (paths, process signals)
• CI uses rust-cache so dependency versions may differ from your local lockfile
• Ensure cargo fmt --all -- --check passes (CI enforces this)