Running the test suite in parallel

The Spec Kitty test suite runs safely in parallel locally and in CI, typically at least 2× faster on a machine with four or more cores. This page explains the one correct local command, why it is shaped the way it is, and how to reproduce the coverage-neutrality gates CI uses.

For what to do when a test goes red on CI unrelated to your diff — budget gates vs. correctness flakes vs. environmental flakes, and why we never retry-to-green — see the test-flakiness handling policy.

The local command

PWHEADLESS=1 pytest tests/ -n auto --dist loadfile -p no:cacheprovider
# daemon/real-port tests run serially:
PWHEADLESS=1 pytest tests/sync/test_orphan_sweep.py -n0 -q

The first command runs the bulk of the suite across worker processes. The second command runs the daemon/real-port tests serially. Run both; the parallel command deliberately leaves the serial-only tests for the second pass.

Why --dist loadfile (never bare --dist load)

pytest-xdist supports several distribution modes. We always use loadfile:

• loadfile keeps every test that lives in the same file on a single worker. File-scoped fixtures (scope="module", file-level collection ordering, shared module state) keep working exactly as they do serially.
• load (the bare default) scatters a single file's tests across multiple workers. That breaks file-scoped fixtures and any test that relies on collection order within a file.

For that reason: always pass --dist loadfile; never use bare --dist load. CI uses loadfile for the same reason.

-p no:cacheprovider disables pytest's cache plugin so a parallel run never races on the shared .pytest_cache directory.

Per-worker HOME isolation (the master enabler)

A parallel run never touches the real ~/.spec-kitty. Each pytest-xdist worker — and the serial "master" run when you omit -n auto — gets its own isolated home directory. The isolation is set up in tests/conftest.py:

• pytest_configure points HOME / USERPROFILE and the XDG dirs (XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_STATE_HOME) at a per-worker base before collection, so modules that bind a home-derived path at import time (for example specify_cli.sync.daemon.SPEC_KITTY_DIR) resolve into the isolated home.
• An autouse, function-scoped fixture re-asserts the HOME / USERPROFILE / XDG env vars for every test, keyed by worker id, so call-time Path.home() reads are isolated too. It does not monkeypatch Path.home (that approach was the cycle-1 regression that broke ~16 tests/sync cases — the fixture relies on Path.home() natively resolving HOME via expanduser), so a test that sets up its own tmp home via setenv('HOME', ...) cleanly overrides the per-worker baseline.

The per-worker base is keyed by the xdist test-run UID and the worker id, so two workers in the same run get distinct homes (no collision) and successive runs do not reuse stale state. The regression guard tests/architectural/test_real_home_isolation_guard.py (SC-006) and tests/test_worker_home_isolation.py prove this invariant.

Because the real ~/.spec-kitty is never bound, you do not need to back it up or worry about a parallel run truncating your real queue.db.

The serial daemon pass

Per-worker HOME isolation protects per-user state, but it does not protect OS-global resources such as real TCP ports or singleton daemons. Tests that bind the reserved daemon port range (9400–9449) — tests/sync/test_orphan_sweep.py — must run in their own serial pass:

PWHEADLESS=1 pytest tests/sync/test_orphan_sweep.py -n0 -q

-n0 forces serial execution even when xdist is installed. These tests are excluded from the parallel pool so two workers never contend for the same port.

Volume env gates (SPEC_KITTY_ULID_VOLUME_FULL)

Some tests exercise large-volume ULID generation. By default they run at a reduced scale so the local default stays fast; the full scale is reachable via an env gate (and is exercised on the nightly/full path). The assertion logic is identical across scales — only the volume changes.

pytest <ulid_test> -q                               # reduced (fast, default)
SPEC_KITTY_ULID_VOLUME_FULL=1 pytest <ulid_test> -q  # full (nightly parity)

Running the stability ratchet locally

Before any shard is flipped to parallel, it must pass the stability ratchet (C-RATCHET): N consecutive green parallel runs with no new flakes. The same entrypoint CI uses is available locally (the WP02 coverage-safety harness):

python -m tests._support.coverage_safety.ratchet -n 3 -- tests/agent -m "not slow"

Exit code 0 means all N runs were green and the flip is accepted; 1 means it was rejected and the summary names any new or flaky failures. The Python API is run_ratchet(...) from tests._support.coverage_safety. See tests/_support/coverage_safety/README.md for the full harness (collection equivalence and anti-vacuity mutation checks).

Validate the acceleration (copy-pasteable)

These are the mission's reproducible validation steps. Run them from the repo root to confirm the parallel run is coverage-neutral and at least 2× faster than serial. (.venv/bin/pytest is the synced project interpreter; substitute pytest if you run it directly.)

# 1. Serial baseline (whole-suite wall clock) and a per-shard nodeid reference.
time .venv/bin/pytest tests/ -q -p no:cacheprovider     # serial baseline
.venv/bin/pytest tests/charter --collect-only -q | sort > /tmp/charter-serial.nodeids

# 2. Collection equivalence: serial vs parallel must collect identical nodeids.
.venv/bin/pytest tests/charter -n auto --dist loadfile --collect-only -q \
  | sort > /tmp/charter-par.nodeids
diff /tmp/charter-serial.nodeids /tmp/charter-par.nodeids   # must be empty

# 3. Stability ratchet: 3 consecutive green parallel runs (the same gate CI uses).
python -m tests._support.coverage_safety.ratchet -n 3 -- \
  tests/charter -m "fast and not windows_ci"

# 4. Parallel-vs-serial timing: target ≥2× faster on a ≥4-core machine.
time PWHEADLESS=1 .venv/bin/pytest tests/ -n auto --dist loadfile -p no:cacheprovider \
  --deselect tests/sync/test_orphan_sweep.py
time PWHEADLESS=1 .venv/bin/pytest tests/sync/test_orphan_sweep.py -n0 -q   # serial pass

# 5. Real home untouched: mtime/inode unchanged (or path still absent) after the run.
ls -la ~/.spec-kitty 2>/dev/null

CI shard-topology mission status

Point-in-time mission-status snapshots (named mission IDs, CI-confirmation state, and per-job PENDING wall-clock records for the ci-test-topology-performance-01KXBJRT shard-topology re-flip) have moved to testing-parallel-ci-topology-status.md in engineering notes — this how-to page stays focused on the durable local workflow above.

Reproducing #3115 (the folded-uuid render-width defect)

sync status / sync doctor render a Project column with overflow="fold" (src/specify_cli/cli/commands/sync.py:1440, deliberate). At an 80-column console width a 36-character project uuid folds across two lines and stops being a contiguous substring of the captured output, so a plain uuid in out assertion fails even though the journal is fully populated. The 80-column width comes from rich.console.Console.size's if self.is_dumb_terminal: branch, which returns the hardcoded ConsoleDimensions(80, 25) above the COLUMNS read — so COLUMNS, which the affected tests' isolation fixture already sets, is never consulted. is_terminal is true whenever FORCE_COLOR is set to a non-empty value, and is_dumb_terminal additionally requires TERM to be dumb or unknown.

This is reproduced with two environment variables, one victim file, one process — no pytest-xdist, because --dist loadfile's worker assignment is dynamic and work-stealing, so a reproducer that depends on a particular assignment would not be reproducible by construction.

TERM=dumb FORCE_COLOR=1 ./scripts/repro_3115_render_width.sh

This runs tests/cli/commands/test_sync_status_per_project_3030.py (4 collected tests) alone, in one process, and reds with:

FAILED tests/cli/commands/test_sync_status_per_project_3030.py::test_status_names_every_project_with_count_age_and_consent
AssertionError: aaaaaaaa-0000-0000-0000-000000000001 is in the journal but `status` did not name it
1 failed, 3 passed in ~46-56s

The sibling victim file reproduces the same defect independently:

TERM=dumb FORCE_COLOR=1 ./scripts/repro_3115_render_width.sh doctor

tests/cli/commands/test_sync_doctor_per_project_3030.py collects 12 tests and reds 1 failed, 11 passed, with assertion text aaaaaaaa-0000-0000-0000-000000000001 is in the journal but doctor did not name it (no backticks around doctor — the two victim files' f-strings are not identical, so quote each one's text verbatim rather than assuming they match).

Only the first of the three seeded projects (CONSENTED, aaaaaaaa-0000-0000-0000-000000000001) demonstrates the defect: the other two (SILENT, OPTED_OUT) pass at width 80 anyway, via an un-tabled warning paragraph that reprints their identity outside the folding table. Queue 0 event(s) / Queue size 0 / 100,000 appear in the captured output regardless of outcome (OfflineQueue().size(), sync.py:5182-5185) — they are not a signature of this defect and must not be read as one.

Both counted lines are collected-count-relative: test_sync_status_per_project_3030.py collects 4 (red is 1 failed, 3 passed), test_sync_doctor_per_project_3030.py collects 12 (red is 1 failed, 11 passed). A count line that does not reconcile against its file's own --collect-only -q count is not evidence.

Control — the same command plus TTY_COMPATIBLE=0 must PASS. This is what distinguishes "the width is the cause" from "this file is just broken":

TERM=dumb FORCE_COLOR=1 TTY_COMPATIBLE=0 ./scripts/repro_3115_render_width.sh
# 4 passed
TERM=dumb FORCE_COLOR=1 TTY_COMPATIBLE=0 ./scripts/repro_3115_render_width.sh doctor
# 12 passed

Determinism is not established by repetition alone: pytest-randomly is not installed on this tree, so nothing randomises order and "the same node-id comes up red three times in a row" is trivially true regardless of whether the red is order-dependent. The clause that can actually fail is running the failing case alone, by node-id, with no file-siblings collected first:

TERM=dumb FORCE_COLOR=1 python3 -m pytest \
  "tests/cli/commands/test_sync_status_per_project_3030.py::test_status_names_every_project_with_count_age_and_consent"
# 1 failed, collected count 1, identical assertion text

A red that needs its file-siblings to run first would be order-dependent and would not be reproducible by construction (C-004).

./scripts/repro_3115_render_width.sh computes its own repo root from its own location and sets PYTHONPATH to that repo's src/ before invoking pytest. This matters in a git worktree: the shared .venv's _editable_impl_spec_kitty_cli.pth holds the main checkout's absolute src path, so a bare pytest run inside a worktree using that .venv silently imports the main checkout's live tree instead of the worktree's own source. The script's PYTHONPATH line exists specifically to defeat that.

The whole reproducer (one victim file, one process) completes in well under 2 minutes (measured around 46-59s per file on this codebase's base commit, bb2020fea9) — cheap enough that nothing downstream needs to take the defect on faith.