Testing guide

This guide describes where tests live, how to run them locally, and how they map to CI. For coding standards and PR workflow, see Contributing Guide. The canonical CI definition is .github/workflows/ci.yaml.

1. Test organization

Layout (simplified from the repository root):

Primus/
├── runner/
│   └── lib/
│       └── common.sh              # Shared logging/helpers sourced by the shell test runner
├── tests/
│   ├── runner/                    # Shell integration tests
│   │   ├── run_all_tests.sh       # Master shell test runner
│   │   ├── lib/                   # test_common.sh, test_config.sh, test_validation.sh
│   │   ├── helpers/               # Hook and env tests
│   │   ├── test_primus_cli.sh
│   │   ├── test_primus_cli_direct.sh
│   │   ├── test_primus_cli_container.sh
│   │   └── test_primus_cli_slurm.sh
│   ├── unit_tests/                # Python unit tests (pytest)
│   │   ├── agents/                # Tuning-agent tests
│   │   ├── backends/
│   │   ├── ci/
│   │   ├── cli/
│   │   ├── core/                  # config, backend, launcher, patches, projection, pipeline_parallel, runtime, trainer, utils
│   │   ├── megatron/              # Megatron-specific unit tests
│   │   ├── modules/
│   │   └── tools/
│   ├── trainer/                   # Integration tests (typically need GPU + data)
│   │   ├── test_megatron_trainer.py
│   │   ├── test_torchtitan_trainer.py
│   │   └── test_maxtext_trainer.py
│   ├── scripts/                   # CI unit/integration launch scripts and UT patches
│   ├── utils.py                   # Shared test utilities
│   └── run_unit_tests.py          # Optional orchestrator (walks tests/, see below)

tests/runner/run_all_tests.sh sources shared helpers from runner/lib/common.sh at the repository root (not under tests/runner/).

2. Running tests

Shell integration tests (CLI behavior, config loading, hooks, environment):

bash ./tests/runner/run_all_tests.sh

Python unit tests:

pytest tests/unit_tests/ --maxfail=1 -s

Trainer integration tests (GPU and data; may require Hugging Face access):

# Megatron
DATA_PATH=<path> pytest tests/trainer/test_megatron_trainer.py -s

# TorchTitan
DATA_PATH=<path> pytest tests/trainer/test_torchtitan_trainer.py -s

# MaxText (JAX) — often run via the orchestrator in CI
python ./tests/run_unit_tests.py --jax

tests/run_unit_tests.py walks tests/ and runs every test_*.py it finds, except for tests/trainer/test_maxtext_trainer.py in the default mode (that file is only selected when --jax is set). That means the default orchestrator run includes tests/unit_tests/ and tests/trainer/ (and any other matching tests), which is broader than pytest tests/unit_tests/ alone.

Orchestrator (default—all discovered tests except MaxText trainer):

python ./tests/run_unit_tests.py

Orchestrator (JAX / MaxText trainer only):

python ./tests/run_unit_tests.py --jax

3. Test types

• Shell tests: Exercise runner scripts, CLI wiring, configuration loading, hook execution, and environment setup. Implemented as bash scripts under tests/runner/ and orchestrated by run_all_tests.sh.
• Unit tests: Cover configuration parsing, preset loading, CLI behavior, patch registration, adapters, and other library logic under tests/unit_tests/.
• Trainer tests: End-to-end training against real backends; require AMD GPUs and appropriate data paths (and sometimes tokens). See .github/workflows/ci.yaml for CI values such as DATA_PATH, MASTER_PORT, and HSA_NO_SCRATCH_RECLAIM.

4. Writing new tests

• Pytest: Add files named test_*.py under tests/unit_tests/, following existing patterns and reusing fixtures from conftest.py where present.
• Shell: Add scripts under tests/runner/ or extend tests/runner/run_all_tests.sh to invoke new suites, consistent with existing test_primus_cli*.sh scripts.
• Backends: Prefer tests/unit_tests/backends/<backend>/ for adapter-focused tests.

5. CI pipeline details

From .github/workflows/ci.yaml:

• code-lint: Python 3.12 on GitHub-hosted runners. Runs pre-commit run --all-files --show-diff-on-failure, so checks follow .pre-commit-config.yaml.
• dependency-review: Runs actions/dependency-review-action on pull requests to flag dependency changes.
• run-unittest-torch: Self-hosted GPU runner. Installs requirements.txt, runs bash ./tests/runner/run_all_tests.sh, then pytest tests/unit_tests/ under coverage (--cov=primus --cov-report=term-missing) with specific tests --deselected (currently some megatron/cco TP-overlap and megatron/transformer/moe dispatcher cases—see the workflow file). Trainer steps set MASTER_PORT, DATA_PATH, HSA_NO_SCRATCH_RECLAIM=1, and HF_TOKEN for Megatron and TorchTitan trainer tests. A follow-up coverage step combines unit and E2E coverage.
• run-unittest-jax: JAX runner. Installs requirements-jax.txt, runs the same shell test script, then python ./tests/run_unit_tests.py --jax with CI environment variables (for example JAX_SKIP_UT=1 and DATA_PATH as defined in the workflow).

The build-docker job builds images after lint passes; unit test jobs depend on code-lint, not on build-docker.

6. Pre-commit hooks

Install once per clone:

pip install pre-commit
pre-commit install

Run manually on the whole tree:

pre-commit run --all-files

Hooks include: trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files, check-merge-conflict, isort, autoflake, black, and shellcheck (as configured in .pre-commit-config.yaml). These align with the code-lint job in CI; see Contributing Guide for manual equivalents.