Diff-based CI test selection

This document explains the diff-driven test-selection system that lets some GPU CI workflows run only the tests a PR could affect, instead of the whole suite, while still satisfying Required status checks.

It currently drives modal-torch-latest (which runs tests/unit/v1/ on modal.com GPUs), but is built to drive more workflows from one config — see Adding a workflow.

• TL;DR
• Why
• Moving parts
• How a decision is made
• Driving it (as a contributor)
• Changing it (as a maintainer)
• Security model
• Failure modes & guarantees
• FAQ / troubleshooting

TL;DR

• On a PR, ci/tests_fetcher.py diffs your branch against the base branch, traces the import graph from your changed files to the impacted tests, and writes the list to ci/.test_selection/test_list.txt.
• The trusted controller validates that list, then fetches, installs, and tests the exact candidate SHA inside a no-secret Modal Sandbox. push to master and manual runs always run everything.
• It is fail-safe: anything it can't reason about safely → run the full suite. It never silently runs fewer tests than reality.
• Preview locally:

  python ci/tests_fetcher.py --base origin/master            # what would CI run?
  python ci/tests_fetcher.py --base origin/master --explain  # ...and why?
  

• Force a full run: put [test all] (or [no filter]) in a commit message.

Why

modal-torch-latest is a Required check, so it must report a status on every PR — which means we can't use GitHub's path filters (on.<event>.paths) to skip it, because a skipped Required job blocks merges. Instead, the job always runs but selects what to test: a docs-only PR resolves to "no impacted tests" and exits fast (still green), while a PR touching a leaf module runs only the tests that import it.

The design is a small, self-contained take on HuggingFace transformers' utils/tests_fetcher.py.

Moving parts

File	Role
.github/workflows/modal-torch-latest.yml	The workflow: a no-secret collect-tests job gating a trusted deploy controller job.
ci/tests_fetcher.py	The selector. AST-parses the repo, builds an import graph, decides all / subset / none, writes the test-list file, emits a job summary.
ci/test_tests_fetcher.py	Self-tests for the selector (pure stdlib; run in collect-tests).
ci/torch_latest.py	Pure-stdlib metadata/selection validation plus the trusted Modal Sandbox controller.
ci/.test_selection/test_list.txt	The hand-off artifact (one pytest target per line). Git-ignored.

Job flow

                pull_request_target / push / workflow_dispatch
                                  │
                    ┌─────────────┴─────────────┐
                    │        collect-tests       │   (no secrets)
                    │  checkout exact base SHA   │
                    │  public-fetch exact PR SHA │
                    │  self-test the fetcher     │
                    │  parse candidate as data   │
                    │   → mode (all|subset|none) │
                    │   → validate/upload list   │
                    └─────────────┬──────────────┘
                                  │ needs:
                                  ▼
                    ┌────────────────────────────┐
                    │           deploy            │   (Modal token: controller only)
                    │  if mode != none            │
                    │  checkout exact base SHA    │
                    │  validate mode/list + SHA   │
                    │  create one L40S:2 Sandbox  │
                    │    no secrets or mounts     │
                    │    fetch exact PR SHA       │
                    │    install + run pytest     │
                    │  terminate/observe Sandbox  │
                    └────────────────────────────┘

mode controls deploy:

• none → deploy is skipped. The Required status is still satisfied (a skipped dependent job counts as success here).
• subset → deploy runs pytest on exactly the impacted files.
• all → deploy runs the whole scope (tests/unit/v1).

How a decision is made

TestSelector.select() in ci/tests_fetcher.py runs these checks in order; the first that matches wins:

1. No base ref (push / manual) → all.
2. Base ref unresolvable → all.
3. No merge-base with the base (e.g. shallow clone, unrelated history) → all. A diff here would be wrong, so we never narrow on it.
4. Commit message tag [test all] / [no filter] anywhere on the branch → all.
5. A changed file matches a run-all glob (COMMON_RUN_ALL_GLOBS + the workflow's extra_run_all_globs) → all. These are files too central or too dynamic to narrow safely: CI scripts, build system, csrc/, op_builder/, accelerator/, shared fixtures (tests/unit/common.py, tests/conftest.py, pytest.ini), and core runtime hubs (deepspeed/__init__.py, deepspeed/runtime/engine.py, deepspeed/comm/**, deepspeed/accelerator/**, …).
6. A deleted module is still imported by a surviving file (a dangling import the graph can't follow) → all. A clean deletion (importers removed/updated in the same PR) does not trigger this.
7. Otherwise, narrow via the import graph (below). If nothing is impacted → none; if everything is → all; else → subset.

The import graph

• Nodes are Python files under the package roots: deepspeed/** and the unit test-helper package at tests/unit/**.
• An edge A → B means "B imports A". The selector walks backwards from each changed file to every test that (transitively) imports it.
• Opaque hub modules (OPAQUE_MODULES: deepspeed, deepspeed.comm, deepspeed.accelerator): almost every test imports unit.common, which imports these. Their __init__.py files eagerly pull in huge subtrees, so if treated as normal nodes any deepspeed/** change would fan out to the whole suite. We therefore don't expand their __init__ imports; instead, changes to the hubs themselves are caught by the run-all globs in step 5.
• conftest.py changes select every test under that conftest's directory.
• New test files are selected directly (they have no importers yet).

Dynamic edges (DYNAMIC_EDGES)

Some dependencies are wired at runtime (monkey-patching, plugin/registry lookup, JIT-loaded ops, deepspeed.initialize()-time replace_module injection), so a test can depend on code it never imports. DYNAMIC_EDGES is a curated map of changed-file glob → extra test-path globs that patches these blind spots. It is additive on top of the static graph (and is only consulted if step 5 didn't already short-circuit to all).

Driving it (as a contributor)

Preview what CI will run

The fetcher is pure stdlib — no DeepSpeed/torch install needed, just git:

# Selection for your branch vs. the base branch
python ci/tests_fetcher.py --base origin/master
cat ci/.test_selection/test_list.txt

# Explain *why* each test was selected (prints import chains)
python ci/tests_fetcher.py --base origin/master --explain

--explain output looks like:

deepspeed/shared.py impacts:
    tests/unit/v1/test_shared.py <- deepspeed/shared.py
    tests/unit/v1/moe/test_moe.py <- tests/unit/v1/moe/test_moe.py <- deepspeed/shared.py

Escape hatches

• Force the full suite for a push: include [test all] (or [no filter]) anywhere in a commit message on the branch.
• Touch an infra file: any change to a run-all glob runs everything.
• Found a missed test? It's likely a runtime/dynamic dependency the static graph can't see — add a DYNAMIC_EDGES entry (see below) and/or report it.

Changing it (as a maintainer)

All knobs live in ci/tests_fetcher.py. After any change, run the self-tests:

python ci/test_tests_fetcher.py          # standalone
# or: pytest ci/test_tests_fetcher.py

Add a run-all trigger

Add a glob to TestSelector.COMMON_RUN_ALL_GLOBS (shared across workflows) or to a specific workflow's extra_run_all_globs. Globs match repo-root-relative POSIX paths; dir/** matches everything under dir/.

Add a dynamic edge

Add to TestSelector.DYNAMIC_EDGES:

DYNAMIC_EDGES = {
    # changed-file glob : test-path globs to pull in when it changes
    "deepspeed/module_inject/**": ("tests/unit/v1/moe/**",),
}

Keep entries conservative: too broad just wastes GPU time; too narrow misses coverage. Prefer fixing it here over widening a run-all glob when only a slice of tests is truly affected.

Mark a module opaque

If a new universal hub starts fanning every change out to the whole suite, add it to OPAQUE_MODULES and (usually) add the hub file itself to the run-all globs so real changes to it still run everything.

Add a new workflow

The engine is config-driven (WORKFLOWS registry of WorkflowConfig). To drive another workflow:

1. Add an entry:

   WORKFLOWS["my-workflow"] = WorkflowConfig(
       name="my-workflow",
       test_scopes=("tests/unit/v2",),               # dirs this workflow runs
       extra_run_all_globs=(".github/workflows/my-workflow.yml",),
   )
   

2. In that workflow's YAML, mirror modal-torch-latest.yml's collect-tests job and call the fetcher with --workflow my-workflow.
3. Point the runner at ci/.test_selection/test_list.txt.
4. Add coverage to ci/test_tests_fetcher.py if the scope/behavior differs.

Add a self-test

ci/test_tests_fetcher.py builds throwaway git repos (TmpRepo) from a synthetic BASELINE tree and asserts the resulting mode/tests. Add a test_* function for any new behavior; it runs both standalone and under pytest, and executes in the collect-tests CI job, so a broken selector is caught before it mis-picks tests.

Security model

The workflow triggers on pull_request_target, so it runs in the base repo's context and the deploy controller authenticates to Modal. The trust boundary is:

• Both jobs check out only the exact trusted base SHA (or the exact push/manual SHA), with persisted credentials, LFS, and submodules disabled. They never use the fork head with actions/checkout.
• collect-tests holds no repository secrets. Trusted base code fetches the public head and base repositories at their exact event SHAs, disables credentials, hooks, submodules, and LFS smudging, and rejects checkout symlinks that escape the candidate root. It treats the candidate tree only as git/AST data; it never imports, installs, builds, or executes candidate code.
• The selector logic and its self-tests always come from the trusted checkout. A PR's ci/ changes still appear in the diff, so the base selector's ci/** run-all rule widens them to the full suite.
• The handoff is a fixed all / subset / none mode plus one bounded, validated regular file. Test paths must stay under tests/unit/v1, cannot be options or traversal paths, and are passed after pytest's -- separator.
• deploy runs only the trusted controller. Candidate metadata is restricted to a public owner/repository and full 40-hex SHA; event values enter Python through environment variables, not workflow shell interpolation.
• The Modal service token remains in the controller process. The Sandbox gets no Modal, GitHub, Hugging Face, OIDC, or other secret; no local checkout, mount, volume, network filesystem, port, proxy, or workload identity is attached. The controller constructs a positive allowlist of non-sensitive git/pip environment flags instead of forwarding its environment.
• Candidate acquisition, dependency installation, DeepSpeed installation, and pytest all happen inside one isolated l40s:2 Sandbox with a 3600-second server-side lifetime. Every subprocess uses structural arguments, and every nonzero clone/install/test status fails the check.
• The Sandbox retains outbound network access because it must reach public GitHub, package indexes, PyTorch wheels, and optionally Transformers. Those services do not have stable CIDRs suitable for the SDK's CIDR allowlist, so the containment control is the absence of secrets, identities, and mounts.
• The pull_request_target trigger types are review_requested, ready_for_review, and synchronize. Because synchronize re-runs on every push to an open PR (not just on a maintainer action), the maintainer review is a mitigation, not the trust boundary. Exact trusted base code plus Sandbox isolation is the primary protection.

Consequence: changes to ci/* (including tests_fetcher.py itself) take effect under pull_request_target only after they're merged. A PR that changes this launcher cannot prove its new PR-triggered end-to-end path by running itself. Before merge, use pure-stdlib/unit/static validation and, only when separately authorized credentials are available, a direct Modal smoke. Do not describe mock coverage as live Modal evidence.

Failure modes & guarantees

The selector is built to fail safe — to all, never to none:

• Missing/unresolvable base, no merge-base, a parse error on a file, or any unexpected exception in the selector → it falls back to the full suite (the top-level handler in main() logs the traceback and sets mode=all).
• Checkout, selector self-test, list validation, write, or upload failures make collect-tests fail. deploy then enters its explicit failure guard, so a broken collector cannot pass the Required check.
• The only way to run fewer tests is a clean, well-understood narrow decision; every uncertain case widens to everything.
• Missing or invalid repository/SHA metadata, an inconsistent mode/list pair, and any failed Sandbox clone, install, version probe, or pytest command fail the controller. There is no moving-branch fallback.
• Once created, the task-owned Sandbox is terminated and its terminal state is observed on success and failure. Cleanup failure is itself a failure and is reported without hiding the primary command error. Forced controller loss is bounded by the Sandbox's one-hour server lifetime.
• none creates no Sandbox. subset and all create exactly one Sandbox; the GPU shape and lifetime are constants rather than PR or manual inputs.

Every run writes a summary to the GitHub job summary (mode, reason, and the selected files) so the decision is auditable from the Actions UI. Candidate text is control-escaped before logs/summaries and is never written to GITHUB_OUTPUT.

FAQ / troubleshooting

My PR shows mode=none but I changed code. Either the change is non-Python / out of the workflow's scope, or your edits aren't committed (the fetcher diffs committed history, merge-base..HEAD).

A relevant test wasn't selected. Likely a runtime/dynamic dependency. Confirm with --explain, then add a DYNAMIC_EDGES entry. As a stop-gap, push with [test all].

Everything runs even for a tiny change. You touched a run-all glob (CI/build/shared fixture/core runtime), or a hub module fanned out. Check the job summary's reason. If a hub over-fans, consider OPAQUE_MODULES.

It ran the full suite and the summary says "shallow clone?" / "no merge-base". The exact event commits did not provide a usable merge-base. The collector fetches the exact head and base SHAs into fixed refs; verify both event commits are still publicly reachable. Do not replace this with a moving branch fallback.

Why didn't this PR exercise its new pull_request_target controller? GitHub intentionally runs that event's workflow from the trusted base. The new controller becomes the trusted code only after merge; before then, rely on the focused static/unit evidence described in the security model.