Testing in TheRock

TheRock is the integration point for build, test, packaging, and release infrastructure for the ROCm Core SDK. The code here is used by developers building ROCm from source, CI systems validating pull request contributions in repositories like rocm-systems and rocm-libraries, and release workflows publishing nightly and stable releases in rockrel which are trusted by users and downstream projects.

TheRock aims to keep ROCm "ready to release" at any time. Achieving this at scale requires robust automated tests that detect issues as close as possible to their source. Early detection limits the impact of regressions and makes them easier to diagnose and fix, while continuous validation provides the confidence needed to release frequently.

Testing must scale across a broad support surface. ROCm includes 40+ subprojects, supports 25+ GPU targets across multiple hardware generations, runs on multiple operating systems, is distributed via multiple packaging formats, and is used by many downstream frameworks. Testing every combination for every change is not practical, so TheRock layers automated tests according to their cost and the confidence they provide. Presubmit testing prioritizes fast, high-signal test suites and configurations with enough capacity to run for every change. Longer test suites and hardware with limited runner capacity are exercised through nightly, scheduled, and on-demand testing.

Testing should be accessible to all contributors. Wherever possible, code and automation should be structured so that important behavior can be tested quickly on commonly available development machines. Local testing usually provides the fastest feedback, while continuous integration (CI) workflows provide consistent environments for validating changes across representative project-wide configurations and component boundaries.

The ROCm Core SDK is built and released as a single product. Individual subprojects may validate their own behavior in isolation, then TheRock assembles and tests those projects together as often as practical throughout development to provide confidence in cross-component behavior and product-wide properties that component-level testing cannot evaluate.

This page describes how these testing layers work together to validate changes to TheRock and the ROCm subprojects it integrates. It also explains how code and automation are structured for testing, which validation methods apply to each feature area, and the known limitations of the current coverage.

Table of contents

• Testing changes to TheRock
  • Test categories in TheRock
  • TheRock feature area: CMake and super-project build logic
  • TheRock feature area: GitHub Actions workflows
  • TheRock feature area: Python scripts and tools
  • TheRock feature area: Packaging
• Testing changes to ROCm subprojects with TheRock
  • Building subprojects through TheRock
  • Testing subprojects through TheRock

Testing changes to TheRock

Test categories in TheRock

Tests for the code in TheRock itself are split into a few broad categories:

• pre-commit and static checks
  • These are fast checks for formatting, linting, repository policies, and more
  • Example checks (see .pre-commit-config.yaml):
    • actionlint for GitHub Actions workflow files
    • black formatting for Python scripts
• unit tests
  • These are tests for script behavior, runnable on generic hardware
  • Example tests:
    • build_tools/tests/build_topology_test.py
    • build_tools/tests/py_packaging_test.py
    • build_tools/github_actions/tests/workflow_dispatch_inputs_test.py
• integration tests
  • These provide validation for build outputs and packages and may run on real hardware
  • Example tests:
    • tests/test_artifact_structure.py
    • tests/test_rocm_sanity.py
    • build_tools/packaging/linux/native_linux_package_install_test.py
    • build_tools/packaging/python/templates/rocm/src/rocm_sdk/tests/core_test.py

Tests in each category should be runnable as part of local development and also run as part of our CI workflows:

Project features should be tested using a combination of these test types that balance time to signal and representative coverage. For example, Python packages should have both unit tests for package building and integration tests for package installation and runtime behavior.

TheRock feature area: CMake and super-project build logic

Super-project CMake build - Scope

As the centralized build system for ROCm Core, TheRock includes a CMake super-project using code in:

• CMake files like CMakeLists.txt, cmake/therock_amdgpu_targets.cmake, and FLAGS.cmake
• Topology metadata in BUILD_TOPOLOGY.toml
• Sub-project declarations like math-libs/CMakeLists.txt
• Sub-project artifact descriptors like math-libs/BLAS/artifact-blas.toml
• Scripts used by the build system like build_tools/fileset_tool.py

The build system supports a broad matrix of configurations:

Super-project CMake build - Design for testing

The CMake build system is designed to be reproducible, configurable, and debuggable:

• We use the same build system for Linux and Windows with minimal branching.
• Subproject builds can be run in isolation and their configured options can be viewed via _init.cmake and _toolchain.cmake files (see build_system.md).
• Build commands are routed through teatime.py so all logs are written to ${build}/logs/. CI/CD workflow runs upload logs to S3 buckets (see s3_buckets.md) following the schema in workflow_outputs.md.
• Build performance logs are collected by ninja and uploaded together with other logs using post_stage_upload.py.
• Common base CMake option combinations are managed through CMakePresets.json.

Source code:

• First-party dependencies are loaded through pinned git submodules.
• Third-party dependencies are vendored with source mirrors in third-party/ (see also dependencies.md and git_chores.md#updating-a-third-party-mirror).
• Workflows generate commit manifest files and diff reports (manifest_diff.md).

Build environments:

• On Linux we recommend building inside a manylinux Docker container that includes a known-working, minimal set of dependencies. See dockerfiles/README.md.
• Other environments are also supported, though they aren't tested as regularly: environment_setup_guide.md.

Super-project CMake build - Validation methods

We are evaluating adding unit tests for certain features of the CMake build system itself, see https://github.com/ROCm/TheRock/pull/6984 for example.

The CI systems in TheRock and component repositories like rocm-systems continuously build a few slices through our support matrix. For changes to build system files, we generally look for

• The build and test jobs in .github/workflows/multi_arch_ci.yml should not have new failures.
• The build jobs should not significantly regress in duration.
• The build artifacts should not unexpectedly grow in size.

Super-project CMake build - Limitations and known gaps

TheRock feature area: GitHub Actions workflows

GitHub Actions workflows - Scope

We use GitHub Actions in the .github/workflows directory for a variety of workflows:

• Lightweight checks: codeql.yml, gitleaks.yml, pre-commit.yml, unit_tests.yml, therock-pr-bot.yml, etc.
• CI/CD workflows: multi_arch_ci.yml, multi_arch_release.yml, etc.
• Other automation: bump_submodules.yml, copy_release.yml, publish_build_manylinux_x86_64.yml

Many of these workflows are central to day-to-day project development and official releases, so care must be taken to test them thoroughly.

GitHub Actions workflows - Design for testing

Workflows can take hours to run and can be difficult to debug, so we follow these practices to make testing manageable:

• Keep workflows as simple as possible, e.g. by putting logic in Python scripts rather than inline Bash and then writing unit tests for those scripts (see this section in github_actions_style_guide.md).
• Document how to reproduce test environments and run tests locally:
  • test_environment_reproduction.md
  • Running/testing PyTorch and build_tools/github_actions/summarize_test_pytorch_workflow.py
• Support running using prebuilt artifacts/packages and minimal matrices for efficient testing. For example, test_rocm_wheels.yml is used as part of the multi_arch_ci.yml workflow which builds ROCm fully from source, but it can be run directly against prebuilt ROCm Python packages for any specific Python version, runner type, and test container image.
• Where possible, support testing workflows in repository forks (see "Working effectively from forks" in github_actions_debugging.md).
• When workflows and scripts are used across repositories, pin to specific commits so workflow runs are reproducible and updates can be tested prior to rollout.
  • In rocm-libraries and rocm-systems, "TheRock CI" uses commit pins that receive regular update pull requests via build_tools/github_actions/bump_automation.py. These pull requests can be reviewed and fixed when there are breaking changes to the build system, workflows, or scripts.

GitHub Actions workflows - Validation methods

We test our GitHub Actions workflows using a combination of these practices:

• Run workflows through actionlint static analysis (this is required via a pre-commit hook).
• Add unit tests like build_tools/github_actions/tests/workflow_dispatch_inputs_test.py where actionlint falls short.
• Test changes to workflows using "CI" and "dev" environments isolated from production (see "Testing release workflows" in github_actions_debugging.md and s3_buckets.md).
• If a workflow does not run on pull_request events, test manually with workflow_dispatch (https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manually-run-a-workflow) and link the test runs in the pull request description.

GitHub Actions workflows - Limitations and known gaps

Cross-repository workflow design and testing is difficult, so we try to limit such usage and review changes carefully.

uses: ROCm/TheRock/.github/workflows/multi_arch_release.yml@main

TheRock feature area: Python scripts and tools

Python scripts and tools - Scope

Most build system and utility scripts are written in Python, not Bash or other languages. Our GitHub Actions workflows also use Python scripts for the bulk of their logic (see the GitHub Actions workflows section above).

Python scripts and tools - Design for testing

We test our Python scripts using pytest, aiming to follow the style guidelines in python_style_guide.md and particularly the "testing standards" section.

Python scripts and tools - Validation methods

All Python unit tests should be run as part of .github/workflows/unit_tests.yml, with the help of files like build_tools/pyproject.toml.

Note that simple unit tests do not fully replace integration testing using real build tools, packages, or remote APIs.

Python scripts and tools - Limitations and known gaps

TheRock feature area: Packaging

Packaging - Scope

The artifacts produced by the build system are assembled into tarballs/archives, Python packages, and native operating system packages for distribution.

Packaging - Design for testing

• Packages should be buildable from ROCm artifacts using scripts in build_tools/packaging/ with documentation in docs/packaging/.
• With the exception of package signing, developers and downstream projects should be able to build packages exactly as TheRock's CI/CD system does.
• Packages should use dev/nightly/stable versions following docs/packaging/versioning.md for version/channel sorting and auditability.

Packaging - Validation methods

Packages are tested using a combination of these practices:

• Unit tests for package construction scripts
  • Test structural metadata for inputs and outputs, file inclusion/exclusion filters, script portability across environments
• Installation tests which check that packages can be installed and used:
  • Package self-tests (example: rocm-sdk test, see Python Packaging - Testing).
  • Install tests may run on multiple operating systems / distros since we build packages to be portably distributed.
  • Integration and regression tests for interactions between multiple packages, ensuring that ROCm packages are self-sufficient, don't conflict with system packages, and can be used together with other ecosystem packages

Native Linux packaging unit tests live under build_tools/packaging/linux/tests/.

Packaging - Limitations and known gaps

Testing changes to ROCm subprojects with TheRock

This section focuses on how TheRock is used to test changes to ROCm subprojects such as amd-llvm, hip-clr, RCCL, MIOpen, etc.

Subprojects which get built and released by TheRock as "ROCm core" store their own source code in separate repositories such as:

• ROCm/rocm-systems
• ROCm/rocm-libraries
• ROCm/llvm-project

During development in those repositories, changes may be tested with a combination of

• "Component CI" that can be specialized per subproject and generally exercises individual subprojects directly
• "TheRock CI" that leverages the unified build system, builds and tests multiple subprojects together, and closely matches the release environment

Attempts are made to "bump" each submodule reference in TheRock regularly (see .github/workflows/bump_submodules.yml), and these "bump PRs" run more exhaustive builds and tests than "TheRock CI" runs in the subproject source repositories.

Workflows running in https://github.com/ROCm/rockrel build release packages using code from TheRock and then trigger even more comprehensive tests across available GPU test runners.

Building subprojects through TheRock

Once a subproject's source code is included in TheRock, it can be integrated into the build system to produce artifacts by following build_system.md - Adding subprojects. Artifacts produced for subprojects are included in packages using the configuration code in build_tools/packaging/.

Testing subprojects through TheRock

Subproject build tests

Subprojects can run CPU-only tests during the build by using the therock_cmake_subproject_build_test() function defined in cmake/therock_subproject.cmake, like so:

# Using ctest
therock_cmake_subproject_build_test(rocjitsu
  COMMAND "${CMAKE_CTEST_COMMAND}" --output-on-failure --no-tests=error
)

# Using a python test runner
therock_cmake_subproject_build_test(amd-comgr
  COMMAND
    "${Python3_EXECUTABLE}" "${_llvm_lit_script}"
    "${CMAKE_BINARY_DIR}/compiler/amd-comgr/build/test-lit" -v
)

These build tests can be run from the superproject CMake build via

cmake --build "${BUILD_DIR}" --target therock-build-tests -- -k 0

 amd-comgr_build.log
+amd-comgr_build_test_1.log
+amd-comgr_build_test_2.log
 amd-comgr_configure.log
 amd-comgr_install.log

Subproject test runner

Subprojects can run installed component tests, including GPU tests, after the build by adding a test script to /build_tools/github_actions/test_executable_scripts/.

Tests should

• Use test filter levels per docs/development/test_filtering.md
• Follow the guidelines in test_executable_scripts/README.md
• Use the common test_executable_scripts/test_runner.py