Adding tests to TheRock

Test Flow

After TheRock builds its artifacts, we test those artifacts through test_artifacts.yml. The testing flow works as:

graph LR
    test_sanity_check --> configure_test_matrix --> test_components

where we:

1. Check that the artifacts pass sanity tests.
2. The configure_test_matrix step runs fetch_test_configurations.py, where we generate a test matrix for which tests to run.
3. After we generate the matrix, test_components executes those tests in parallel.

How these tests are executed

These tests are retrieved from fetch_test_configurations.py, where we generate a matrix of tests to run for various AMD GPU families from amdgpu_family_matrix.py on both Linux and Windows test machines.

These tests are run per pull request, main branch commit, workflow_dispatch and nightly runs.

What kind of tests are suitable for TheRock

Since TheRock is the open source build system for HIP and ROCm, we are interested in tests for individual subprojects as well as tests that exercise multiple subprojects, especially for build and runtime dependencies. We also perform higher level testing of overall user-facing behavior and downstream frameworks like PyTorch.

Adding tests

To add tests, add your executable logic to github_actions/test_executable_scripts with a Python file (in order to be compatible with Linux and Windows). Below is an example for hipblaslt.py:

cmd = [f"{THEROCK_BIN_DIR}/hipblaslt-test", "--gtest_filter=*pre_checkin*"]
logging.info(f"++ Exec [{THEROCK_DIR}]$ {shlex.join(cmd)}")
subprocess.run(
    cmd,
    cwd=THEROCK_DIR,
    check=True,
)

After creating your script, please refer below to create your test entry in fetch_test_configurations.py

Fields for the test matrix

Add an entry in test_matrix, then your test will be enabled in the test workflow

In fetch_test_configurations.py, a test option (in this example rocBLAS) in test_matrix is setup as:

"rocblas": {
    "job_name": "rocblas",
    "fetch_artifact_args": "--blas --tests",
    "timeout_minutes": 5,
    "test_script": f"python {SCRIPT_DIR / 'test_rocblas.py'}",
    "platform": ["linux", "windows"],
}

Field Name	Type	Platform	Description
job_name	string	Any	Name of the job
fetch_artifact_args	string	Any	Arguments for which artifacts for install_rocm_from_artifacts.py to retrieve
timeout_minutes	int	Any	The timeout (in minutes) for the test step
test_script	string	Any	The path to the test script
platform	array	Any	An array of platforms that the test can execute on, options are linux and windows
container_image	string	Linux	The name of a container image to use for this component
container_options	string	Linux	Additional options to be passed when launching the container