CI Workflow Outputs Layout

June 9, 2026 · View on GitHub

This document describes the directory layout for outputs produced by TheRock's CI workflow runs, and the Python modules for computing paths, uploading, and downloading those outputs.

Overview

Every CI workflow run produces a set of outputs (build artifacts, logs, manifests, python packages) that are uploaded to S3. Three modules in _therock_utils handle the path computation and I/O:

Module	Role	Key types
`storage_location`	Backend-agnostic location	`StorageLocation`
`workflow_outputs`	CI path computation (no I/O)	`WorkflowOutputRoot`
`storage_backend`	Upload I/O (write)	`StorageBackend`, `S3StorageBackend`, `LocalStorageBackend`
`artifact_backend`	Download I/O (read)	`ArtifactBackend`, `S3Backend`, `LocalDirectoryBackend`

StorageLocation is the bridge between path computation and I/O. WorkflowOutputRoot produces StorageLocation instances; backends consume them.

WorkflowOutputRoot ──produces──> StorageLocation ──consumed by──> StorageBackend
                                                            ArtifactBackend

S3 Layout

All outputs for a given run live under a common prefix:

s3://{bucket}/{external_repo}{run_id}-{platform}/

Component	Example	Description
`bucket`	`therock-ci-artifacts`	Selected based on repo, fork status, release type
`external_repo`	`""` or `"githubuser-TheRock/"`	Non-empty for forks and non-TheRock repos (format: `{owner}-{repo}/`)
`run_id`	`12345678901`	GitHub Actions workflow run ID
`platform`	`linux` or `windows`	Build platform

Directory structure

There are two CI pipeline architectures with different log layouts:

Single-stage CI (ci.yml) — one monolithic build per artifact group. Logs from all subprojects land in a single flat directory.
Multi-arch CI (multi_arch_ci.yml) — the build is split into stages (compiler-runtime, runtime-tests, math-libs, etc.). Each stage runs as a separate job and uploads its own logs, organized by stage name and GPU family.

artifact_group is the CI matrix variant, composed of a target family plus an optional variant suffix (e.g., gfx94X-dcgpu, gfx94X-dcgpu-asan). It is used as the subdirectory key for logs, manifests, and packages. Artifact filenames contain the target_family (e.g., gfx94X); see #3381 for ongoing work to propagate artifact group naming consistently.

Single-stage CI layout

{prefix}/
    {artifact_name}_{component}_{target_family}.tar.xz
    {artifact_name}_{component}_{target_family}.tar.xz.sha256sum
    index-{artifact_group}.html

    logs/{artifact_group}/
        {subproject}_build.log
        {subproject}_configure.log
        {subproject}_install.log
        ninja_logs.tar.gz
        build_observability.html          (when generated)
        index.html
        therock-build-prof/               (resource profiling subdirectory)
            comp-summary.html
            comp-summary.md
        comp-summary.html                 (flattened copy for direct linking)
        comp-summary.md                   (flattened copy for direct linking)

    manifests/{artifact_group}/
        therock_manifest.json

    python/{artifact_group}/
        *.whl
        *.tar.gz
        index.html

    tarballs/
        therock-dist-{platform}-{family}-{version}.tar.gz

The comp-summary.* files appear both in the therock-build-prof/ subdirectory (uploaded as part of the recursive directory upload) and at the log root (uploaded explicitly for direct linking).

Multi-arch CI layout

Logs are organized by stage, with a subdirectory per GPU family for per-arch stages. Generic stages (compiler-runtime, runtime-tests, profiler-apps) have no family subdirectory. Per-arch stages (e.g., math-libs) fan out across GPU families in parallel, producing identically-named log files (e.g., rocBLAS_build.log) that are kept separate by the family subdirectory.

{prefix}/
    {artifact_name}_{component}_{target_family}.tar.zst
    {artifact_name}_{component}_{target_family}.tar.zst.sha256sum

    logs/{stage_name}/                          (generic stages)
        {subproject}_build.log
        {subproject}_configure.log
        {subproject}_install.log
        ninja_logs.tar.gz

    logs/{stage_name}/{amdgpu_family}/          (per-arch stages)
        {subproject}_build.log
        {subproject}_configure.log
        {subproject}_install.log
        ninja_logs.tar.gz

    packages/deb/                               (APT repository)
        pool/main/
            *.deb
        dists/stable/
            Release
            main/binary-amd64/
                Packages
                Packages.gz

    packages/rpm/                               (DNF/Zypper repository)
        x86_64/
            *.rpm
            repodata/
                repomd.xml
                primary.xml.gz
                ...

    python/
        *.whl                                   (generic wheels, e.g., rocm_sdk_core)
        {amdgpu_family}/
            *.whl                               (per-family wheels, e.g., rocm_sdk_devel)
            *.tar.gz                            (sdist)
            index.html

    tarballs/
        therock-dist-{platform}-{family}-{version}.tar.gz
        therock-dist-{platform}-multiarch-{version}.tar.gz  (KPACK split only)

Example for a run with compiler-runtime + runtime-tests + math-libs stages:

12345-linux/
    logs/compiler-runtime/
        amd-llvm_build.log
        amd-llvm_configure.log
        amd-llvm_install.log
        ninja_logs.tar.gz

    logs/runtime-tests/
        hip-tests_build.log
        rocrtst_build.log
        ninja_logs.tar.gz

    logs/math-libs/gfx1151/
        rocBLAS_build.log
        MIOpen_build.log
        ninja_logs.tar.gz

    logs/math-libs/gfx110X-all/
        rocBLAS_build.log
        MIOpen_build.log
        ninja_logs.tar.gz

Artifacts in multi-arch CI use .tar.zst compression (vs .tar.xz in single-stage CI) and are managed by artifact_manager.py, not post_build_upload.py. Index pages for logs are generated server-side.

Bucket selection

The bucket is determined by _retrieve_bucket_info() in workflow_outputs.py. See S3 Buckets for the full list of buckets and authentication details.

RELEASE_TYPE=dev/nightly/prerelease? ──Yes──> therock-{RELEASE_TYPE}-artifacts
       │
       No (RELEASE_TYPE=ci)
       │
ROCm/TheRock (not fork)? ─────────────Yes──> therock-ci-artifacts
       │
       No
       │
       └──────────────────────────────────> therock-ci-artifacts-external

Valid artifact RELEASE_TYPE values are ci, dev, nightly, and prerelease.

Python API

StorageLocation

A frozen dataclass representing a single file or directory in S3 (or a local staging directory). Backend-agnostic — usable for CI run outputs, release artifacts, or any S3 path.

from _therock_utils.storage_location import StorageLocation

loc = StorageLocation(
    bucket="therock-ci-artifacts", relative_path="12345-linux/file.tar.xz"
)
loc.s3_uri  # "s3://therock-ci-artifacts/12345-linux/file.tar.xz"
loc.https_url  # "https://therock-ci-artifacts.s3.amazonaws.com/12345-linux/file.tar.xz"
loc.local_path(Path("/tmp/staging"))  # Path("/tmp/staging/12345-linux/file.tar.xz")

WorkflowOutputRoot

A frozen dataclass that computes StorageLocation for every output type.

from _therock_utils.workflow_outputs import WorkflowOutputRoot

# Inside a CI workflow (env vars provide bucket info, no API call)
root = WorkflowOutputRoot.from_workflow_run(run_id="12345", platform="linux")

# Fetching artifacts from another run (API call for fork/cutover detection)
root = WorkflowOutputRoot.from_workflow_run(
    run_id="12345", platform="linux", lookup_workflow_run=True
)

# For local development (no API calls, no env vars needed)
root = WorkflowOutputRoot.for_local(run_id="local", platform="linux")

# Location methods — each returns an StorageLocation
root.root()
root.artifact(filename="blas_lib_gfx94X.tar.xz")
root.artifact_index()
root.log_dir(artifact_group="gfx94X-dcgpu")
root.log_stage_dir(stage_name="math-libs", amdgpu_family="gfx1151")
root.log_stage_dir(stage_name="compiler-runtime")  # generic stage, no family
root.log_file(artifact_group="gfx94X-dcgpu", filename="build.log")
root.log_index(artifact_group="gfx94X-dcgpu")
root.build_observability(artifact_group="gfx94X-dcgpu")
root.manifest_dir(artifact_group="gfx94X-dcgpu")
root.manifest(artifact_group="gfx94X-dcgpu")
root.native_linux_packages(
    pkg_type="deb"
)  # {run_id}-linux/packages/deb — APT repo root
root.native_linux_packages(
    pkg_type="rpm"
)  # {run_id}-linux/packages/rpm — DNF/Zypper repo root
root.python_packages(artifact_group="gfx110X-all")
root.tarballs()

The lookup_workflow_run parameter controls whether from_workflow_run() calls the GitHub API to fetch workflow run metadata (for fork detection and bucket cutover dating). Most callers running inside their own CI workflow do not need this — environment variables (GITHUB_REPOSITORY, IS_PR_FROM_FORK) suffice. Set lookup_workflow_run=True when looking up another repository's workflow run, e.g. when fetching artifacts.

StorageBackend

An abstract base class for uploading files to S3 or a local directory. Use create_storage_backend() to get the right implementation.

from _therock_utils.storage_backend import create_storage_backend

backend = create_storage_backend()  # S3 (default)
backend = create_storage_backend(staging_dir=Path("/tmp/out"))  # local directory
backend = create_storage_backend(dry_run=True)  # print only

backend.upload_file(source_path, dest_location)
backend.upload_directory(source_dir, dest_location, include=["*.tar.xz*"])

Content-type is inferred from file extension — callers don't need to specify it.

Adding new output types

To add a new output type:

Add a method to WorkflowOutputRoot that returns StorageLocation
Add tests to build_tools/tests/workflow_outputs_test.py
Update this document

Consumers

Upload scripts

File	Uses
`post_build_upload.py`	`WorkflowOutputRoot` + `StorageBackend` for artifacts, logs, manifests
`upload_package_repo.py`	`WorkflowOutputRoot.native_linux_packages()` for deb/rpm package repositories
`post_stage_upload.py`	`WorkflowOutputRoot` + `StorageBackend` for multi-arch stage logs
`upload_tarballs.py`	`WorkflowOutputRoot` + `StorageBackend` for tarballs
`upload_python_packages.py`	`WorkflowOutputRoot` + `StorageBackend` for Python wheels and index
`upload_pytorch_manifest.py`	`WorkflowOutputRoot` + `StorageBackend` for PyTorch manifests
`upload_test_report_script.py`	`WorkflowOutputRoot` for S3 base URI (upload not yet migrated to backend)

Download scripts

File	Uses
`fetch_artifacts.py`	`WorkflowOutputRoot.from_workflow_run(lookup_workflow_run=True)` + `S3Backend`
`find_artifacts_for_commit.py`	`WorkflowOutputRoot.from_workflow_run(workflow_run=...)` for bucket/prefix
`artifact_backend.py`	`WorkflowOutputRoot` for `S3Backend` construction
`artifact_manager.py`	Via `create_backend_from_env()`