SALT-FM: An LLVM-based Source Analysis Toolkit for HPC

The TAU Performance System is a powerful and versatile performance monitoring and analysis system. TAU supports several mechanisms for instrumenting applications, including source-based instrumentation. Source-based instrumentation inserts instrumentation points around areas of interest by parsing and modifying an application's source code. However, source-based instrumentation can be difficult to use, especially for large code-bases written in modern languages such as C++ and Fortran. To improve the usability of TAU, we introduce a next-generation source analysis toolkit called SALT-FM. SALT-FM is built using LLVM compiler technology, including the libTooling library, Clang's C and C++ parsers, and f18/Flang's Fortran parser. SALT-FM supports the following LLVM/Clang versions, each tested in CI against a matching salt-dev container image. The same images are also published to Docker Hub under the paratools/salt-dev repository.

LLVM/Clang version	GitHub Container Registry	Docker Hub
19	ghcr.io/paratoolsinc/salt-dev:1.3	paratools/salt-dev:1.3
20	ghcr.io/paratoolsinc/salt-dev:1.4	paratools/salt-dev:1.4
21	ghcr.io/paratoolsinc/salt-dev:1.5	paratools/salt-dev:1.5
22	ghcr.io/paratoolsinc/salt-dev:1.6	paratools/salt-dev:1.6

Getting Started

The build system for SALT-FM uses CMake as the build system generator (also known as a meta-build system). It also requires either a full installation of LLVM and Clang including the clang-cmake-exports and cmake-exports components, or a more minimal set of features and components (including both of the aforementioned cmake-exports) together with a patch applied to two of the LLVM/Clang installed CMake files.

The development and testing of SALT-FM utilizes the salt-dev container, which is posted to Docker Hub. The container image includes an installation of git, cmake, llvm/clang, and ninja. The installed llvm and clang include the patched CMake files.

Below are the steps required to get started using SALT-FM:

1. Clone this repo:

git clone --recursive https://github.com/ParaToolsInc/salt.git

Including the --recursive flag will ensure you have the patches that are applied to the llvm/clang installed CMake files available to you. This is not a requirement if using the salt-dev container image. The patches prevent SALT-FM's configuration with CMake from failing when incomplete/minimal installations of LLVM and Clang are present. A minimal subset of components must be installed. If SALT-FM is failing to be configured in the next step, and you have write access to the LLVM and/or Clang installation, applying the patches from https://github.com/ParaToolsInc/salt-llvm-patches may fix the problem.

2. Launch the salt-dev Docker image (optional)

$ docker pull paratools/salt-dev:latest
$ docker run -it --pull always --tmpfs=/dev/shm:rw,nosuid,nodev,exec \
    --privileged -v $(pwd):/home/salt/src paratools/salt-dev:latest
# cd /home/salt/src
# ccache --show-stats

This step is optional but recommended. The llvm/clang CMake files are patched, and this is the environment used in CI testing. The docker image also includes ccache and two configurations of TAU for testing with both GCC and Clang. (See /usr/local/x86_64/lib for the two installed TAU_MAKEFILEs.)

3. Configure and build SALT-FM:

This step is most easily performed in the salt-dev container, but can also be performed if a suitable installation of LLVM and Clang is present.

# Tell CMake to use the clang/clang++ compiler to build SALT-FM
export CC=clang
export CXX=clang++
# configure SALT-FM and generate build system
cmake -Wdev -Wdeprecated -S . -B build
cmake --build build --parallel # Add --verbose to debug something going wrong

To specify a TAU installation for testing other than the one used by the salt-dev container, you can add -DTAU_ROOT=<dir> to the first cmake invocation.

If the Ninja build system is present and you prefer it to Makefiles (it's already present in the salt-dev container image), then you may add -G Ninja to the penultimate line above:

cmake -Wdev -Wdeprecated -S . -B build -G Ninja

macOS (Homebrew LLVM 22)

Apple's bundled clang does not ship flang. Install the Homebrew formulae (instructions and flang):

brew install llvm flang ninja

Configure and build the same way as on Linux, pointing the compiler at the Homebrew LLVM clang so flang-new is reachable:

cmake -Wdev -Wdeprecated -G Ninja -S . -B build \
  -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
  -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

CMake auto-detects the rest:

• CMAKE_PREFIX_PATH is seeded from the active C++ compiler's prefix, llvm-config, and flang-new (no Homebrew/Spack/conda queries) so find_package(LLVM) and find_package(Flang) resolve without -DCMAKE_PREFIX_PATH=....
• CMAKE_OSX_SYSROOT falls back to the active Xcode SDK; you do not need to pass it explicitly.
• A configure-time try_compile of <ryml_all.hpp> with the resolved toolchain catches misconfiguration with an actionable error before the real build starts.

Notes:

• CMAKE_OSX_DEPLOYMENT_TARGET defaults to 11.0 and is auto-raised to match the linked LLVM dylib's LC_BUILD_VERSION minos if higher (avoids ld "object file built for newer macOS version" warnings).
• If TAU is not available, pass -DSALT_REQUIRE_TAU=OFF; TAU-dependent tests are skipped automatically.

Once this is finished, you will have an executable cparse-llvm in the build directory.

4. Running the tests (optional):

Running the tests is encouraged to verify functionality.

cd build
ctest --output-on-failure

The tests are all located in the tests subdirectory of the project. CMake test fixtures and test dependencies ensure that:

• The tests run in an order such that inter-test dependencies are satisfied
• The SALT-FM configuration files are correctly staged to the build directory
• Old instrumented versions of test sources are cleaned up just in time before being instrumented again
  • This is so the user can inspect instrumented sources
• Old object files are cleaned up
• Old profiles associated with each test are removed just in time
• New profiles are moved to subdirectories indicating which test they are associated with

By default, the tests assume a TAU installation matching the salt-dev development image (located at /usr/local/x86_64/ with a GCC and Clang configuration).

To use a different TAU installation, add -DTAU_ROOT=<dir> to the cmake invocation. The specified directory should point to a TAU install built with at least these two configurations:

-pthread -cc=clang -c++=clang++ -bfd=download -unwind=download -dwarf=download -otf=download

and

-pthread -bfd=download -unwind=download -libdwarf=download -otf=download

5. Example usage:

# Point to the appropriate config file if config_files/config.yaml is not in CWD
./cparse-llvm --config_file=/path/to/config.yaml ../tests/hello.c

This will produce a file hello.inst.c. Passing the correct defines (see the CMakeLists.txt file) and setting TAU_MAKEFILE=/usr/local/x86_64/lib/Makefile.tau-pthread will allow you to compile hello.inst.c with tau_cc.sh -optLinkOnly -D... hello.inst.c -o hello. Running tau_exec ./hello should then produce a profile.0.0.0 file.

Notes for package maintainers

When SALT-FM is configured inside a git checkout, the build system installs client-side git hooks (in the working tree's .git/config via core.hooksPath) to enforce .VERSION / tag agreement during local development. This is convenience tooling for contributors and is irrelevant when packaging release tarballs or building from a source snapshot pulled by a package manager.

Distribution build recipes (Spack, RPM, Debian, conda-forge, Nix, etc.) should pass:

-DSALT_INSTALL_GIT_HOOKS=OFF

to the cmake invocation, regardless of whether the source tree happens to be a git checkout. The flag is a no-op when there is no .git/ directory, but setting it explicitly documents intent and avoids modifying the local clone's git configuration in environments where that is unexpected.

Funding Acknowledgement

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of SBIR and STTR Programs under Award Number DE-SC0022511.

This material is based upon work supported by the National Aeronautics and Space Administration under Contract Number 80NSSC24PB401.