README.md

August 27, 2026 · View on GitHub

PrimusBench

PrimusBench is a set of scripts for multi-node system preflight checks and benchmarking. It supports both bare-metal (Ansible-driven) and SLURM-based runs, covering network connectivity, node health checks, I/O benchmarks, and key system metrics such as computation-communication overlap and kernel launch overhead.

Features

  • Preflight: SSH reachability, node configuration, network connectivity
  • I/O Benchmarks: fio, IOR (optional)
  • System Benchmarks: Computation-Communication Overlap, Kernel Launch Overhead
  • Three run modes: Bare metal via Ansible, SLURM cluster, or Kubernetes with PyTorchJob

Prerequisites

  • Target nodes run a basic Linux environment (Ubuntu 22.04 or equivalent recommended)
  • Nodes are mutually reachable; the control node can SSH to targets (passwordless or auto-configured by scripts)
  • Bare-metal mode requires Ansible on the control node; SLURM mode requires salloc/srun
  • ROCm/PyTorch dependencies are typically provided by the container image

Quick Start

Running on Bare Metal

  1. Prepare host inventory Create or edit hosts.ini at the repo root:
[all]
host1
host2

Hostnames or IPs are supported; ensure they resolve.

  1. Run The entry script will:
  • Install Docker across nodes via Ansible
  • Start preflight checks and benchmarks, writing logs to outputs/<TIMESTAMP>

Common environment variables:

  • IMAGE: Benchmark image (default: primussafe/primusbench:202510210028)
  • INVENTORY_FILE: Ansible inventory file (default: hosts.ini)
  • IO_BENCHMARK_MOUNT: Enable and mount a directory for I/O benchmarks (optional)

Run:

bash run_bare_metal.sh

Key steps in run_bare_metal.sh:

  • Install Docker: playbooks/bare_metal/install_docker.yaml
  • Run benchmarks: playbooks/bare_metal/bench.yaml
  • Logs directory: outputs/<TIMESTAMP> (main log primusbench.log)

Running on SLURM

The SLURM mode automatically handles resource allocation and benchmark execution. By default, run_slurm.sh will use salloc to allocate resources before running the benchmarks.

Default Usage (Automatic Allocation):

# Automatically allocate 2 nodes and run benchmarks (default behavior)
bash run_slurm.sh

# Allocate 4 nodes with custom settings
NNODES=4 PARTITION=gpu TIME=2:00:00 bash run_slurm.sh

Within Existing SLURM Allocation:

# If already in a SLURM job, skip auto-allocation
bash run_slurm.sh --no-allocate

Excluding Problem Nodes:

# Use default exclude list (configured in script)
bash run_slurm.sh

# Custom exclude list
EXCLUDE_NODES="chi[2770-2772]" bash run_slurm.sh

# Don't exclude any nodes
EXCLUDE_NODES="" bash run_slurm.sh

run_slurm.sh will:

  • Automatically allocate nodes via salloc (unless --no-allocate is used)
  • Display excluded nodes if configured
  • Use srun to execute containers on each node
  • Run node and network preflight checks
  • Filter out unhealthy nodes and keep healthy ones for benchmarks
  • Generate comprehensive health reports
  • Write outputs to outputs/<TIMESTAMP>

Environment variables (configured in config.sh, can be overridden):

  • NNODES: Number of nodes (default: 2)
  • PARTITION: SLURM partition (default: configured in config.sh)
  • TIME: Job time limit (default: 4:30:00)
  • CPUS_PER_TASK: CPUs per task (default: 128)
  • EXCLUDE_NODES: Nodes to exclude from allocation
  • IMAGE: Container image for benchmarks
  • MASTER_PORT: Master port for distributed operations
  • See config.sh for complete configuration options

Running on Kubernetes with PyTorchJob

For Kubernetes clusters with Kubeflow PyTorchJob operator, you can run distributed benchmarks using the provided PyTorchJob specification.

Prerequisites:

  • Kubernetes cluster with Kubeflow Training Operator installed
  • Access to a shared storage (hostPath or PVC) for workspace and outputs
  • Container image available in your cluster registry

Steps:

  1. Prepare the PyTorchJob manifest Edit kubernetes/pytorchjob.yaml and replace the following placeholders:
  • <host_path to be replaced, shared directory>: Path to shared workspace directory on nodes
  • <pvc_name to be replaced， io_benchmark_pvc>: Name of PVC for I/O benchmarks (if using PVC)
  • Adjust replicas count for Master and Worker as needed
  • Update resource limits (GPUs, memory, CPU) according to your cluster
  1. Configure environment variables Key variables in the YAML (adjust as needed):
  • NCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME: Network interface name
  • NCCL_IB_HCA: RDMA device names (comma-separated)
  • IO_BENCHMARK_MOUNT: Mount point for I/O benchmarks
  • Container image: Update image field with your image tag
  1. Deploy the PyTorchJob
kubectl apply -f kubernetes/pytorchjob.yaml
  1. Monitor the job
# Check job status
kubectl get pytorchjob network

# View logs from master pod
kubectl logs -f <master-pod-name>

# View logs from worker pods
kubectl logs -f <worker-pod-name>
  1. Collect results Results will be written to the shared workspace directory specified in the volumeMounts.

Important notes:

  • The job uses hostNetwork: true for optimal RDMA performance
  • Privileged security context is required for device access
  • Ensure all nodes have the necessary RDMA devices and drivers installed
  • Worker replicas count determines the number of worker nodes (adjust based on cluster size)

Key scripts and directories

  • run_bare_metal.sh: bare-metal entrypoint; installs Docker, calls Ansible playbooks, streams logs
  • run_slurm.sh: SLURM execution with automatic resource allocation
  • run_slurm.sh: preflight and network checks inside the SLURM job
  • run.sh: container entrypoint; runs I/O benchmarks, node/network preflight, system benchmarks, and collects results
  • playbooks/: Ansible playbooks (bare-metal install and benchmarks)
  • kubernetes/: Kubernetes manifests including PyTorchJob specifications
  • benchmarks/: benchmark implementations and build scripts
  • preflight/: SSH, node, and network preflight components
  • build/: Dockerfile and build.sh, which build the images with the necessary dependencies and tools

Outputs and logs

Default output locations:

  • Bare metal: outputs/<TIMESTAMP>
  • SLURM: output/<TIMESTAMP> (or SHARE_PATH/output/<TIMESTAMP>)
  • Kubernetes: Written to shared workspace volume specified in PyTorchJob manifest

Important files (examples):

  • primusbench.log: main runtime log (streamed by bare-metal entry)
  • io_benchmarks.log: I/O benchmark log (when enabled)
  • preflight_node.log / preflight_network.log: preflight results
  • overlap_results.json / kernel_overhead_results.json: system benchmark results

Container image

Default images are set via environment variables in the scripts:

  • Benchmark image: IMAGE (used by run_bare_metal.sh / Ansible playbooks)
  • Preflight images: PREFLIGHT_NODE_IMAGE, PREFLIGHT_NETWORK_IMAGE (SLURM)

Build the image

  • Build from this repo:
docker build -t primussafe/primusbench:{{TAG}} .

FAQ

  • Ansible connection failures: Verify hostnames/IPs in hosts.ini resolve; confirm SSH port and passwordless access; set INVENTORY_FILE to a custom inventory if needed.
  • Logs not generated: In bare-metal mode, main log is outputs/<TIMESTAMP>/primusbench.log; if missing, check bare-metal or install phase logs first.
  • Nodes marked unhealthy: Inspect preflight_node.log and preflight_network.log; fix reported issues (ports, drivers, clocks, DNS, bandwidth/latency anomalies, etc.).
  • I/O benchmarks not running: Set IO_BENCHMARK_MOUNT to the target mount point.

License

This project is licensed under the terms described in the LICENSE file.