Validating a Cluster

Task-oriented walkthrough for running aicr validate against a GPU cluster — from capturing a snapshot through interpreting results. Covers both training and inference workloads and all three validation phases (deployment, conformance, performance).

For per-flag reference, see CLI reference: aicr validate. For the architectural view of how snapshot + recipe flow into the validator, see Data flow: Stage 3 Validate.

When to validate

Phase	What it answers	Typical trigger
deployment	Are the components the recipe asks for actually installed and healthy?	After ./deploy.sh finishes, before running any workload
conformance	Does the cluster support workload-specific capabilities (DRA, gang scheduling, autoscaling, ...)?	Before opening the cluster to real workloads
performance	Does the cluster hit expected bandwidth / throughput thresholds?	After components are ready; before going to production

Readiness pre-flight constraints (K8s version, OS, kernel) run implicitly before any phase. If pre-flight fails, no validator Jobs are deployed.

The workflow

  aicr snapshot ─┐
                 ├─▶ aicr validate ─▶ CTRF report
  aicr recipe ───┘                    (passed / failed / skipped per check)

1. Snapshot — capture current cluster state (K8s / OS / GPU / topology) once.
2. Recipe — generate the target configuration for your workload (training vs inference, platform, accelerator).
3. Validate — run one or all phases against the snapshot and live cluster.

Prerequisites

• aicr CLI installed (see installation).
• kubectl configured for the target cluster (validator dispatches K8s Jobs; pre-flight only needs the snapshot).
• Cluster service account with RBAC to create Jobs, ConfigMaps, and read cluster state (AICR creates its own aicr-validation namespace on first run).

Training performance validation

Training performance runs an NCCL all-reduce benchmark — a Kubeflow TrainJob that runs all_reduce_perf across GPU nodes and measures aggregate bus bandwidth. Three check variants are available; the recipe picks the one (or ones) that match the target fabric:

Check	Transport	Default applicability (from recipe criteria)
nccl-all-reduce-bw	Auto-detect (whatever NCCL picks)	H100/H200 on EKS, H100 on GKE, H100 on AKS (ND-series InfiniBand — NCCL's built-in IB/verbs transport over the rdma/hca_shared_devices_a shared device pool), and B200/GB200 on self-managed clusters (service=any). Preserves the pre-variant behavior.
nccl-all-reduce-bw-net	NET (EFA on EKS by default; ConnectX RoCE via AICR_NCCL_FABRIC=roce)	GB200 + EKS. Asserts EFA actually carried traffic — catches silent fallback to Socket when the NVIDIA driver is missing NVreg_GrdmaPciTopoCheckOverride=1.
nccl-all-reduce-bw-nvls	NVLS (MNNVL across an NVL72 IMEX domain)	GB200 + EKS, and GB200 + OKE. Asserts the NVLS communicator actually initialized — catches silent fallback to EFA (EKS) or Socket (OKE) when the IMEX domain is misconfigured.

The applicability column is the default, derived from the recipe's criteria. A recipe whose criteria fall outside it can still run these benchmarks explicitly — either by borrowing an embedded benchmark profile whose fabric matches its hardware, or, for a private service whose fabric matches no embedded template, by supplying its own benchmark runtime.

The -net check defaults to the AWS EFA fabric. On a ConnectX RoCE cluster (e.g. DGXC GB300 p6e-gb300r), set AICR_NCCL_FABRIC=roce in the aicr validate environment to run the NET test over NCCL's built-in IB/verbs transport across roce.networking.k8s.aws DRA devices instead. The value is scoped to the -net check only; unset (or efa) leaves every existing recipe on the EFA path unchanged, and any other value is rejected. The RoCE runtime image installs openssh-server at startup, so the GPU nodes need apt egress; on an air-gapped cluster the RoCE NET test cannot bootstrap. This env override is interim — snapshot-based fabric auto-detection (and removing the runtime package install once a CUDA-13 image ships sshd) is tracked in NVIDIA/aicr#1413.

GB200/EKS recipes (both training and inference intents) enable -net and -nvls together rather than the auto-detect variant, because those nodes expose two inter-node fabrics simultaneously and a single auto-detect test would only exercise one of them.

GB200/OKE recipes enable -nvls only: OKE NET/RDMA stays out of the support matrix until the OCI testbed proves a non-Socket NCCL transport end to end, so OKE validates the NVL72 IMEX fabric without an EFA/NET counterpart.

# Capture snapshot, generate training recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml

aicr recipe --service eks --accelerator h100 --os ubuntu \
            --intent training --platform kubeflow \
            --output recipe.yaml

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance

The generated recipe lists the selected variant(s) under validation.performance.checks with a platform-tuned bandwidth constraint (example: >= 300 GB/s for H100 + EFA; >= 40 GB/s NET and >= 500 GB/s NVLS for GB200 + EFA, each sized for a 2-node pair).

Node-shape assumption. These bus-bandwidth floors are fixed absolute values calibrated on full, high-bandwidth nodes (8-GPU H100 NVLink/SXM with multi-NIC transport). They are not normalized for node fabric or GPU count, so a smaller or different-fabric H100 SKU (e.g. a single-GPU-per-node shape) can false-fail a healthy run. Making the NCCL gate fabric/transport-class aware is tracked in #1256.

Expected flow (~5–10 min per variant): readiness pre-flight → deploy TrainingRuntime + TrainJob in aicr-validation → worker pods reach Running → run all_reduce_perf → parse peak bus bandwidth → verify the intended transport actually carried traffic (for -net / -nvls) → compare to recipe constraint (10 % tolerance) → cleanup.

A passing CTRF entry:

{
  "name": "nccl-all-reduce-bw-net",
  "status": "passed",
  "suite": ["performance"],
  "stdout": [
    "NCCL All Reduce bandwidth (nccl-all-reduce-bw-net): <actual> GB/s",
    "Constraint: >= <threshold> → true"
  ]
}

Note: this guide does not yet list per-platform expected-bandwidth baselines (EKS + EFA, GKE + TCPXO, AKS, etc.). The recipe's constraint value is the current pass/fail floor; measured values above that floor are treated as passing regardless of platform.

To run deployment validation first (recommended — verifies GPU Operator, DRA driver, and Kubeflow Trainer are installed and healthy before the benchmark):

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase deployment

Opting external recipes into a benchmark profile

The default applicability above is keyed to service + accelerator pairs the validator ships templates for. A recipe whose criteria fall outside that set — typically a criteria.service registered only through an external --data directory, or an embedded service extended to a new accelerator — would otherwise report the NCCL checks as skipped even when its validation block declares them. Such a recipe opts into an embedded benchmark with the nccl-benchmark-profile performance constraint (a bare {accelerator}/{service} value, no comparator):

validation:
  performance:
    checks:
      - nccl-all-reduce-bw-net
      - nccl-all-reduce-bw-nvls
    constraints:
      - name: nccl-benchmark-profile   # run the GB200-on-EKS benchmarks
        value: gb200/eks
      - name: nccl-all-reduce-bw-net
        value: ">= 40"
      - name: nccl-all-reduce-bw-nvls
        value: ">= 500"

The profile selects the benchmark's template and environment handling — transport-class runtime template, EFA/TCPXO discovery, worker scheduling defaults, and preflight checks — as if the cluster were the named pair, so pick the profile whose fabric matches the hardware. GPU nodes are still matched by the recipe's own criteria.accelerator (via the GFD nvidia.com/gpu.product label) when a matcher exists, but the profile's service keys the rest of node handling: an …/eks profile narrows workers to a single node.kubernetes.io/instance-type, and the …/any profiles (b200/any, gb200/any) require an explicit --node-selector identifying the GPU nodes, exactly as service: any recipes do. When --node-selector is passed it replaces the automatic filters rather than narrowing them.

Valid profiles are the pairs in the applicability table above: b200/any, gb200/any, gb200/eks, gb200/oke, h100/aks, h100/eks, h100/gke, h200/eks. A malformed or unknown value fails the check rather than silently skipping it. A valid profile that doesn't implement a requested variant (e.g. gb200/eks with the auto-detect nccl-all-reduce-bw check) skips just that check with a message naming the profile. When a profile is set it wins over criteria-derived applicability, and the pass/fail thresholds stay in the same-named check constraints as always.

Supplying a benchmark runtime for a private service

A profile only helps when an embedded template already matches the cluster's hardware and fabric. A private service introduced entirely through --data — a criteria.service (and possibly criteria.accelerator) the validator ships no template for, with a fabric that reuses none of the embedded ones — has no compiled pair to point a profile at. Such a recipe supplies the benchmark itself: it ships a Kubeflow TrainingRuntime as a file in its --data tree and references it with the nccl-benchmark-runtime-ref performance constraint (a bare {accelerator}/{service} value). aicr validate reads that file and gates the benchmark keyed on the recipe's own criteria, with no compiled applicability entry required.

Lay the runtime out exactly where the embedded templates live, so it is a drop-in for upstreaming later:

mydata/                                   # your --data directory
├── overlays/
│   └── mycloud-training.yaml             # the recipe overlay (below)
└── validators/performance/testdata/
    └── gb200/mycloud/runtime.yaml        # the TrainingRuntime, real file

# overlays/mycloud-training.yaml
validation:
  performance:
    checks:
      - nccl-all-reduce-bw
    constraints:
      - name: nccl-all-reduce-bw
        value: ">= 450"
      - name: nccl-benchmark-runtime-ref
        value: gb200/mycloud   # -> validators/performance/testdata/gb200/mycloud/runtime.yaml

# validators/performance/testdata/gb200/mycloud/runtime.yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
  name: nccl-all-reduce-runtime
spec:
  template:
    spec:
      replicatedJobs:
        - name: node          # required: the worker cohort
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: <your NCCL benchmark image>
                      resources:
                        limits:
                          nvidia.com/gpu: "${GPU_COUNT_PER_NODE}"
                      # ...command, fabric resources, sidecars...

aicr validate -r recipe.yaml -s snapshot.yaml \
              --data ./mydata --phase performance

The supplied runtime owns its fabric wiring end to end: because the recipe opted in explicitly, the validator bypasses the compiled applicability gate and skips every service-specific setup step — EFA/TCPXO/RDMA NIC discovery, the GB200-NVreg / GKE-TCPXO preflights, and NVLS/IMEX auto-provisioning. It renders the runtime, sizes it (against the runtime's own nodeSelector when it pins one, so WorkerCount matches placement), applies it alongside the shared TrainJob, and evaluates the bandwidth threshold from the launcher logs. Transport verification is not skipped: pairing the runtime with the -net or -nvls check still asserts that transport actually carried traffic (the NCCL markers are fabric-agnostic), so a named variant can't pass on bandwidth alone. What it must honor:

• It must be a Kubeflow TrainingRuntime (apiVersion: trainer.kubeflow.org/v1alpha1); any other kind/apiVersion is rejected. The validator force-sets its name and namespace, so a recipe can supply a TrainingRuntime and nothing else — never an arbitrary resource.
• It must declare a node replicatedJob (the worker cohort the shared TrainJob sizes and the validator injects scheduling into and reads logs from), mirroring the embedded templates.
• The generic template variables ${NAMESPACE}, ${WORKER_COUNT}, ${GPU_COUNT_PER_NODE}, ${GPU_COUNT}, ${TEST_TYPE}, ${MIN_MESSAGE_SIZE}, and ${MAX_MESSAGE_SIZE} are substituted; the service-specific ones (EFA/RDMA/GKE) are not, so the runtime must pin its own fabric resources.
• The runtime should pin its own worker nodeSelector/tolerations, or pass --node-selector to aicr validate, so workers land on the intended GPU nodes.

nccl-benchmark-runtime-ref and nccl-benchmark-profile are mutually exclusive — a recipe supplies its own runtime or borrows an embedded one, never both. An absent or blank ref falls back to criteria/profile-derived applicability. A malformed ref, or a file missing from --data, is rejected by aicr validate before any validator Job is deployed (a resolution error on stderr). A resolved file that is not a TrainingRuntime with a node replicatedJob fails the check itself (in the pod). Either way it fails rather than silently skipping.

Sizing note. The worker cohort is sized against the runtime's own nodeSelector when it pins one, but richer scheduling constraints (nodeAffinity, un-tolerated node taints) are the Kubernetes scheduler's job, not the validator's — a runtime that restricts placement below the sized cohort surfaces as a launcher timeout, not a false pass. Pass an explicit --node-selector when you want the sized and scheduled cohorts to match exactly.

Upstreaming. Because the file already sits at validators/performance/testdata/{accelerator}/{service}/runtime.yaml, adopting it into AICR is a copy: move the file into the repo, add the accelerator/service tuple to the compiled supportedNCCLCombinations matrix, and drop the nccl-benchmark-runtime-ref constraint — the benchmark then runs from the recipe's plain criteria like any other embedded one.

Inference performance validation

Inference performance runs the inference-perf check — deploys a DynamoGraphDeployment with a vLLM-served model (Qwen/Qwen3-8B by default, overridable per accelerator — see below) plus an AIPerf benchmark Job, and measures end-to-end output-token throughput and time-to-first-token (TTFT) p99.

Warm-up: AIPerf sends a wave of warm-up requests before the measured run, so vLLM's one-time CUDA-graph / JIT compilation (tens of seconds on a cold worker) is excluded from the reported throughput and p99 TTFT — the numbers reflect steady state, not cold start. Warm-up scales with concurrency and is tunable via AICR_INFERENCE_PERF_WARMUP_PER_CONCURRENCY (see the validator reference).

Determinism: the benchmark is driven reproducibly so the verdict reflects the deployment, not run-to-run RNG — a fixed random seed, fixed input/output token counts (stddev 0), a pinned synthetic-prompt pool, and greedy decoding (temperature: 0). Note that throughput (not the latency tail) is the stable, discriminating signal at high concurrency; TTFT p99 near the saturation knee can still vary with batching/scheduling, which is why the TTFT constraint is a generous ceiling rather than a tight target.

# Capture snapshot, generate inference recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml

aicr recipe --service eks --accelerator h100 --os ubuntu \
            --intent inference --platform dynamo \
            --output recipe.yaml

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance

The generated recipe includes dynamo-platform in componentRefs and lists inference-perf under validation.performance.checks with pass/fail constraints plus benchmark inputs:

validation:
  performance:
    checks: [inference-perf]
    constraints:
      # Pass/fail thresholds (10% tolerance applied by the evaluator). Values
      # shown are the measured H100 gate at 8B/256; each overlay sets its own.
      - name: inference-throughput   # output tokens/sec
        value: ">= 50000"
      - name: inference-ttft-p99     # time-to-first-token p99 in ms
        value: "<= 2000"
      # Optional per-accelerator inputs (bare value, no comparator).
      # Precedence: recipe > AICR_INFERENCE_PERF_* env > compiled default.
      - name: inference-model                # HF model ID; default Qwen/Qwen3-8B
        value: Qwen/Qwen3-8B
      - name: inference-concurrency-per-gpu  # positive integer; default 256
        value: "256"
      - name: inference-routing-mode         # dynamo-router or gateway-epp
        value: dynamo-router

Node-shape assumption. The inference-throughput floor is a fixed absolute full-node value calibrated on a full node (the shared >= 50000 gate was measured on 8-GPU H100; GB200 on a 4-GPU node). It is not normalized for GPU count, and the evaluator only scales it down for partial occupancy — not across node sizes — so a smaller H100 SKU (e.g. 1-/2-GPU shapes such as p5.4xlarge, AKS NC80adis) can false-fail a healthy run. inference-ttft-p99 is a per-request latency at fixed concurrency-per-GPU and does not need GPU-count normalization. A normalized per-GPU throughput floor is tracked in #1254.

inference-model and inference-concurrency-per-gpu are optional: omit them to use the compiled defaults (Qwen3-8B at 256 concurrent requests per GPU), set them per overlay to tune model and load for each accelerator, or override globally with the AICR_INFERENCE_PERF_MODEL / AICR_INFERENCE_PERF_CONCURRENCY_PER_GPU catalog knobs (recipe wins over catalog env wins over default).

inference-routing-mode selects the Dynamo 1.2 Kubernetes routing path. The default dynamo-router mode deploys a Dynamo frontend with load-aware least-loaded routing (DYN_ROUTER_MODE=least-loaded), which balances by each worker's active in-flight load so a transiently-slow worker stops receiving its full share (see issue #1197). Normal frontend-to-worker request/response traffic uses Dynamo's request plane (Dynamo 1.2 defaults to TCP); AICR does not set DYN_REQUEST_PLANE=nats. Workers still run the vLLM ZMQ KV-cache event publisher relayed onto the NATS event plane, but least-loaded routing does not consume those events. Set it to gateway-epp to exercise GAIE/EPP: the validator deploys an EPP component, worker frontend sidecars in direct mode, and an HTTPRoute through the AICR-managed inference gateway. The direct-mode sidecars honor EPP routing headers; they are not the ZMQ-to-NATS relay.

Model-weights cache and AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS. The benchmark downloads the model once into a PVC and serves all workers from it (on by default; avoids per-IP Hugging Face throttling). The cache PVC needs a StorageClass: it uses the cluster's default StorageClass unless you set AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS. On a cluster with no default StorageClass (common on EKS — e.g. only a non-default gp2) and no value set, the check fails fast in seconds with guidance rather than hanging; set AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS=<name> (e.g. gp2/gp3 on EKS, standard-rwo on GKE) on the inference-perf catalog entry's env (or via a catalog overlay in the aicr validate --data <dir> directory), or disable the cache with AICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off. Like the other AICR_INFERENCE_PERF_* knobs, this is a catalog/--data setting — it is not read from the shell environment of the process running aicr validate (only HF_TOKEN is). AICR-deployed EKS clusters get a default gp3 StorageClass from the aws-ebs-csi-driver component, so the cache works there with no knob.

Debugging a failed run with AICR_INFERENCE_PERF_NO_CLEANUP. By default the validator deletes the per-run namespace (DGD, workers, frontend, AIPerf Job) on both success and failure. To investigate a failure — e.g. a timed out waiting for inference endpoint to serve requests — set AICR_INFERENCE_PERF_NO_CLEANUP=1 and the validator leaves everything in place so you can kubectl logs the frontend/workers and curl /v1/models and /v1/chat/completions live. Unlike the other AICR_INFERENCE_PERF_* knobs, this one is read from the shell environment of the process running aicr validate (forwarded to the inference-perf pod, like HF_TOKEN), not from the catalog. Debug-only: you must delete the aicr-inference-perf-<suffix> namespace manually afterward, or it keeps GPU workers running.

Expected flow (~5–7 min on H100): readiness pre-flight → deploy a DynamoGraphDeployment in a per-run namespace aicr-inference-perf-<8-hex-suffix> → wait for state=successful (image pull

• model load) → endpoint readiness via a real /v1/chat/completions inference request (stricter than a /health probe, which returns 200 before the model can actually serve) → AIPerf benchmark Job parses throughput + TTFT p99 → compare to recipe constraints (10 % tolerance) → cleanup. Worker GPU wiring is configuration-selected and capability-verified for recipe-backed runs — the recipe's resolved allocation policy picks the mechanism, and the inspection probe (the same one the conformance checks use) verifies the cluster serves it, failing closed on mismatch; only recipe-less standalone runs (unspecified) select by capability. The wiring supports exactly two GPU allocation configurations: (1) nodes publishing node-local gpu.nvidia.com ResourceSlices (full-GPU DRA — the experimental opt-in; stock recipes default to device-plugin allocation since the #1327 flip), where workers bind a DRA ResourceClaimTemplate sized from the validated per-node device count — kai-scheduler treats nodes bearing raw node-local GPU ResourceSlices as DRA-only and rejects scalar GPU requests, so the claim path is the only schedulable wiring there; and (2) ComputeDomain-only / no full-GPU slices (device-plugin nodes — the production default), where workers request GPUs via nvidia.com/gpu limits, which need no gpu.nvidia.com DeviceClass. Anything outside those two states fails fast with an actionable error instead of guessing: full-GPU ResourceSlices from a non-NVIDIA GPU driver, allocated ResourceClaims requesting a non-NVIDIA "gpu"-named DeviceClass, gpu.nvidia.com slices using non-node-local topologies (nodeSelector/allNodes/per-device node selection), pools published by node-local slices of multiple nodes, and allocated ResourceClaims from pools no node-local slice publishes (occupancy is attributed to nodes through the slices' spec.nodeName — the K8s API does not require pool names to be node names) are all rejected — these are outside the NVIDIA DRA driver's supported configuration (#1327); generalized topology/driver support is tracked in #1652. When the recipe configures a GPU allocation policy (see Configured GPU allocation policy), the wiring is forced instead of capability-selected, and under dra-resource-claim candidate GPU nodes are discovered from the probe's validated DRA facts rather than scalar nvidia.com/gpu allocatable — so DRA-only nodes (device plugin disabled) are discoverable; full DRA-only aicr validate remains experimental (see the #1327 graduation checklist). The chosen node and wiring mode are recorded in the check's evidence output.

All Dynamo Frontend and worker pods pin to a single GPU node via kubernetes.io/hostname for a stable per-node baseline. On a shared cluster where some GPUs on a candidate node are already in use by other workloads, the validator picks the candidate with the most free GPUs within the node's own allocation ledger and sizes the benchmark to that count: DRA ResourceClaim allocations subtract from DRA capacity, and device-plugin nvidia.com/gpu occupancy from device-plugin capacity. A DRA-wired candidate carrying any scalar nvidia.com/gpu workload is skipped entirely — the device plugin's physical device assignments are invisible to the DRA allocator, so claims placed there could double-book a plugin-held GPU. The check therefore does not need an explicit hostname override to avoid saturated nodes. The inference-throughput gate is a full-node baseline, so when the benchmark runs on fewer than the node's full GPU count the gate is scaled down by the same freeGPUs / nodeGPUs fraction (throughput scales ~linearly at fixed concurrency-per-GPU) — a healthy per-GPU result on a partially occupied node is not failed against a full-node number. TTFT p99 is a per-request latency and is not scaled. Concurrent aicr validate invocations are isolated from each other by the run-specific suffix on both the namespace and the inner AIPerf Job name.

A passing CTRF entry (measured on EKS H100, 8 × H100 GPUs, Qwen/Qwen3-8B at 256 concurrency/GPU):

{
  "name": "inference-perf",
  "status": "passed",
  "suite": ["performance"],
  "stdout": [
    "RESULT: Inference throughput: 108789.87 tokens/sec",
    "RESULT: Inference TTFT p99: 687.50 ms",
    "Throughput constraint: >= 50000 → PASS",
    "TTFT p99 constraint: <= 2000 → PASS"
  ]
}

The RESULT: prefix on the first two lines is the contract documented in pkg/validator/validator.go — any check that wants its summary lines echoed to the CLI's own output (not just the CTRF report) opts in by emitting that prefix. The validator runtime strips the prefix when echoing; the full prefixed line stays in stdout[].

To run deployment validation first (recommended — verifies GPU Operator, DRA driver, Dynamo operator, KAI scheduler, and supporting components are installed and healthy):

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase deployment

Skip scenarios

The inference validator has three explicit skip guards so it never runs where it can't succeed. Each produces a status: skipped CTRF entry with a specific reason. Skipped checks are not failures: the validator container exits with code 2 internally (mapped to CTRF skipped), but aicr validate itself exits 0 for skipped/passed/other phases — a skipped inference check never drives a non-zero CLI exit on its own.

Guard	Trigger	Skip message
A	Recipe lists inference-perf in checks: but no matching inference-throughput / inference-ttft-p99 constraints	no inference-throughput or inference-ttft-p99 constraint in recipe
B	inference-perf is selected but dynamo-platform is not in recipe componentRefs	skipped - dynamo-platform not in recipe components
C	dynamo-platform is declared but the DynamoGraphDeployment CRD is not installed on the cluster (operator not deployed yet)	skipped - DynamoGraphDeployment CRD not installed on cluster (dynamo-platform component declared but operator not deployed yet)

Guards fire before any cluster mutation, so skips are cheap (typically < 10 s).

Configured GPU allocation policy

When you validate with a recipe, AICR resolves a whole-GPU allocation policy from the recipe's fully hydrated component values and carries it to the validators (#1327):

Policy	Meaning
device-plugin-extended-resource	Whole GPUs via the device plugin (nvidia.com/gpu requests)
dra-resource-claim	Whole GPUs via DRA (gpu.nvidia.com ResourceClaims)
dra-extended-resource	Reserved (KEP-5004 mapped extended resource) — not yet validated
unspecified	No recipe context (standalone validator runs): capability-driven automatic selection

The nvidia-dra-driver-gpu value resources.gpus.enabled is the switch: true resolves dra-resource-claim; explicit false — or the component absent or disabled — resolves device-plugin-extended-resource. On an enabled DRA component the switch must be explicitly set: the upstream chart's declared default is true, so an absent value would diverge from what Helm deploys. Three configurations are rejected at resolution time with an invalid-request error: an enabled nvidia-dra-driver-gpu component with resources.gpus.enabled absent (pin it explicitly in the recipe; stock recipes always do), gpus.enabled=true without gpuResourcesEnabledOverride=true (the upstream chart install guard refuses it), and no whole-GPU advertiser remaining — gpus.enabled off with the GPU operator component (gpu-operator, or gpu-operator-ocp on OpenShift recipes) absent, disabled, or carrying devicePlugin.enabled=false. Two further states are likewise rejected (they warned during the transition to the device-plugin production default and are errors since the flip): dual advertisement (both mechanisms enabled — exactly one whole-GPU advertiser is required) and an inert gpuResourcesEnabledOverride=true with gpus.enabled=false (the waiver would disarm the upstream chart's install-guard tripwire). Stock recipes ship the production default: gpus.enabled=false, gpuResourcesEnabledOverride=false, and devicePlugin.enabled=true; the experimental DRA opt-in flips all three together in a recipe overlay.

Upgrading a cluster from the dual-advertised (pre-flip) configuration: applying the flipped bundle does not drain existing workloads — a running full-GPU gpu.nvidia.com claim pod keeps its prepared GPU while the nvidia.com/gpu ledger, which cannot see that assignment, admits newly converted scalar workloads onto the same device. Migrate in this order:

1. Stop/delete all workloads holding full-GPU gpu.nvidia.com ResourceClaims (ComputeDomain/IMEX claims are unaffected and stay).
2. Confirm no allocated or reserved full-GPU claims remain, using the driver-specific, fail-closed drain check below. On IMEX platforms, allocated compute-domain.nvidia.com claims legitimately remain, so a plain kubectl get resourceclaims -A can never come back empty — filter on the allocation driver instead, and treat ANY output (or any query failure) as unsafe to proceed.
3. Apply the flipped bundle (upgrade nvidia-dra-driver-gpu).
4. Wait until the gpu.nvidia.com ResourceSlices and DeviceClass disappear (compute-domain.nvidia.com slices must remain).
5. Confirm scalar nvidia.com/gpu allocatable is present on the GPU nodes.
6. Only then start the scalar (device-plugin) workloads.

The step-2 drain check — capture and test each stage separately (a piped kubectl ... | jq ... masks a failed List as an empty, safe-looking result), and proceed only when it exits 0:

claims="$(kubectl get resourceclaims -A -o json)" || { echo "claim query FAILED — do not proceed"; exit 1; }
unsafe="$(printf '%s' "$claims" | jq -r '
  .items[]
  | select([.status.allocation.devices.results[]?.driver] | index("gpu.nvidia.com"))
  | "\(.metadata.namespace)/\(.metadata.name) reservedFor=\([.status.reservedFor[]?.name] | join(","))"')" \
  || { echo "claim filter FAILED — do not proceed"; exit 1; }
[ -z "$unsafe" ] || { printf 'unsafe full-GPU claims remain:\n%s\n' "$unsafe"; exit 1; }
echo "no full-GPU gpu.nvidia.com claims — safe to proceed"

While migrating, also delete any pending (unallocated) claims whose spec references the gpu.nvidia.com DeviceClass: they hold no GPU (no over-admission risk, so the check above rightly ignores them), but the flip removes that DeviceClass, leaving such claims — and any pods referencing them — stranded unschedulable forever.

Validators compare the configured policy against the inspected cluster state and fail closed on mismatch — a cluster that cannot serve the configured mechanism is a validation failure by design, never a silent fallback to the other mechanism. Only recipe-less standalone runs (unspecified) keep the capability-driven automatic selection.

Running all phases

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml
# equivalent to: --phase deployment --phase conformance --phase performance

Phases run sequentially. By default all phases run and produce results regardless of earlier failures. Pass --fail-fast to stop after the first phase that fails (e.g., to skip a 65-minute inference-perf run when deployment already failed).

Scoping CNCF submission evidence to specific features

The --feature flag scopes which CNCF AI conformance features get behavioral evidence collected. It only applies to the CNCF-submission evidence collector and is rejected by the CLI unless --cncf-submission is also set (which in turn requires --evidence-dir). It does not scope the regular --phase conformance validator run — that one always evaluates every check defined in the recipe.

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
  --phase conformance \
  --cncf-submission \
  --evidence-dir ./evidence \
  --feature dra-support --feature gang-scheduling

Empty --feature (the default) collects evidence for every feature.

Valid feature names (from pkg/evidence/cncf/collector.go):

Name	What it checks
dra-support	Dynamic Resource Allocation driver and ResourceSlices
gang-scheduling	Gang-scheduler presence and PodGroup support
secure-access	Cluster authn/authz posture for AI workloads
accelerator-metrics	GPU metrics exporter and Prometheus scrape config
ai-service-metrics	Inference-service metrics via custom-metrics API
inference-gateway	Gateway API + Inference Extension installation; also records the gateway LoadBalancer network exposure (open 0.0.0.0/0 vs scoped source ranges). Fails on an open gateway only when AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY=true.
robust-operator	Operator readiness and leader-election posture
pod-autoscaling	HPA-driven pod autoscaling: external GPU metric + behavioral scale-up/down test (pod-scoped custom metrics collected best-effort — absent for DRA-allocated GPUs, not a failure)
cluster-autoscaling	Karpenter (preferred) or EKS managed node-group autoscaling fallback

Evidence verdicts and exit status

Each collected feature records exactly one verdict in its evidence file:

• PASS — the capability was exercised and behaved as expected.
• SKIP — an optional prerequisite is absent (e.g. no inference gateway, no supported operator, an unsupported cluster-autoscaling provider, or an operator that is installed but has no workload to reconcile). A SKIP is not a failure.
• FAIL — a present capability was unhealthy, a required query failed, or the evidence file carried no single valid verdict (fail closed).

A FAIL in any feature makes --cncf-submission exit non-zero, so a submission that records a genuine failure no longer reports overall success. Absent optional prerequisites remain successful SKIPs and do not fail the run.

Emitting recipe evidence

When a recipe PR targets hardware AICR maintainers cannot independently re-run, the contributor needs to attach a signed evidence bundle so a maintainer can verify the recipe offline. aicr validate produces the bundle as a side effect when --emit-attestation is set; adding --push signs it (cosign keyless via Sigstore) and uploads it to an OCI registry. This is a different artifact from the CNCF-submission evidence above — the two flag families produce independent outputs and may run from a single aicr validate invocation.

aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<owner>/aicr-evidence

The --push tag is just a human-readable label — the sha256: digest is what pins the bundle, so tag choice never affects verification (the verifier pulls by digest). Omit the tag, as above, and aicr derives a unique per-recipe one, <recipe-slug>-<short-fingerprint> (e.g. ghcr.io/<owner>/aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e), so distinct attestations never collide on a shared tag. Pass an explicit tag to override.

After the command finishes:

./out
├── pointer.yaml                  # locator; copy into recipes/evidence/
└── summary-bundle/
    ├── recipe.yaml               # canonical post-resolution recipe
    ├── snapshot.yaml             # snapshot at validate-time (minimized by default)
    ├── bom.cdx.json              # CycloneDX BOM (auto-generated from
    │                             #   recipe + validator catalog when
    │                             #   --bom is omitted)
    ├── ctrf/                     # per-phase test results (per-test stdout/message omitted by default)
    ├── manifest.json             # per-file sha256 inventory
    ├── statement.intoto.json     # unsigned in-toto Statement
    └── attestation.intoto.jsonl  # signed (when --push is set)

The bundle is minimized by default: snapshot.yaml keeps only an allowlisted set of fields (dropping node names, provider instance IDs, the node label/taint set, OS tuning, loaded modules, and systemd config) and the CTRF reports omit per-test stdout/message. The signed predicate records the applied policy in a redaction block, and the bundle self-verifies exactly like a full one. Pass --full to publish the raw payloads instead.

Commit pointer.yaml to its per-source path recipes/evidence/<recipe>/<src>/<digest>.yaml — the emit output prints the exact copyTo path, and <src> is the slug derived from your signer identity (see Artifact Verification). The bundle itself lives in OCI. Then self-verify before opening the PR — the same verifier runs against the committed pointer in the CI gate, so exit 0 locally means the gate will pass:

aicr evidence verify recipes/evidence/<recipe>/<src>/<digest>.yaml

Flag reference:

Flag	What it does
--emit-attestation <dir>	Write the bundle to <dir>. Required to produce evidence. The bundle is minimized by default — see --full.
--full	Emit the full (unredacted) bundle. By default the snapshot is reduced to an allowlisted set of fields and per-test CTRF stdout/message are omitted, keeping node names, provider instance IDs, the node label/taint set, OS tuning, and raw container logs out of the published artifact. Minimal bundles record the policy in predicate.redaction and self-verify normally.
--push <oci-ref>	Sign via cosign keyless OIDC and push to the registry. The digest pins the bundle, so the tag is just a label; omit it and aicr derives a unique per-recipe tag (<recipe-slug>-<short-fingerprint>). Pass an explicit tag to override. Without --push, the bundle is unsigned (development/self-debug only).
--bom <path>	Embed an existing CycloneDX BOM instead of the auto-generated one. Pass make bom output for an exhaustive BOM that includes chart-default sub-images.
--identity-token <token>	Pre-fetched OIDC identity token, skipping the browser flow. Reads COSIGN_IDENTITY_TOKEN.
--oidc-device-flow	Use OAuth device-code flow instead of opening a browser. Reads AICR_OIDC_DEVICE_FLOW.
--plain-http	HTTP instead of HTTPS (local-registry tests only).
--insecure-tls	Skip TLS verification (self-signed registries).

Registry requirements: the registry must support the OCI 1.1 Referrers API (or its tag-schema fallback) so the Sigstore Bundle can be attached to the artifact. Known-good registries: GHCR, GitLab Container Registry, Harbor (≥ 2.8), AWS ECR, Google Artifact Registry, Azure Container Registry, JFrog Artifactory. Without referrer support the bundle pushes but the signature is not discoverable, and the verifier records signature-verify as "skipped (unsigned)" even on a signed bundle.

OIDC token resolution. --push resolves an identity token through this precedence chain: --identity-token (or COSIGN_IDENTITY_TOKEN) → ambient GitHub Actions OIDC (ACTIONS_ID_TOKEN_REQUEST_URL present) → --oidc-device-flow (or AICR_OIDC_DEVICE_FLOW=true) → interactive browser. CI pipelines typically rely on the ambient GitHub Actions path; local workstations get the browser flow.

Local-only mode (no registry access). Omitting --push still produces a complete bundle on disk — the verifier records the signature step as "skipped (unsigned)" and the manifest-hash chain becomes self-consistency only. Useful for catching accidental corruption during development, but unsuitable for the CI gate, which requires a signed bundle bound to a pointer.

For the full producer-and-consumer walkthrough — including OCI-only verification, the tamper demo, and JSON output for CI gates — see Recipe Evidence Demo. For the bundle format and verifier semantics, see ADR-007. For the maintainer-side review checklist, see Maintaining Recipe Contributions. For the per-flag reference on aicr evidence verify, see CLI reference.

Input modes

Snapshot and recipe can come from a file, an HTTPS URL, or a Kubernetes ConfigMap:

# File (default)
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml

# HTTPS URL
aicr validate \
  --recipe https://artifacts.example.com/recipes/h100-eks-inference.yaml \
  --snapshot https://artifacts.example.com/snapshots/prod-cluster.yaml

# Kubernetes ConfigMap (for in-cluster operators)
aicr validate \
  --recipe cm://gpu-operator/aicr-recipe \
  --snapshot cm://gpu-operator/aicr-snapshot

The ConfigMap form is useful when the snapshot is captured by an in-cluster agent — see agent deployment.

Dry-run mode

--no-cluster runs the validator against the snapshot alone, skipping all Kubernetes API calls. Declarative constraints still evaluate; behavioral checks report skipped - no-cluster mode (test mode).

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --no-cluster

Useful for CI pipelines that validate a recipe against a captured snapshot without needing cluster access.

CI/CD integration

aicr validate exits non-zero when any phase fails. CTRF JSON is emitted to stdout (or to --output <file>), so a pipeline can gate promotion on both the exit code and the structured report:

aicr validate \
  --recipe recipe.yaml \
  --snapshot cm://gpu-operator/aicr-snapshot \
  --output ctrf.json

Exit codes follow Unix conventions and are derived from the CLI's structured error codes (see pkg/errors/exitcode.go):

Code	Meaning
0	All phases reported status passed, skipped, or other
2	Invalid input or request (ErrCodeInvalidRequest) — bad CLI flag, malformed argument, or a validator rejecting a recipe value (e.g., an inference constraint that uses the wrong comparator direction)
5	CLI-layer timeout before a check runs — snapshot-agent Job never completes within --timeout, or the validator Job as a whole exceeds its wait deadline
8	One or more phases reported status failed, including per-check internal timeouts (e.g., DynamoGraphDeployment not ready within InferenceWorkloadReadyTimeout)

Important: two quirks to be aware of when gating a pipeline on exit code:

1. Only phase status failed drives a non-zero exit. A phase whose status is other (check crashed, pod OOM, activeDeadlineSeconds exceeded) still produces exit 0. Pipelines that need to catch those outcomes must inspect the CTRF report and look at per-phase status or the summary.other count, not rely on exit code alone.
2. Exit 5 is narrower than it sounds. A timeout inside a check's own logic (DynamoGraphDeployment not ready, inference endpoint never healthy, AIPerf Job pod-wait deadline) surfaces as a failed phase, not as a structured ErrCodeTimeout, so the CLI exits 8. Only timeouts at the CLI-to-cluster layer (snapshot-agent wait, validator-Job wait) retain their ErrCodeTimeout classification all the way through to exit 5.

Scripts that gate on validation outcome should treat any non-zero code as failure rather than branching on specific values, and should additionally check CTRF summary.failed and summary.other for a complete picture.

For informational-only runs (report results without failing the build):

aicr validate ... --fail-on-error=false

Troubleshooting

Readiness pre-flight fails

The CLI logs each readiness constraint comparison before any phase runs:

readiness constraint failed: name=K8s.server.version expected=">= 1.34" actual=v1.33.0-eks-abc

Fix: upgrade the cluster, or pick a recipe whose readiness constraints match the cluster's actual versions.

Non-standard GPU labels or taints

Default GPU-node discovery looks for nodeGroup, node.kubernetes.io/instance-type, or GPU-related label substrings. If your cluster uses custom labels, override the scheduling of inner workloads with --node-selector and --toleration:

aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml --phase performance \
  --node-selector my-org/gpu-pool=h100 \
  --toleration dedicated=worker-workload:NoSchedule \
  --toleration dedicated=worker-workload:NoExecute

These flags affect the inner benchmark pods that run on GPU nodes (NCCL workers, Dynamo workers), not the validator orchestrator Job itself. For inference-perf specifically, --node-selector narrows the pool of candidate GPU nodes — the validator then picks the candidate with the most free GPUs (subtracting same-ledger occupancy only — DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity — and skipping DRA candidates that carry scalar nvidia.com/gpu workloads) and pins all Dynamo Frontend + worker pods to that node via kubernetes.io/hostname. The AIPerf benchmark runner pod is CPU-only, uses a tolerate-all / no-nodeSelector pod spec, and is unaffected by these flags.

A check reports skipped unexpectedly

Skips are always deliberate and always carry a reason, but the location of the reason in the CTRF entry depends on how the skip happened:

• Check-level skips (the CheckFunc ran and returned validators.Skip(reason) — e.g., Guards A/B/C on inference, --no-cluster from inside a check): reason appears in stdout as level=INFO msg=SKIP reason="…".
• Phase-level skips (the CheckFunc never ran — e.g., with --fail-fast, a prior phase failed so subsequent phases synthesize skip entries; also --no-cluster for checks that the runner marks skipped before dispatch): reason appears in message, not stdout.

Common reasons and their cause:

Reason (excerpt)	Where it appears	Meaning	Fix
no inference-throughput or inference-ttft-p99 constraint in recipe	stdout	Check was invoked but recipe is missing the matching constraints	Re-generate the recipe or add the constraints
dynamo-platform not in recipe components	stdout	Inference check selected but dynamo-platform absent from componentRefs	Use --platform dynamo when generating the recipe
DynamoGraphDeployment CRD not installed	stdout	Recipe declares dynamo-platform but the operator is not deployed	Run aicr bundle + ./deploy.sh first, or wait for bootstrap to complete
requires Service + Accelerator to be implemented	stdout	The recipe's criteria are outside the NCCL benchmark's default applicability (e.g. a service registered via --data)	Add an nccl-benchmark-profile constraint — see Opting external recipes into a benchmark profile — or, if no embedded template fits, an nccl-benchmark-runtime-ref constraint — see Supplying a benchmark runtime for a private service
benchmark profile … does not implement the … NCCL variant	stdout	The declared nccl-benchmark-profile is valid but has no template for this check variant	Drop the non-applicable check from validation.performance.checks, or pick a profile that implements it
failed to resolve nccl-benchmark-runtime-ref=… expected … in the --data tree	stderr	The referenced runtime file is missing from --data (or --data wasn't passed)	Place the TrainingRuntime at validators/performance/testdata/{accelerator}/{service}/runtime.yaml in your --data dir — see Supplying a benchmark runtime for a private service
nccl-benchmark-runtime … must be a … TrainingRuntime … must declare a "node" replicatedJob	stdout	The referenced file is not a Kubeflow trainer.kubeflow.org/v1alpha1 TrainingRuntime, or lacks the node replicatedJob	Fix the runtime file to be a valid TrainingRuntime with a node replicatedJob
nccl-benchmark-runtime and nccl-benchmark-profile are mutually exclusive	stdout	The recipe declares both escape hatches at once	Keep only one: borrow an embedded profile or supply your own runtime
skipped - no-cluster mode	message	--no-cluster was passed — the runner short-circuits every phase before dispatching any Job	Remove the flag to run behavioral checks
skipped due to previous phase failure	message	--fail-fast was set and an earlier phase failed, so subsequent phases were skipped	Fix the earlier phase first, or drop --fail-fast to run all phases regardless

ai-service-metrics fails with "Prometheus unreachable"

On EKS clusters that split worker and system pods across separate security groups (e.g. DGXC EKS with distinct customer/system ENI subnets), the conformance check ai-service-metrics can fail non-deterministically with:

[SERVICE_UNAVAILABLE] Prometheus unreachable at http://kube-prometheus-prometheus.monitoring.svc:9090 — verify network connectivity

The validator orchestrator Job tolerates every taint and sets a preferred dependencyAffinity toward Prometheus, so the scheduler co-locates it with the Prometheus pod when possible. The preference is best-effort, so on fallback it can still land on any worker node — including one whose ENI is in a security group whose ingress to the Prometheus-hosting SG is missing or asymmetric. On such a fallback the outcome is not stable across re-runs: image-locality scoring tends to keep the pod on whatever node won the first scheduling decision, so a passing run on a fresh cluster does not prove the SG topology is correct.

This is a cluster-side prerequisite, not an AICR bug per se — see EKS Dynamo Networking Prerequisites for the SG ingress rules required for Prometheus (tcp/9090). The preferred dependencyAffinity (#933, resolved) makes a bad placement far less likely, but the 9090 SG rule remains the reliable guarantee since the affinity is best-effort.

Workaround when SG changes are not available: re-run the check until the orchestrator lands on a node whose SG can reach Prometheus, then leave the image cached there so image-locality keeps subsequent runs on the same node. This is unreliable and should not be used as the steady-state validation strategy.

Benchmark Job stuck or timed out

Each performance check has a Job-level activeDeadlineSeconds set by the catalog's timeout:. For inference-perf, the full pipeline (workload ready → endpoint health → benchmark) can take up to 30 min on cold-start clusters. If it still times out:

# validator orchestrator Job + AIPerf benchmark Job both live in aicr-validation.
# The orchestrator is named aicr-inference-perf-<hex> (random suffix per run);
# the AIPerf Job is named aicr-aiperf-<run-id-hash>.
kubectl -n aicr-validation get jobs | grep -E 'aicr-inference-perf-|aicr-aiperf-'

# tail each by full job name (label selectors require exact match)
kubectl -n aicr-validation logs -l job-name=aicr-inference-perf-<hash> --tail=200
kubectl -n aicr-validation logs -l job-name=aicr-aiperf-<run-id-hash>  --tail=200

# the Dynamo workload (DynamoGraphDeployment, Frontend, worker pods)
# lives in a separate per-run namespace:
kubectl get ns | grep aicr-inference-perf-
# (resourceclaimtemplates is populated in DRA wiring mode; empty in
# device-plugin mode — both are normal)
kubectl -n aicr-inference-perf-<suffix> get dynamographdeployments,pods,svc,resourceclaimtemplates

Common causes: image pull throttling, vLLM model load slowness, and every eligible candidate GPU node being saturated by existing workloads. In the saturated case the validator fails fast with a message like no eligible candidate GPU node has free GPUs (N matched; eligible candidates are saturated by existing workloads in the selected mechanism's ledger ...). Occupancy is subtracted per allocation ledger (DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity), and nodes excluded for other reasons — NotReady, probe-ineligible, kai-blocked raw slices, or DRA-capable but carrying scalar nvidia.com/gpu workloads — are not counted as saturated; the fix is to free GPUs on one of the candidate nodes, or to pass --node-selector kubernetes.io/hostname=<node> to target a specific node you know is free. GPU-occupancy accounting fails closed rather than treating GPUs as free: a pod list failure always fails the check, and so does any error listing DRA resourceclaims (e.g. RBAC denied, timeout) — with one exception: when the resource.k8s.io API is not served at all (NotFound), which deterministically means zero DRA usage, the validator proceeds with device-plugin pod requests alone.

Related

• CLI reference: aicr validate — full flag reference and per-command examples
• CLI reference: aicr snapshot — snapshot capture options
• CLI reference: aicr recipe — recipe generation flags
• Agent deployment — capture snapshots via an in-cluster Job
• Data flow: Stage 3 Validate — how the validator engine is built
• Validator Development Guide — add a new validator (contributor-facing)