Data Extension via --data

July 31, 2026 · View on GitHub

Extend AICR's embedded recipe catalog with your own overlays, components, and criteria values at runtime — no fork, no rebuild. This is how operators add private/proprietary content (internal cloud providers, in-house GPU SKUs, commercial platforms, customer-specific scheduling) on top of the OSS catalog shipped with the binary.

The --data <dir> flag layers an external directory on top of the embedded catalog. The embedded catalog is precedence-low; your directory is precedence-high. Adding a file under the right path either supplements (for catalog content) or overrides (for component files) the embedded equivalent.

Use cases

NeedHow --data helps
Add a non-public Kubernetes service (service=ncp-internal)Drop an overlay declaring that criteria value; the criteria registry admits it.
Add a proprietary platform (platform=runai, platform=nvmesh)Same — an overlay's spec.criteria.platform registers the value.
Add a future GPU SKU before AICR's next releaseAdd accelerator: <name> in an overlay; CLI / API admit it on the fly.
Add an internal component (e.g., an in-house operator)Add a component definition under components/ and reference it in an overlay or mixin.
Override an embedded chart version / values fileDrop a same-path file under your --data dir; external takes precedence.
Customize per-deployment scheduling without an overlayUse --config aicr-config.yaml with bundle.scheduling.* — --data is for catalog content, not per-cluster ephemera.

Folder layout

The external directory mirrors AICR's embedded recipes/ tree. Drop only the paths you need; AICR loads any subset.

my-external-data/
├── registry.yaml             # REQUIRED — your component definitions
├── mixins/                   # Optional — composable overlay fragments
│   └── platform-internal.yaml
├── overlays/                 # Optional — your overlays (flat or subdirs ok)
│   └── ncp-internal-h100-training.yaml
├── components/               # Optional — Helm values, raw manifests, etc.
│   └── my-internal-operator/
│       ├── values.yaml
│       └── manifests/
│           ├── namespace.yaml
│           └── rbac.yaml
└── validators/               # Optional — validator catalog overrides
    └── catalog.yaml           # merged by validator name with the embedded catalog

The loader walks the tree recursively (filepath.WalkDir), so subdirectories inside overlays/ are supported and useful for organizing by service / customer / team:

overlays/
├── ncp-customer-a/
│   ├── h100-training.yaml
│   └── h100-inference.yaml
├── ncp-customer-b/
│   └── gb200-training.yaml
└── runai/
    └── h100-eks-training.yaml

registry.yaml is required

Even if your directory only adds overlays (no new components), AICR requires a registry.yaml at the root. The minimal stub is:

apiVersion: aicr.run/v1alpha2
kind: ComponentRegistry
components: []

External components in this file are merged with the embedded registry; on name collision, the external definition wins. An entry replaces its embedded counterpart wholesale — to override just helm.defaultVersion, copy the full embedded entry and change that one field, or the other coordinates (repository, chart, scheduling paths) are silently dropped.

Overriding a component's defaultVersion this way takes effect for every embedded overlay that references the component: embedded overlays do not pin versions that equal the registry default, so resolution falls back to your merged default. The exception is an overlay with an explicit, intentionally divergent pin — an explicit pin always wins over the registry default at resolution. (The versionPinExemptions list is CI policy for recipes embedded in the AICR repo, not a runtime input: the guard test that consumes it never walks --data trees, so your external overlays may pin freely without declaring anything or rebuilding AICR.) See issue #1616 and Chart Version Pinning.

Adding a criteria value

Criteria value validation (service, accelerator, intent, os, platform) is data-driven: the static OSS list is the fast path, and the runtime criteria registry picks up any value declared in a loaded overlay's spec.criteria. So adding a new value to an overlay automatically makes it a valid CLI / API input. No code change, no rebuild.

Example overlay for an internal NCP:

apiVersion: aicr.run/v1alpha2
kind: RecipeMetadata
metadata:
  name: ncp-internal-h100-training
spec:
  base: base
  criteria:
    service: ncp-internal       # NEW — registers as a valid --service value
    accelerator: h100
    intent: training
  componentRefs:
    - { name: gpu-operator }
    - { name: my-internal-operator }

Run it:

aicr recipe \
  --service ncp-internal \
  --accelerator h100 \
  --intent training \
  --data ./my-external-data \
  --output recipe.yaml

Without --data, --service ncp-internal is rejected (the value isn't in the embedded catalog and the registry hasn't been seeded). With --data pointing at the overlay above, the registry registers ncp-internal at catalog-load time and the CLI admits it.

The same applies to accelerator, intent, os, and platform — any field on a RecipeMetadata's spec.criteria.

Validating a recipe with a new criteria value. Most validation checks apply to an external recipe as-is: the deployment and conformance phases gate on component presence and cluster state, not criteria. The NCCL performance benchmarks additionally key their default applicability to embedded service + accelerator pairs, so a recipe with a new service or accelerator value would skip them (new intent, os, or platform values alone do not affect NCCL applicability) — declare an nccl-benchmark-profile performance constraint (e.g. gb200/eks) in the overlay's validation block to opt into one of the embedded benchmarks. The profile selects the benchmark template and fabric handling; node identification still follows the recipe's own accelerator. See Opting external recipes into a benchmark profile. If the private service's fabric matches none of the embedded templates, ship the benchmark itself: put a Kubeflow TrainingRuntime in your --data tree at validators/performance/testdata/{accelerator}/{service}/runtime.yaml and point the nccl-benchmark-runtime-ref constraint at it ({accelerator}/{service}). Pass the same --data <dir> to aicr validate so it can resolve the referenced file; validation then gates it keyed on the recipe's own criteria with no compiled entry required, and the file is a drop-in should the runtime ever be upstreamed. See Supplying a benchmark runtime for a private service.

Adding a component

registry.yaml declares the component's identity and source:

apiVersion: aicr.run/v1alpha2
kind: ComponentRegistry
components:
  - name: my-internal-operator
    displayName: My Internal Operator
    helm:
      defaultRepository: https://charts.example.com
      defaultChart: example/my-internal-operator
      defaultVersion: v1.2.3

…or, for a Kustomize-shipped component:

  - name: my-kustomize-app
    displayName: My Kustomize App
    kustomize:
      defaultSource: https://github.com/example/my-app
      defaultPath: deploy/production
      defaultTag: v1.0.0

Component values are not auto-discovered by filename. For a Helm component, a values file under components/<name>/ is consumed only when an overlay's componentRef names it via a valuesFile: path relative to the data directory (a componentRef — and thus valuesFile — lives on an overlay, not on the registry.yaml entry); the merge order is base values → valuesFile → inline overrides. For a Kustomize component, the deployable source is built from the registry entry's defaultSource / defaultPath / defaultTag (overridable per componentRef) — there is no implicit kustomization.yaml pickup.

Reference the component from an overlay's componentRefs: to include it in recipes that match the overlay's criteria.

Helm components must resolve with an effective chart version. Declare helm.defaultVersion in the registry entry (preferred — it keeps the version in one place, matching the embedded catalog's convention; see Chart Version Pinning), or pin version: on every componentRef that references the component. A Helm componentRef that resolves without one is rejected at recipe resolution (INVALID_REQUEST) rather than passed through — several deployers would otherwise emit the empty version verbatim and Helm would silently install "latest" at deploy time. Whitespace-only versions and a bare v count as absent — Flux and Argo CD strip a leading v for non-OCI outputs (Helm resolves the empty remainder as "latest"), non-vendored Helm/Helmfile and OCI outputs preserve it, and vendored wrappers substitute a fabricated default; a bare v is rejected uniformly to avoid output-dependent chart identities. Also, chart, source, and version values carrying surrounding whitespace are rejected outright, since deployers consume those fields verbatim. Manifest-only Helm components are exempt: a ref whose chart and source are both empty and that ships at least one primary manifestFiles entry has no chart version to pin. preManifestFiles alone do not qualify — pre-manifests are auxiliary to a primary release, so a ref with only pre-manifests is rejected as having no deployable primary.

Precedence rules

ResourceBehavior
registry.yamlMerged: embedded + external component lists. On name collision, external wins.
validators/catalog.yamlMerged: embedded + external validator lists, by validator name. A same-named external validator replaces the embedded one; new validators are appended.
Files in components/, mixins/, overlays/Replaced: any external file at the same relative path completely replaces the embedded equivalent. No partial-content merge.

When in doubt, aicr --debug recipe ... --data <dir> logs the resolved source (embedded / external / merged) for every loaded file.

Converting a family to a configuration profile (AKS example)

recipes/overlays/aks.yaml declares the gpuStack configuration profile (azure-managed default, operator-managed alternative) over the GPU driver/toolkit ownership paths (see Configuration Profiles). When an external data directory replaces a declaring overlay, or converts a family to a profile, the replacement rules above interact with the profile mechanics:

  • A same-path replacement replaces the declaration too. An external overlays/aks.yaml completely replaces the embedded file — including its spec.profile block. Keep the declaration in the replacement — dropping it while keeping apiVersion: aicr.run/v1alpha3 fails catalog validation (the version⟺declaration cross-check). That guardrail protects an integrator editing a v1alpha3 file: de-profiling one requires BOTH removing the declaration AND downgrading the overlay to the legacy apiVersion. It does NOT protect the upgrade path — a pre-existing legacy (aicr.run/v1alpha2) external overlays/aks.yaml, authored before the family's conversion, already satisfies both conditions. Upgrading AICR with such a catalog in --data silently preserves the unprofiled family: resolution succeeds with no error, no selectedProfile on the recipe, and no K8s.aks-gpu-pools.gpu-driver constraint. On upgrade, diff each external overlay against its embedded counterpart and port the spec.profile declaration plus the apiVersion bump (or consciously keep the fork unprofiled). A load-time detection of an external overlay shadowing an embedded profile declaration (with an explicit opt-out) is a candidate follow-up; it is not shipped.
  • Static assignments to now-owned paths are superseded. Descendant or external overlays that statically assign profile-owned paths (for the AKS declaration: gpu-operator driver.enabled, operator.runtimeClass, toolkit.enabled; nvidia-dra-driver-gpu nvidiaDriverRoot) are overwritten by the selected value's fragment at resolution. The enabled entries in ownedPaths are different: they are the SYNTHETIC component-presence locks — fragments are forbidden from assigning enabled, and the lock instead rejects removing (or bundle-subsetting away) an owned component. Review those overlays when converting a family — a static assignment that used to take effect no longer does.
  • Owned paths lock per surface. The enforcement matrix:
    1. aicr bundle static overrides (ANY static source — --set, --set-json, --set-file, or a config-file override) on an ordinary owned value path (e.g. driver.enabled, operator.runtimeClass): a divergent value fails closed; an identical value is accepted. For the synthetic enabled presence key, typed sources (--set-json, --set-file) are ALWAYS rejected — even when the value is identical to the selected one — because routing the toggle through a typed flag would write a stray literal enabled: chart value instead of toggling the component (pkg/cli/bundle_config.go). aicr mirror list exposes only the repeatable scalar --set (no typed flags), with the same identical-accepted / divergent-rejected rule; it does not apply a config file's spec.bundle.deployment.set overrides (pkg/cli/mirror.go).
    2. --dynamic exports: rejected on mere INTERSECTION with an owned path, regardless of value — install-time mutability of a locked path is itself the violation.
    3. argocd-helm install-time values: ANY install-time key whose path equals, contains, or is contained by an owned path fails closed at Helm render time, even when the value is identical — key presence alone trips the guard (pkg/bundler/deployer/argocdhelm/argocdhelm.go).
    4. Component presence (the synthetic enabled owned path): not changeable by profile reselection — fragments cannot assign enabled, so no --profile choice adds or removes a component; the lock rejects removing (or bundle-subsetting away) an owned component. Presence changes are catalog/composition changes. Reselecting a profile changes owned VALUE paths only.
  • Selection is explicit. Pick a value with aicr recipe --profile name=value (or spec.recipe.profile in --config); omission applies the declared default.
  • Evidence identity gains a profile path segment. A profiled recipe's evidence identity appends a lowercase -<name>-<value> segment, so per-value evidence lands in distinct directories rather than colliding on the criteria-only path.

Strict mode — gating the OSS catalog

--criteria-strict (or AICR_CRITERIA_STRICT=1, or spec.recipe.criteriaStrict: true in --config) rejects any criteria value not in the embedded OSS catalog, ignoring --data contributions entirely.

This is intended for CI gates in the OSS repo so the upstream catalog cannot accidentally start depending on internal-only values during development. Integrator workflows that legitimately need --data-supplied values should leave it off.

# Internal: accepts ncp-internal because --data registered it.
aicr recipe --service ncp-internal --data ./internal -o /dev/null

# OSS CI: rejects ncp-internal even though --data registered it,
# because strict mode hides external contributions.
AICR_CRITERIA_STRICT=1 \
  aicr recipe --service ncp-internal --data ./internal -o /dev/null
# → error: invalid service type: ncp-internal

make qualify in the OSS repo runs unit tests with AICR_CRITERIA_STRICT=1 exported automatically.

Verifying what loaded

Use aicr --debug to inspect external-data discovery and per-file source resolution:

aicr --debug recipe --service eks --accelerator h100 --data ./my-external-data

Sample output (truncated):

[cli] initializing external data provider: directory=./my-external-data
[cli] layered data provider initialized: external_dir=./my-external-data external_files=12
[cli] data provider set: generation=1
[cli] external data provider initialized successfully: directory=./my-external-data
[cli] building recipe from criteria: criteria=criteria(service=eks, accelerator=h100, intent=any, os=any)
[cli] recipe generation completed: output=stdout components=8 overlays=2

Tab-completion for --service / --accelerator / --os / --intent / --platform reflects values from the registry at the moment the help text is rendered. Run with --data early in the command line to populate it before shell completion kicks in.

Pinning your extension catalog

Treat your --data directory like any other artifact: tag it (git tag, OCI tag, semver) and pin which AICR binary version it was tested against. The overlay schema is the AICR YAML schema; bumping AICR may add new optional fields but rarely changes existing ones, so backward compatibility is the default — but check the AICR release notes when you upgrade the binary.

Typical organization patterns:

  • One repo per team / customer. Each team owns its overlay catalog and releases it independently of AICR.
  • One central internal repo. A single org-wide --data catalog with per-team subdirectories (overlays/team-a/, overlays/team-b/).
  • OCI distribution. Package the directory into an OCI artifact and pull on demand; aicr itself doesn't care about source, only that the path contains a registry.yaml and the expected sub-tree.