Contributing to NVIDIA AI Cluster Runtime (AICR)

June 30, 2026 · View on GitHub

We welcome contributions from developers of all backgrounds and experience levels.

Code of Conduct

This project follows NVIDIA's commitment to fostering an open and welcoming environment. Please be respectful and professional in all interactions. See CODE_OF_CONDUCT.md for details.

Getting Started

Before contributing:

  1. Read the README.md to understand the project
  2. Check existing issues to avoid duplicates
  3. Review the security policy for security-related contributions
  4. Set up your development environment following DEVELOPMENT.md
  5. If using coding assistants, review AGENTS.md for project rules and workflows

How to Contribute

Reporting Bugs

  • Use the bug report template
  • Describe the issue clearly with steps to reproduce
  • Include system information (OS, Go version, Kubernetes version)
  • Attach logs or screenshots if applicable
  • Check if the issue already exists before creating a new one

Suggesting Enhancements

  • Use the feature request template
  • Clearly describe the proposed feature and its use case
  • Explain how it benefits the project and users
  • Provide examples or mockups if applicable

Improving Documentation

  • Fix typos, clarify instructions, or add examples
  • Update README.md for user-facing changes
  • Update API documentation when endpoints change
  • Ensure code comments are accurate and helpful

Contributing Code

  • Fix bugs, add features, or improve performance
  • Follow the development workflow in DEVELOPMENT.md
  • Ensure all tests pass and code meets quality standards
  • Write tests for new functionality

Go dependencies (vendor)

This project vendors Go dependencies. After changing go.mod or go.sum, run make tidy (which runs go mod vendor) and commit go.mod, go.sum, and the vendor/ directory. CI will fail if vendor/ is out of sync.

Adding Validation Constraints

AICR uses a validator framework to check cluster state against requirements. To add new validation constraints:

Quick Start:

# Generate all necessary files
make generate-validator ARGS="--constraint Deployment.my-app.version --phase deployment --description 'Validates my-app version'"

This creates three files with TODOs guiding implementation:

  • Helper functions with validation logic
  • Unit tests with table-driven test cases
  • Integration test with automatic registration

Next Steps:

  1. Implement the TODOs in generated files
  2. Add comprehensive test cases
  3. Run make test - registration validation ensures completeness
  4. Submit PR - CI enforces all requirements

See docs/contributor/validator.md for complete guide with examples, architecture overview, and troubleshooting.

Adding a Component

AICR components are declarative — add an entry to recipes/registry.yaml with Helm or Kustomize settings, create a values.yaml, and optionally add a health check. No Go code needed.

Validate your component:

make build
make component-test COMPONENT=my-component

This auto-detects the right test tier, creates a Kind cluster, deploys the component, and runs its health check. See tools/component-test/README.md for details.

Design Principles

These principles guide all design decisions in AICR. When faced with trade-offs, these principles take precedence.

Local Development Equals CI

The same tools, same versions, and same validation run locally and in CI.

What: Tool versions are centralized in .settings.yaml. make tools-setup (first-time install), make tools-update (upgrade to current pins), make tools-check (verify), and GitHub Actions all use this single source of truth. make qualify runs the exact same checks as CI.

Why: "Works on my machine" is not acceptable. If a contributor can run make qualify locally and it passes, CI will pass. This eliminates surprise failures and reduces feedback loops. Note that this only holds when your local toolchain matches .settings.yaml — run make tools-update after a git pull that touches .settings.yaml, or whenever make tools-check shows a ⚠ for a lint-sensitive tool. A behind-CI golangci-lint will silently miss lint findings that CI catches.

Adoption Comes from Idiomatic Experience

The system integrates into how users already work. We provide validated configuration, not a new operational model.

What: AICR outputs standard formats (Helm values, Kubernetes manifests) that work with existing tools (kubectl, Argo CD, Flux). Users don't need to learn "the AICR way" of deploying.

Why: If adoption requires retraining users on a new workflow, our design has failed. Value comes from correctness, not from lock-in.

Correctness Must Be Reproducible

Given the same inputs, the same system version must always produce the same result (e.g. recipe, bundle artifacts).

What: No hidden state, no implicit defaults, no non-deterministic behavior. A recipe/bundle/image digest generated using the same version of aicr today must be identical to one generated tomorrow.

Why: Reproducibility is a prerequisite for debugging, validation, and trust. If users can't reproduce a result, they can't trust it.

Metadata Is Separate from Consumption

Validated configuration exists independent of how it is rendered, packaged, or deployed.

What: Recipes define what is correct. Bundlers and deployers determine how to deliver it (Helm, Argo CD, raw manifests). The recipe doesn't change based on the deployment mechanism.

Why: This prevents tight coupling of correctness to a specific tool, workflow, or delivery mechanism. Users can adopt new deployment tools without re-validating their configurations.

Recipe Specialization Requires Explicit Intent

More specific recipes are never matched unless explicitly requested. Generic intent cannot silently resolve to specialized configurations.

What: If a user requests a "training" recipe, they get the training configuration. The system never silently upgrades to a more specific variant (e.g., "training-distributed-horovod") without explicit opt-in.

Why: This prevents accidental misconfiguration and preserves user control. Surprises in infrastructure configuration are dangerous.

Trust Requires Verifiable Provenance

Trust is established through evidence, not assertions. Every released artifact carries verifiable proof of origin and build process.

What: All releases include SLSA build provenance (build level under review, #1536), SBOM attestations, and Sigstore signatures. Users can verify exactly which commit, workflow, and build produced any artifact.

Why: This underpins supply-chain security, compliance, and confidence. "Trust us" is not a security model.

Pull Request Process

Before Submitting

  1. Ensure all checks pass:

    make qualify

  2. Update documentation if needed:

    • README.md for user-facing changes
    • DEVELOPMENT.md for developer workflow changes
    • Code comments and godoc for API changes

  3. Sign and sign off every commit: all contributors use git commit -s -S — -s adds the DCO sign-off, -S cryptographically signs the commit. See Developer Certificate of Origin for one-time setup and what it certifies.

Creating the Pull Request

  1. Push your branch and open a PR against main
  2. Fill out the PR template completely:
    • Summary: Brief description of changes
    • Type of Change: Bug fix, feature, breaking change, etc.
    • Testing: What testing was performed
    • Checklist: Verify all items
  3. Do not use the issue priority labels P0, P1, or P2 on PRs. They are reserved for issues and are automatically removed from pull requests by automation.

Review Process

  1. Automated Checks run via GitHub Actions — the same gate make qualify runs locally (tests with race detector, golangci-lint, YAML linting, security scan, coverage, E2E). See Full Qualification in the development guide.

  2. Maintainer Review covers:

    • Correctness and functionality
    • Code style and Go idioms
    • Test coverage and quality
    • Documentation completeness

  3. Address Feedback by pushing new commits (signed and signed off, same as every commit):

    git commit -s -S -m "address review: improve error handling"
git push origin your-branch

  4. Merge: Once approved and CI passes, a maintainer will merge

AI-Assisted Contributions Policy

We welcome the use of AI tools (e.g., GitHub Copilot, ChatGPT, Claude) to help you write code, brainstorm, or refactor. However, we maintain a strict human-in-the-loop policy for all submissions:

  • Full accountability: By submitting a PR, you (the human author) accept full responsibility for the code — its correctness, security, maintainability, and license compliance. "The AI wrote it" is not an acceptable explanation for bugs or security flaws.
  • Understand what you submit: Do not submit AI-generated code you do not fully understand. Reviewers expect you to explain and defend every line of code in your PR.

Issue and PR Lifecycle

Automated bots manage the lifecycle of issues and pull requests:

DayAction
0Issue/PR opened, needs-triage label added to issues
14Inactive PRs receive a reminder comment
30Inactive PRs marked lifecycle/stale
44Stale PRs auto-closed
60Inactive issues marked lifecycle/stale
74Stale issues auto-closed
90+Closed issues/PRs locked

To prevent auto-close: Add the lifecycle/frozen label. PRs with do-not-merge are also exempt.

Claiming an Issue

To pick up an unassigned issue, comment /assign and a bot assigns it to you (or /assign @user to assign someone else). Issues use a single-owner model: if one is already assigned, the bot refuses and asks the current assignee to release it first with /unassign. Comment /unassign to release an issue assigned to you — it only ever removes your own claim. GitHub only allows assigning users with triage/write access or prior activity in the repository; the bot comments if it cannot assign a requested user.

After Merging

# Update your local repository
git checkout main
git pull upstream main

# Delete your feature branch
git branch -d your-branch
git push origin --delete your-branch

Developer Certificate of Origin

Every commit — from every contributor — must be both signed off and cryptographically signed:

git commit -s -S -m "Your commit message"
  • -s (lowercase) adds a Signed-off-by line, certifying the Developer Certificate of Origin 1.1 below.
  • -S (uppercase) attaches a GPG or SSH signature, proving the commit came from you.

The two are independent — use both, every time. A branch ruleset enforces Require signed commits on every branch, so a push containing an unsigned (-S-less) commit is rejected before review. The Signed-off-by line looks like:

Signed-off-by: Jane Developer <jane@example.com>

One-Time Setup

# Identity used in the Signed-off-by line (must match your signing key)
git config user.name "Your Name"
git config user.email "your.email@example.com"

# Sign every commit by default, so you only need -s going forward
git config commit.gpgsign true

You also need a signing key registered with GitHub. Follow GitHub's guide to generate a GPG or SSH signing key and add it to your account, then point git at it (git config user.signingkey <key>).

Forgot to Sign or Sign Off?

Fix the most recent commit and re-push:

git commit --amend -s -S --no-edit
git push --force-with-lease origin your-branch

For an entire branch, re-sign every commit at once:

git rebase --exec 'git commit --amend -s -S --no-edit' origin/main
git push --force-with-lease origin your-branch

What You're Certifying

By signing off, you certify the Developer Certificate of Origin 1.1:

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Tips for Contributors

First-Time Contributors

Recommended starting points:

  1. Start with issues labeled good first issue
  2. Read existing code in the package you're modifying before writing
  3. Run make tools-check to verify your environment; run make tools-update if any tool is behind
  4. Study the Design Principles section

Good first contributions:

  • Documentation improvements (typos, clarifications)
  • Adding test cases to existing tests
  • Improving error messages with better context

Writing Good Commit Messages

Short summary (50 chars or less)

More detailed explanation if needed. Wrap at 72 characters.
Explain the problem being solved and why this approach was chosen.

- Bullet points are fine
- Use present tense ("Add feature" not "Added feature")
- Reference issues: "Fixes #123" or "Related to #456"

Signed-off-by: Your Name <your@email.com>

Code Style

  • Follow existing patterns in the codebase
  • Use pkg/errors for error handling (not fmt.Errorf)
  • Always check ctx.Done() in loops and long operations
  • Write table-driven tests for multiple test cases
  • Use functional options for configuration

Getting Help

  • GitHub Issues: Create an issue with the "question" label
  • Existing Issues: Search for similar questions first
  • Recent PRs: Look at merged PRs for examples

Additional Resources