README.md

Description

This module simplifies the following functionality:

• Applying Kubernetes manifests to GKE clusters: It provides flexible options for specifying manifests, allowing you to either directly embed them as strings content or reference them from URLs, files, templates, or entire .yaml and .tftpl files in directories.
• Deploying commonly used infrastructure like Kueue,Jobset, NCCL gIB plugin or asapd-lite daemonset.

Note: Kueue can work with a variety of frameworks out of the box, find them here

Explanation

• Manifest:

  • Raw String: Specify manifests directly within the module configuration using the content: manifest_body format.

  • File/Template/Directory Reference: Set source to the path to:

    • A single URL to a manifest file. Ex.: https://github.com/.../myrepo/manifest.yaml.

    Note: Applying from a URL has important limitations. Please review the Considerations & Callouts for Applying from URLs section below.

    • A single local YAML manifest file (.yaml or .yml). Ex.: ./manifest.yaml.
    • A template file (.tftpl) to generate a manifest. Ex.: ./template.yaml.tftpl. You can pass the variables to format the template file in template_vars.
    • A directory containing multiple YAML or template files. Ex: ./manifests/ or ./manifests. The module correctly identifies directories even if the trailing slash is omitted. For security and stability, the module only processes files with .yaml, .yml, or .tftpl extensions. Other files in the directory (like README.md etc. ) are automatically ignored.

Manifest Example

- id: existing-gke-cluster
  source: modules/scheduler/pre-existing-gke-cluster
  settings:
    project_id: $(vars.project_id)
    cluster_name: my-gke-cluster
    region: us-central1

- id: kubectl-apply
  source: modules/management/kubectl-apply
  use: [existing-gke-cluster]
  settings:
    - content: |
        apiVersion: v1
        kind: Namespace
        metadata:
          name: my-namespace
    - source: "https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml"
    - source: $(ghpc_stage("manifests/configmap1.yaml"))
    - source: $(ghpc_stage("manifests/configmap2.yaml.tftpl"))
      template_vars: {name: "dev-config", public: "false"}
    - source: $(ghpc_stage("manifests"))/
      template_vars: {name: "dev-config", public: "false"}

Pre-build infrastructure Example

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      kueue:
        install: true
        config_path: $(ghpc_stage("manifests/user-provided-kueue-config.yaml"))
      jobset:
        install: true

The config_path field in kueue installation accepts a template file, too. You will need to provide variables for the template using config_template_vars field.

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      kueue:
        install: true
        config_path: $(ghpc_stage("manifests/user-provided-kueue-config.yaml.tftpl"))
        config_template_vars: {name: "dev-config", public: "false"}
      jobset:
        install: true

TPU Flavor Quotas Override (Pathways active)

When Pathways is active, the TPU worker pods request both TPU and CPU/Memory resources. To prevent Kueue from blocking these pods, the default configuration uses high "unlimited" defaults (999999 and 999999T) for CPU and Memory quotas on the TPU flavor.

If you want to enforce strict capacity sharing on the TPU pool, you can override these defaults by specifying tpu_flavor_cpu_quota and tpu_flavor_memory_quota inside config_template_vars:

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      kueue:
        install: true
        config_template_vars:
          tpu_flavor_cpu_quota: 1024
          tpu_flavor_memory_quota: "4096G"

You can specify a particular kueue version that you would like to use using the version flag. By default, we recommend customers to use v0.17.1. You can find the list of supported kueue versions here.

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      kueue:
        install: true
        version: 0.17.1
        config_path: $(ghpc_stage("manifests/user-provided-kueue-config.yaml.tftpl"))
        config_template_vars: {name: "dev-config", public: "false"}
      jobset:
        install: true

You can also install the gib plugin by setting the gib input variable. The path field accepts a template file. You will need to provide variables for the template using template_vars field and can also specify a particular gib version that you would like to use using the version flag. You can find the list of supported machine types for the gib plugin here.

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      gib:
        install: true
        path: $(ghpc_stage("manifests/daemonset-gib.yaml.tftpl"))
        template_vars:
          version: v1.1.0
          accelerator_count: 2

You can install the asapd-lite daemonset for A4X-Max Bare Metal (gke-a4x-max-bm) by setting the asapd_lite input variable and providing the path to the installer manifest using the config_path field.

  - id: workload_component_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      asapd_lite:
        install: true
        config_path: $(ghpc_stage("manifests/asapd-lite-installer.yaml"))

NOTE:

The project_id and region settings would be inferred from the deployment variables of the same name, but they are included here for clarity.

Terraform may apply resources in parallel, leading to potential dependency issues. If a resource's dependencies aren't ready, it will be applied again up to 15 times.

Callouts

Helm-based Manifest Application

1. Large Manifests and CRDs

Helm stores the entire release state (including the generated manifests) as a standard Kubernetes Secret in the release namespace. Before storing the state, Helm runs the YAML through GZIP compression and base64 encoding. This effectively raises the limit to ~1MB or more, allowing for the deployment of very large manifests and complex CRDs without requiring Server-Side Apply (SSA). This behaviour is guaranteed because the Terraform Helm Provider directly imports the official Helm Go SDK.

2. Helm-Release Suffixes

To make releases more identifiable, the module generates deterministic Helm release names based on the following precedence hierarchy:

• If you provide a name field in the apply_manifests list object, it will be used directly. Explicit names must be unique across the list.
• If applying from a local file or URL, it extracts the file basename and removes common extensions like .yaml, .yml, and .tftpl (including combined extensions like .yaml.tftpl).
• For raw content without a source or name, it falls back to using the module ID and a short hash: ${module_id}-raw-${hash}.The result is truncated to 30 characters, and a short 7-character hash of the manifest configuration is appended to ensure uniqueness. This ensures the total length does not exceed Helm's 53-character limit.

3. Re-deployment Conflicts

If a deployment fails, the atomic = true setting ensures that Helm automatically rolls back the release, preventing the cluster from being left in a "half-applied" state. If you encounter persistent conflicts during re-deployment due to immutable fields, you may need to manually delete the resource or the Helm release before re-applying.

Applying Manifests from URLs: Considerations & Callouts

While this module supports applying manifests directly from remote http:// or https:// URLs, this method introduces complexities not present when using local files. For production environments, we recommend sourcing manifests from local paths or a version-controlled Git repository. Moreover, this method will be deprecated soon. Hence we recommend to use other methods to source manifests.

If you choose to use the URL method, be aware of the following potential issues and their solutions.

1. Apply Order and Race Conditions

The module applies manifests from the apply_manifests list in parallel. This can create a race condition if one manifest depends on another. The most common example is applying a manifest with custom resources (like a ClusterQueue) at the same time as the manifest that defines it (the CustomResourceDefinition or CRD).

There is no guarantee that the CRD will be applied before the resource that uses it. This can lead to non-deterministic deployment failures with errors like:

Error: resource [kueue.x-k8s.io/v1beta2/ClusterQueue] isn't valid for cluster

Recommended Workaround: Two-Stage Apply

To ensure a reliable deployment, you must manually enforce the correct order of operations.

1. Initial Deployment: In your blueprint, include only the manifest(s) containing the CustomResourceDefinition (CRD) resources in the apply_manifests list.

   Example settings for the first run:

   settings:
     apply_manifests:
     # This manifest contains the CRDs for Kueue
     - source: "https://raw.githubusercontent.com/GoogleCloudPlatform/cluster-toolkit/refs/heads/develop/modules/management/kubectl-apply/manifests/kueue-v0.11.4.yaml"
   

2. Run the deployment (gcluster deploy or terraform apply).

3. Second Deployment: Once the first apply is successful, add the manifests containing your custom resources (like ClusterQueue, LocalQueue) to the list.

   Example settings for the second run:

   settings:
     apply_manifests:
     # The CRD manifest is still present
     - source: "https://raw.githubusercontent.com/GoogleCloudPlatform/cluster-toolkit/refs/heads/develop/modules/management/kubectl-apply/manifests/kueue-v0.11.4.yaml"
   
     # Now, add your configuration manifest
     - source: "https://gist.githubusercontent.com/YourUser/..." # Your configuration URL
   

4. Run the deployment command again. Since the CRDs are now guaranteed to exist in the cluster, this second apply will succeed reliably.

2. Terraform Template Files (.tftpl)

• Limitation: This module cannot render a template file (.tftpl) when sourced from a remote URL.
• Workaround: You must render the template into a pure YAML file locally, host that rendered file at a URL, and provide the URL of the rendered file in your blueprint.

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Providers

Modules

Resources

Inputs

list(object({
    name             = optional(string, null)
    enable           = optional(bool, true)
    content          = optional(string, null)
    source           = optional(string, null)
    template_vars    = optional(map(any), null)
    wait_for_rollout = optional(bool, true)
    namespace        = optional(string, null)
  }))

object({
    install              = optional(bool, false)
    config_path          = optional(string, null)
    config_template_vars = optional(map(any), {})
  })

object({
    install = optional(bool, false)
    version = optional(string, "v1.17.2")
  })

object({
    install = bool
    path    = string
    template_vars = object({
      image   = optional(string, "us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib")
      version = string
      node_affinity = optional(any, {
        requiredDuringSchedulingIgnoredDuringExecution = {
          nodeSelectorTerms = [{
            matchExpressions = [{
              key      = "cloud.google.com/gke-gpu",
              operator = "In",
              values   = ["true"]
            }]
          }]
        }
      })
      accelerator_count = number
      max_unavailable   = optional(string, "50%")
    })
  })

{
  "install": false,
  "path": "",
  "template_vars": {
    "accelerator_count": 0,
    "version": ""
  }
}

object({
    install = optional(bool, false)
    version = optional(string, "v25.3.0")
  })

object({
    install           = optional(bool, false)
    version           = optional(string, "0.10.1")
    controller_cpu    = optional(string, null)
    controller_memory = optional(string, null)
  })

object({
    # ATTENTION: If you update the KUEUE's default version below, please also update the corresponding
    # defaultKueueVersion constant in pkg/orchestrator/gke/infra_manager.go. (note the 'v' prefix there)
    version                         = optional(string, "0.17.1")
    install                         = optional(bool, false)
    config_path                     = optional(string, null)
    config_template_vars            = optional(map(any), null)
    enable_pathways_for_tpus        = optional(bool, false)
    enable_dynamic_slicing_for_tpus = optional(bool, false)
    accelerator_topology_mode       = optional(string, null)
    machine_type                    = optional(string, null)
    controller_cpu                  = optional(string, null)
    controller_memory               = optional(string, null)
    controller_replicas             = optional(number, null)
    slice_controller_cpu_request    = optional(string, "8000m")
    slice_controller_memory_request = optional(string, "16Gi")
    slice_controller_cpu_limit      = optional(string, "12000m")
    slice_controller_memory_limit   = optional(string, "32Gi")
  })

object({
    install          = optional(bool, false)
    version          = optional(string, "v25.3.0")
    accelerator_type = optional(string, "nvidia-gb200")
  })

map(object({
    namespace                 = string
    gcp_service_account_email = string
  }))

Outputs