README.md

Description

This module creates a Google Kubernetes Engine (GKE) node pool.

NOTE: This is an experimental module and the functionality and documentation will likely be updated in the near future. This module has only been tested in limited capacity.

Example

The following example creates a GKE node group.

  - id: compute_pool
    source: modules/compute/gke-node-pool
    use: [gke_cluster]

Also see a full GKE example blueprint.

Taints and Tolerations

By default node pools created with this module will be tainted with user-workload=true:NoSchedule to prevent system pods from being scheduled. User jobs targeting the node pool should include this toleration. This behavior can be overridden using the taints setting. See docs for more info.

Local SSD Storage

GKE offers two options for managing locally attached SSDs.

The first, and recommended, option is for GKE to manage the ephemeral storage space on the node, which will then be automatically attached to pods which request an emptyDir volume. This can be accomplished using the local_ssd_count_ephemeral_storage variable.

The second, more complex, option is for GCP to attach these nodes as raw block storage. In this case, the cluster administrator is responsible for software RAID settings, partitioning, formatting and mounting these disks on the host OS. Still, this may be desired behavior in use cases which aren't supported by an emptyDir volume (for example, a ReadOnlyMany or ReadWriteMany PV). This can be accomplished using the local_ssd_count_nvme_block variable.

The local_ssd_count_ephemeral_storage and local_ssd_count_nvme_block variables are mutually exclusive and cannot be mixed together.

Also, the number of SSDs which can be attached to a node depends on the machine type.

See docs for more info.

Considerations with GPUs

When a GPU is attached to a node an additional taint is automatically added: nvidia.com/gpu=present:NoSchedule. For jobs to get placed on these nodes, the equivalent toleration is required. The gke-job-template module will automatically apply this toleration when using a node pool with GPUs.

Nvidia GPU drivers must be installed. The recommended approach for GKE to install GPU drivers is by applying a DaemonSet to the cluster. See these instructions.

However, in some cases it may be desired to compile a different driver (such as a desire to install a newer version, compatibility with the Nvidia GPU-operator or other use-cases). In this case, ensure that you turn off the enable_secure_boot option to allow unsigned kernel modules to be loaded.

Maximize GPU network bandwidth with GPUDirect and multi-networking

For A3 Series machines to achieve optimal performance , GKE provide two networking stacks for remote direct memory access (RDMA):

• A3 High machine types (a3-highgpu-8g): utilize GPUDirect-TCPX to reduce the overhead required to transfer packet payloads to and from GPUs, which significantly improves throughput at scale compared to GPUs that don't use GPUDirect.
• A3 Mega machine types (a3-megagpu-8g): utilize GPUDirect-TCPXO to improve GPU to GPU communication, and further improves GPU to VM communication.

To achieve this, when creating nodepools with A3 Series machine type, pass in a multivpc module to the gke-node-pool module, and the gke-node-pool module would detect the eligible machine type and enable GPUDirect for it. More specifically, the below components will be installed in the nodepool for enabling GPUDirect.

• Install NCCL plugin for GPUDirect TCPX or TCPXO
• Install NRI device injector plugin
• Provide support for injecting GPUDirect required components(annotations, volumes, rxdm sidecar etc.) into the user workload in the form of Kubernetes Job.
  • Provide sample workload to showcase how it will be updated with the required components injected, and how it can be deployed.
  • Allow user to use the provided script to update their own workload and deploy.

The GPUDirect supports included in the Cluster Toolkit aim to automate the GPUDirect User Guid and provide better usability.

NOTE: You must enable multi networking feature when creating the GKE cluster. When gke-cluster depends on multivpc (with the use keyword), multi networking will be automatically enabled on the cluster creation. When gke-cluster or pre-existing-gke-cluster depends on multivpc (with the use keyword), the network objects required for multi networking will be created on the cluster.

GPUs Examples

There are several ways to add GPUs to a GKE node pool. See docs for more info on GPUs.

The following is a node pool that uses a2, a3 or g2 machine types which has a fixed number of attached GPUs, let's call these machine types as "pre-defined gpu machine families":

  - id: simple-a2-pool
    source: modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g

Note: It is not necessary to define the guest_accelerator setting when using pre-defined gpu machine families as information about GPUs, such as type, count and gpu_driver_installation_config, is automatically inferred from the machine type. guest_accelerator block is specified, optional fields such as gpu_partition_size need to be specified only if they have non-default values.

The following scenarios require the guest_accelerator block is specified:

• To partition an A100 GPU into multiple GPUs on an A2 family machine.
• To specify a time sharing configuration on a GPUs.
• To attach a GPU to an N1 family machine.

The following is an example of partitioning an A100 GPU:

Note: In the following example, type, count and gpu_driver_installation_config are picked up automatically.

  - id: multi-instance-gpu-pool
    source: modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g
      guest_accelerator:
      - gpu_partition_size: 1g.5gb

The following is an example of GPU time sharing (with partitioned GPUs):

  - id: time-sharing-gpu-pool
    source: modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g
      guest_accelerator:
      - gpu_partition_size: 1g.5gb
        gpu_sharing_config:
          gpu_sharing_strategy: TIME_SHARING
          max_shared_clients_per_gpu: 3

Following is an example of using a GPU attached to an n1 machine:

  - id: t4-pool
    source: modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: n1-standard-16
      guest_accelerator:
      - type: nvidia-tesla-t4
        count: 2

The following is an example of using a GPU (with sharing config) attached to an n1 machine:

  - id: n1-t4-pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      name: n1-t4-pool
      machine_type: n1-standard-1
      guest_accelerator:
      - type: nvidia-tesla-t4
        count: 2
        gpu_driver_installation_config:
          gpu_driver_version: "LATEST"
        gpu_sharing_config:
          max_shared_clients_per_gpu: 2
          gpu_sharing_strategy: "TIME_SHARING"

Finally, the following is adding multivpc to a node pool:

  - id: network
    source: modules/network/vpc
    settings:
      subnetwork_name: gke-subnet
      secondary_ranges:
        gke-subnet:
        - range_name: pods
          ip_cidr_range: 10.4.0.0/14
        - range_name: services
          ip_cidr_range: 10.0.32.0/20

  - id: multinetwork
    source: modules/network/multivpc
    settings:
      network_name_prefix: multivpc-net
      network_count: 8
      global_ip_address_range: 172.16.0.0/12
      subnetwork_cidr_suffix: 16

  - id: gke-cluster
    source: modules/scheduler/gke-cluster
    use: [network, multinetwork]
    settings:
      cluster_name: $(vars.deployment_name)

  - id: a3-megagpu_pool
    source: modules/compute/gke-node-pool
    use: [gke-cluster, multinetwork]
    settings:
      machine_type: a3-megagpu-8g
      ...

Using GCE Reservations

You can reserve Google Compute Engine instances in a specific zone to ensure resources are available for their workloads when needed. For more details on how to manage reservations, see Reserving Compute Engine zonal resources.

After creating a reservation, you can consume the reserved GCE VM instances in GKE. GKE clusters deployed using Cluster Toolkit support the same consumption modes as Compute Engine: NO_RESERVATION(default), ANY_RESERVATION, SPECIFIC_RESERVATION.

This can be accomplished using reservation_affinity.

# Target any reservation
reservation_affinity:
  consume_reservation_type: ANY_RESERVATION

# Target a specific reservation
reservation_affinity:
  consume_reservation_type: SPECIFIC_RESERVATION
  specific_reservations:
  - name: specific-reservation-1

The following requirements need to be satisfied for the node pool nodes to be able to use a specific reservation:

1. A reservation with the name must exist in the specified project(var.project_id) and one of the specified zones(var.zones).
2. Its consumption type must be specific.
3. Its GCE VM Properties must match with those of the Node Pool; Machine type, Accelerators (GPU Type and count), Local SSD disk type and count.

If you want to utilise a shared reservation, the owner project of the shared reservation needs to be explicitly specified like the following. Note that a shared reservation can be used by the project that hosts the reservation (owner project) and by the projects the reservation is shared with (consumer projects). See how to create and use a shared reservation. Note that the shared reservation must be a specific reservation.

reservation_affinity:
  consume_reservation_type: SPECIFIC_RESERVATION
  specific_reservations:
  - name: specific-reservation-shared
    project: shared_reservation_owner_project_id

Target a future reservation OR reservations that are not fulfilled yet

To create a GKE nodepool with a future reservation or reservations that are not fulfilled yet, set the is_reservation_active input variable to false. Note that to use this variable, these input variables should not be set or be set to 0 or null: static_node_count, autoscaling_min_node_count, autoscaling_max_node_count and initial_node_count.

is_reservation_active: false
reservation_affinity:
  consume_reservation_type: SPECIFIC_RESERVATION
  specific_reservations:
  - name: future_or_placeholder_reservation_name

Once the future reservation is active or the reservation is fulfilled, set the is_reservation_active input variable to true. Also set the node count information with either the static_node_count to define the required number of static nodes, or the autoscaling_min_node_count and autoscaling_max_node_count to use node scaling. Remember to use the -w flag in the gcluster deploy command and DO NOT change the deployment_name variable.

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Providers

Modules

Resources

Inputs

list(object({
    network            = string
    subnetwork         = string
    subnetwork_project = string
    network_ip         = string
    nic_type           = string
    stack_type         = string
    queue_count        = number
    access_config = list(object({
      nat_ip       = string
      network_tier = string
    }))
    ipv6_access_config = list(object({
      network_tier = string
    }))
    alias_ip_range = list(object({
      ip_cidr_range         = string
      subnetwork_range_name = string
    }))
  }))

list(object({
    type  = optional(string)
    count = optional(number, 0)
    gpu_driver_installation_config = optional(object({
      gpu_driver_version = string
    }), { gpu_driver_version = "DEFAULT" })
    gpu_partition_size = optional(string)
    gpu_sharing_config = optional(object({
      gpu_sharing_strategy       = string
      max_shared_clients_per_gpu = number
    }))
  }))

object({
    sysctls = optional(map(string), {
      "net.ipv4.tcp_rmem" = "4096 87380 16777216"
      "net.ipv4.tcp_wmem" = "4096 16384 16777216"
    })
    hugepages_config = optional(object({
      hugepage_size_2m = optional(number)
      hugepage_size_1g = optional(number)
    }))
  })

object({
    type         = string
    name         = optional(string)
    tpu_topology = optional(string)
  })

{
  "name": null,
  "tpu_topology": null,
  "type": null
}

object({
    consume_reservation_type = string
    specific_reservations = optional(list(object({
      name    = string
      project = optional(string)
    })))
  })

{
  "consume_reservation_type": "NO_RESERVATION",
  "specific_reservations": []
}

object({
    email  = string,
    scopes = set(string)
  })

[
  "https://www.googleapis.com/auth/cloud-platform"
]

list(object({
    key    = string
    value  = any
    effect = string
  }))

object({
    strategy        = string
    max_surge       = optional(number)
    max_unavailable = optional(number)
  })

{
  "max_surge": 0,
  "max_unavailable": 1,
  "strategy": "SURGE"
}

Outputs