Custom Images in the Cluster Toolkit (formerly HPC Toolkit)

Please review the introduction to image building for general information on building custom images using the Toolkit.

Introduction

This module uses Packer to create an image within an Cluster Toolkit deployment. Packer operates by provisioning a short-lived VM in Google Cloud on which it executes scripts to customize the boot disk for repeated use. The VM's boot disk is specified from a source image that defaults to the HPC VM Image. This Packer "template" supports customization by the following approaches following a recommended use:

• startup-script metadata from raw string or file
• Shell scripts uploaded from the Packer execution environment to the VM
• Ansible playbooks uploaded from the Packer execution environment to the VM

They can be specified independently of one another, so that anywhere from 1 to 3 solutions can be used simultaneously. In the case that 0 scripts are supplied, the source boot disk is effectively copied to your project without customization. This can be useful in scenarios where increased control over the image maintenance lifecycle is desired or when policies restrict the use of images to internal projects.

Minimum requirements

Outbound internet access

Most customization scripts require access to resources on the public internet. This can be achieved by one of the following 2 approaches:

1. Using a public IP address on the VM

• Set var.omit_external_ip to false

1. Configuring a VPC with a Cloud NAT in the region of the VM

• Use the vpc module which automates NAT creation

Inbound internet access

Read order of execution below for a discussion of VM customization solutions and their requirements for inbound SSH access. Environments without SSH access should use the metadata-based startup-script solution.

A simple way to enable inbound SSH access is to use the VPC module with allowed_ssh_ip_ranges set to 0.0.0.0/0.

User or service account executing Packer at command line

The user or service account running Packer must have the permission to create VMs in the selected VPC network and, if use_iap is set, must have the "IAP-Secured Tunnel User" role. Recommended roles are:

• roles/compute.instanceAdmin.v1
• roles/iap.tunnelResourceAccessor

VM service account roles

The service account attached to the temporary build VM created by Packer should have the ability to write Cloud Logging entries so that you may inspect and debug build logs. When using the metadata startup-script customization solution, the service account attached to the temporary build VM created by Packer must have the permission to modify its own metadata and to read from Cloud Storage buckets. Recommended roles are:

• roles/compute.instanceAdmin.v1
• roles/iam.serviceAccountUser
• roles/logging.logWriter
• roles/monitoring.metricWriter
• roles/storage.objectViewer

It is recommended to create this service account as a separate step outside a blueprint due to known delay in IAM bindings propagation.

Example blueprints

A recommended pattern for building images with this module is to use the terraform based startup-script module along with this packer custom-image module. Below you can find links to several examples of this pattern, including usage instructions.

Image Builder

The Image Builder blueprint demonstrates a solution that builds an image using:

• The HPC VM Image as a base upon which to customize
• A VPC network with firewall rules that allow IAP-based SSH tunnels
• A Toolkit runner that installs a custom script

Please review the examples README for usage instructions.

Order of execution

The startup script specified in metadata executes in parallel with the other supported methods. However, the remaining methods execute in a well-defined order relative to one another.

1. All shell scripts will execute in the configured order
2. After shell scripts complete, all Ansible playbooks will execute in the configured order

NOTE: if both startup_script and startup_script_file are specified, then startup_script_file takes precedence.

Recommended use

Because the metadata startup script executes in parallel with the other solutions, conflicts can arise, especially when package managers (yum or apt) lock their databases during package installation. Therefore, it is recommended to choose one of the following approaches:

1. Specify either startup_script or startup_script_file and do not specify shell_scripts or ansible_playbooks.
   • This can be especially useful in environments that restrict SSH access
2. Specify any combination of shell_scripts and ansible_playbooks and do not specify startup_script or startup_script_file.

If any of the startup script approaches fail by returning a code other than 0, Packer will determine that the build has failed and refuse to save the image.

External access with SSH

The shell scripts and Ansible playbooks customization solutions both require SSH access to the VM from the Packer execution environment. SSH access can be enabled one of 2 ways:

1. The VM is created without a public IP address and SSH tunnels are created using Identity-Aware Proxy (IAP).
   • Allow use_iap to take on its default value of true
2. The VM is created with an IP address on the public internet and firewall rules allow SSH access from the Packer execution environment.
   • Set omit_external_ip = false (or omit_external_ip: false in a blueprint)
   • Add firewall rules that open SSH to the VM

The Packer template defaults to using to the 1st IAP-based solution because it is more secure (no exposure to public internet) and because the vpc module automatically sets up all necessary firewall rules for SSH tunneling and outbound-only access to the internet through Cloud NAT.

In either SSH solution, customization scripts should be supplied as files in the shell_scripts and ansible_playbooks settings.

Environments without SSH access

Many network environments disallow SSH access to VMs. In these environments, the metadata-based startup scripts are appropriate because they execute entirely independently of the Packer execution environment.

In this scenario, a single scripts should be supplied in the form of a string to the startup_script input variable. This solution integrates well with Toolkit runners. Runners operate by using a single startup script whose behavior is extended by downloading and executing a customizable set of runners from Cloud Storage at startup.

NOTE: Packer will attempt to use SSH if either shell_scripts or ansible_playbooks are set to non-empty values. Leave them at their default, empty values to ensure access by SSH is disabled.

Supplying startup script as a string

The startup_script parameter accepts scripts formatted as strings. In Packer and Terraform, multi-line strings can be specified using heredoc syntax in an input Packer variables file (*.pkrvars.hcl) For example, the following snippet defines a multi-line bash script followed by an integer representing the size, in GiB, of the resulting image:

startup_script = <<-EOT
  #!/bin/bash
  yum install -y epel-release
  yum install -y jq
  EOT

disk_size = 100

In a blueprint, the equivalent syntax is:

...
    settings:
      startup_script: |
        #!/bin/bash
        yum install -y epel-release
        yum install -y jq
      disk_size: 100
...

Monitoring startup script execution

When using startup script customization, Packer will print very limited output to the console. For example:

==> example.googlecompute.toolkit_image: Waiting for any running startup script to finish...
==> example.googlecompute.toolkit_image: Startup script not finished yet. Waiting...
==> example.googlecompute.toolkit_image: Startup script not finished yet. Waiting...
==> example.googlecompute.toolkit_image: Startup script, if any, has finished running.

Debugging startup-script failures

If the Packer image build fails, the module will output a gcloud command that can be used directly to review startup-script execution.

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

No requirements.

Providers

No providers.

Modules

No modules.

Resources

No resources.

Inputs

list(object({
    playbook_file   = string
    galaxy_file     = string
    extra_arguments = list(string)
  }))

[
  "projects/click-to-deploy-images/global/licenses/hpc-toolkit-vm-image"
]

[
  "https://www.googleapis.com/auth/cloud-platform"
]

object({
    enable_secure_boot          = bool
    enable_vtpm                 = bool
    enable_integrity_monitoring = bool
  })

{
  "enable_integrity_monitoring": true,
  "enable_secure_boot": true,
  "enable_vtpm": true
}

Outputs

No outputs.