README.md

Description

This module performs the following tasks:

• create an instance template from which execute points will be created
• create a managed instance group (MIG) for execute points
• create a Toolkit runner to configure the autoscaler to scale the MIG

It is expected to be used with the htcondor-install and htcondor-setup modules.

Known limitations

This module may be used multiple times in a blueprint to create sets of execute points in an HTCondor pool. If used more than 1 time, the setting name_prefix must be set to a value that is unique across all uses of the htcondor-execute-point module. If you do not follow this constraint, you will likely receive an error while running terraform apply similar to that shown below.

Error: Invalid value for variable

  on modules/embedded/community/modules/scheduler/htcondor-access-point/main.tf line 136, in module "startup_script":
 136:   runners = local.all_runners
    ├────────────────
    │ var.runners is list of map of string with 5 elements

All startup-script runners must have a unique destination.

How to configure jobs to select execute points

HTCondor access points provisioned by the Toolkit are specially configured to honor an attribute named RequireId in each Job ClassAd. This value must be set to the ID of a MIG created by an instance of this module. The htcondor-access-point module includes a setting var.default_mig_id that will set this value automatically to the MIG ID corresponding to the module's execute points. If this setting is left unset each job must specify +RequireId explicitly. In all cases, the default value can be overridden explicitly as shown below:

universe       = vanilla
executable     = /bin/echo
arguments      = "Hello, World!"
output         = out.$(ClusterId).$(ProcId)
error          = err.$(ClusterId).$(ProcId)
log            = log.$(ClusterId).$(ProcId)
request_cpus   = 1
request_memory = 100MB
+RequireId     = "htcondor-pool-ep-mig"
queue

Example

A full example can be found in the examples README.

The following code snippet creates a pool with 2 sets of HTCondor execute points, one using On-demand pricing and the other using Spot pricing. They use a startup script and network created in previous steps.

- id: htcondor_execute_point
  source: community/modules/compute/htcondor-execute-point
  use:
  - network1
  - htcondor_secrets
  - htcondor_setup
  - htcondor_cm
  settings:
    instance_image:
      project: $(vars.project_id)
      family: $(vars.new_image_family)
    min_idle: 2

- id: htcondor_execute_point_spot
  source: community/modules/compute/htcondor-execute-point
  use:
  - network1
  - htcondor_secrets
  - htcondor_setup
  - htcondor_cm
  settings:
    instance_image:
      project: $(vars.project_id)
      family: $(vars.new_image_family)
    spot: true

- id: htcondor_access
  source: community/modules/scheduler/htcondor-access-point
  use:
  - network1
  - htcondor_secrets
  - htcondor_setup
  - htcondor_cm
  - htcondor_execute_point
  - htcondor_execute_point_spot
  settings:
    default_mig_id: $(htcondor_execute_point.mig_id)
    enable_public_ips: true
    instance_image:
      project: $(vars.project_id)
      family: $(vars.new_image_family)
  outputs:
  - access_point_ips
  - access_point_name

Support

HTCondor is maintained by the Center for High Throughput Computing at the University of Wisconsin-Madison. Support for HTCondor is available via:

• Discussion lists
• HTCondor on GitHub
• HTCondor manual

Behavior of Managed Instance Group (MIG)

Regional MIGs are used to provision Execute Points. By default, VMs will be provisioned in any of the zones available in that region, however, it can be constrained to run in fewer zones (or a single zone) using var.zones.

When the configuration of an Execute Point is changed, the MIG can be configured to replace the VM using a "proactive" or "opportunistic" policy. By default, the policy is set to opportunistic. In practice, this means that Execute Points will NOT be automatically replaced by Terraform when changes to the instance template / HTCondor configuration are made. We recommend leaving this at the default value as it will allow the HTCondor autoscaler to replace VMs when they become idle without disrupting running jobs.

However, if it is desired var.update_policy can be set to "PROACTIVE" to enable automatic replacement. This will disrupt running jobs and send them back to the queue. Alternatively, one can leave the setting at the default value of "OPPORTUNISTIC" and update:

• intentionally by issuing an update via Cloud Console or using gcloud (below)
• VMs becomes unhealthy or are otherwise automatically replaced (e.g. regular Google Cloud maintenance)

For example, to manually update all instances in a MIG:

gcloud compute instance-groups managed update-instances \
   <<NAME-OF-MIG>> --all-instances --region <<REGION>> \
   --project <<PROJECT_ID>> --minimal-action replace

Known Issues

When using OS Login with "external users" (outside of the Google Cloud organization), then Docker universe jobs will fail and cause the Docker daemon to crash. This stems from the use of POSIX user ids (uid) outside the range supported by Docker. Please consider disabling OS Login if this atypical situation applies.

vars:
  # add setting below to existing deployment variables
  enable_oslogin: DISABLE

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Providers

Modules

Resources

Inputs

list(object({
    type  = string,
    count = number
  }))

{
  "family": "hpc-rocky-linux-8",
  "project": "cloud-hpc-image-public"
}

list(object({
    network            = string
    subnetwork         = optional(string)
    subnetwork_project = optional(string)
    nic_type           = optional(string)
    stack_type         = optional(string)
    network_ip         = optional(string, "")
    queue_count        = optional(number)
    access_config = optional(list(object({
      nat_ip       = optional(string)
      network_tier = optional(string)
    })), [])
    ipv6_access_config = optional(list(object({
      network_tier = optional(string)
    })), [])
    alias_ip_range = optional(list(object({
      ip_cidr_range         = string
      subnetwork_range_name = string
    })), [])
  }))

list(object({
    server_ip             = string,
    remote_mount          = string,
    local_mount           = string,
    fs_type               = string,
    mount_options         = string,
    client_install_runner = map(string)
    mount_runner          = map(string)
  }))

[
  "https://www.googleapis.com/auth/cloud-platform"
]

object({
    enable_secure_boot          = bool
    enable_vtpm                 = bool
    enable_integrity_monitoring = bool
  })

{
  "enable_integrity_monitoring": true,
  "enable_secure_boot": true,
  "enable_vtpm": true
}

Outputs