Network Storage in the Cluster Toolkit (formerly HPC Toolkit)

January 27, 2026 · View on GitHub

The Cluster Toolkit provides powerful tools for working with network storage.

The Toolkit contains modules that will provision:

The Toolkit also provides a pre-existing-network-storage module to work with a network storage device that is already set up. The pre-existing-network-storage module supports the following file systems types:

nfs
daos
managed-lustre
gcsfuse

Connecting to Network Storage

In addition to provisioning a network storage device, most file system modules contain scripts that will install any required software needed to utilize the device and mount the file system automatically on supported and tested VM images.

Mounting Via Use

The simplest way to mount a network storage device is by using the use field, as shown below:

...
  - id: network1
    source: modules/network/vpc

  - id: homefs
    source: modules/file-system/filestore
    use: [network1]
    settings: {local_mount: /home}

  - id: workstation
    source: modules/compute/vm-instance
    use: [network1, homefs]  # Note this line

The example above is creating a filestore and automatically mounting it to a VM. Take note of the line with the comment saying # Note this line. By adding the homefs id to the use field of workstation several things automatically happen:

The homefs filestore outputs scripts for nfs client installation and mounting.
The workstation VM reads these scripts and generates a startup script for the VM.
The VM startup script will automatically install the nfs client on its first boot (if not already installed).
The VM will add homefs to fstab and mount the file system.

This same pattern works across most modules in the toolkit. The compatibility matrix below shows modules that can use this method.

Mounting Via Startup

Not all mounting scenarios are supported via the use filed. When use is not supported, automated client installation and mounting can be accomplished by using the startup-script module. Use the compatibility matrix to determine when this method is needed.

The following is an example setting up a filestore using startup script:

...
  - id: network1
    source: modules/network/vpc

  - id: homefs
    source: modules/file-system/filestore
    use: [network1]
    settings: {local_mount: /home}

  - id: filestore-setup
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(homefs.install_nfs_client_runner)
      - $(homefs.mount_runner)

NOTE: The exact names of the runners may be different from module to module.

Compatibility Matrix

The following matrix shows the best method by which each type of network storage device should be mounted to each mount capable module.

	Slurm V6	Batch	vm-instance	Packer (client install)	HTCondor*
filestore	via USE	via USE	via USE	via STARTUP	via USE
nfs-server	via USE	via USE	via USE	via STARTUP	via USE
cloud-storage-bucket (GCS)	via USE	via USE	via USE	via STARTUP	via USE
Managed Lustre	via USE	Needs Testing	via USE	Needs Testing	Needs Testing
netapp-volume	via USE	Needs Testing	via USE	Needs Testing	Needs Testing

filestore (pre-existing)	via USE	via USE	via USE	via STARTUP	via USE
nfs-server (pre-existing)	via USE	via USE	via USE	via STARTUP	via USE
Managed Lustre (pre-existing)	via USE	Needs Testing	via USE	Needs Testing	Needs Testing
GCS FUSE (pre-existing)	via USE	via USE	via USE	via STARTUP	via USE

via USE: Client installation and mounting occur automatically when connected with the use field. See mounting via use section.
via STARTUP: Startup scripts are provided that can be used with the startup-script module to install clients and mount. See mounting via startup section.
Needs Testing: May currently work but has not yet been fully tested.
Not Supported: This feature is not supported right now.

* only supported on CentOS 7