Skip to content

README.md

Latest commit

 

History

History
575 lines (470 loc) · 70.7 KB

README.md

File metadata and controls

575 lines (470 loc) · 70.7 KB
Raw

Description

This module creates a slurm controller node via the internal slurm_instance_template module.

More information about Slurm On GCP can be found at the [project's GitHub page][slurm-gcp] and in the Slurm on Google Cloud User Guide.

The user guide provides detailed instructions on customizing and enhancing the Slurm on GCP cluster as well as recommendations on configuring the controller for optimal performance at different scales.

Example

- id: slurm_controller
  source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
  use:
  - network
  - homefs
  - compute_partition
  settings:
    machine_type: c2-standard-8

This creates a controller node with the following attributes:

  • connected to the primary subnetwork of network
  • the filesystem with the ID homefs (defined elsewhere in the blueprint) mounted
  • One partition with the ID compute_partition (defined elsewhere in the blueprint)
  • machine type upgraded from the default c2-standard-4 to c2-standard-8

Live Cluster Reconfiguration

The schedmd-slurm-gcp-v6-controller module supports the reconfiguration of partitions and slurm configuration in a running, active cluster.

To reconfigure a running cluster:

  1. Edit the blueprint with the desired configuration changes
  2. Call gcluster create <blueprint> -w to overwrite the deployment directory
  3. Follow instructions in terminal to deploy

The following are examples of updates that can be made to a running cluster:

  • Add or remove a partition to the cluster
  • Resize an existing partition
  • Attach new network storage to an existing partition

NOTE: Changing the VM machine_type of a partition may not work. It is better to create a new partition and delete the old one.

High Availability (HA)

The schedmd-slurm-gcp-v6-controller module supports High Availability (HA) for the Slurm controller. This feature deploys a secondary "Backup" controller that allows the cluster to continue operating if the primary controller fails. The module manages these controllers using a Google Compute Managed Instance Group (MIG). By default, the MIG is zonal but can be configured as regional to increase fault tolerance across multiple zones.

Architecture

  • Primary Controller: The active Slurm controller.

  • Backup Controller: A passive standby controller that monitors the primary.

  • Shared State: Both controllers share the Slurm state directory (e.g., /var/spool/slurm) via a network mount (Filestore or NFS) to synchronize keys and runtime state.

Enabling HA

To enable High Availability:

  1. Set enable_backup_controller to true.
  2. Set controller_state_disk to null to disable the default local state disk.
  3. Ensure StateSaveLocation points to a shared network mount (e.g., Filestore, NFS) that is mounted on both controllers at /var/spool/slurm.
  4. Configure the Managed Instance Group distribution type using controller_ha_type (set to either "zonal" or "regional", defaults to "zonal").
  5. Provide at least two static IP addresses in static_ips (one for the primary controller and one for the backup controller).
  6. Configure an external database (cloudsql) to manage Slurm accounting and state across the controllers.

CRITICAL: The StateSaveLocation MUST be a shared network filesystem (like Filestore or NFS) that supports simultaneous read/write access from both controllers. Zonal or Regional Persistent Disks (PD) are NOT supported for this purpose as they do not provide the necessary shared access for HA state management.

Example Configuration

- id: slurm_controller
  source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
  use:
  - network
  - homefs
  settings:
    enable_backup_controller: true
    controller_ha_type: regional
    controller_state_disk: null
    static_ips:
    - 10.80.0.10
    - 10.80.0.11
    cloudsql:
      server_ip: $(slurm_db.cloudsql.server_ip)
      user: $(slurm_db.cloudsql.user)
      password: $(slurm_db.cloudsql.password)
      db_name: $(slurm_db.cloudsql.db_name)
    network_storage:
    - server_ip: $(controller_fs.network_storage.server_ip)
      remote_mount: $(controller_fs.network_storage.remote_mount)
      local_mount: /var/spool/slurm
      fs_type: nfs
      mount_options: _netdev,hard,intr

Health Checks and Autohealing

When enable_backup_controller is true, instances are managed by a Managed Instance Group (MIG) with an autohealing policy using a Google Compute Health Check. By default, a TCP health check probes port 6818 with initial_delay_sec = 900 to allow time for controller startup.

You can customize this configuration using the health_check settings block.

Tuning Guidelines

  • Slow Startup (large state/long startup scripts): Increase initial_delay_sec (e.g. 1200+) to prevent premature restarts.
  • Fast Failover: Lower initial_delay_sec (e.g. 600), check_interval_sec, and timeout_sec. Note: timeout_sec must be $\le$ check_interval_sec.
  • Shared VPC: The health check firewall rule must be created in the host project. If permissions are restricted or rules are pre-configured, set create_firewall_rule to false to disable automatic creation. If configuring firewall rules manually, you must allow ingress traffic from GCP health check prober IP ranges (35.191.0.0/16 and 130.211.0.0/22) to the controllers on the health check port.
  • HTTP Health Checks: Set type to "http" and configure request_path (e.g. "/healthz") to check slurmctld endpoints. See slurmctld HTTP server for more details.

Example configuration overrides:

    health_check:
      type: "tcp"
      initial_delay_sec: 1200
      check_interval_sec: 20
      timeout_sec: 10
      create_firewall_rule: true

Internal Load Balancer (ILB) and Virtual IP (VIP)

To provide stable network failover and avoid hardcoding individual controller IPs, an Internal Load Balancer (ILB) can be provisioned in front of the primary and backup controllers.

To enable the Internal Load Balancer:

  1. Ensure enable_backup_controller is set to true.
  2. Set enable_controller_load_balancer to true.
  3. (Optional) Provide a static IP address using controller_load_balancer_ip to assign to the ILB's forwarding rule (VIP). If omitted, a dynamic IP is allocated from the subnetwork.

Active-Passive HTTP Health Check Agent

An Active-Passive HTTP health check agent automatically runs on port 6821 on each controller instance. The health check agent executes scontrol ping locally and reports status to the ILB:

  • Primary Controller (-0): Returns HTTP 200 OK if the primary controller is up and active, and 503 Service Unavailable if offline.
  • Backup Controller (-1): Returns HTTP 200 OK only when the primary controller is down/offline and the backup controller is up and active. If the primary controller is up, the backup returns 503 Service Unavailable (Standby) to prevent traffic routing to the standby instance.

This setup ensures that all Slurm traffic is directed exclusively to the active controller node, enabling seamless and stable failover.

Health Check and Network Integration Details

  • Bypassing Loopback Routing Loops: When enable_controller_load_balancer is active, the primary and backup controllers are configured with their real internal IPs mapped in /etc/hosts. This ensures that they can communicate directly with each other and bypasses potential loopback routing loops through the load balancer VIP. Non-controller nodes (such as login and compute nodes) map the controller hostname directly to the Load Balancer VIP.
  • Multi-Threaded Probing: The health check service uses a multi-threaded HTTPServer (ThreadingHTTPServer) to support simultaneous, concurrent probes from multiple Google Cloud load-balancer IP ranges without blocking or timing out.
  • Optimized Local Timeout: The health check agent executes scontrol ping with a reduced timeout of 3 seconds (down from 5 seconds). This ensures that if the local Slurm daemon hangs, the health check agent times out before the load balancer's typical 5-second probe timeout, returning a clean 503 Service Unavailable response and preventing abrupt TCP teardowns.
  • Security & Least Privilege: The slurm_health_check.service systemd unit runs under the dedicated slurm user and group rather than root, adhering to security best practices.

Backward Compatibility

The HA feature is opt-in. If enable_backup_controller is set to false (the default), the module deploys a standard single-controller cluster, preserving existing behavior.

Custom Images

For more information on creating valid custom images for the controller VM instance or for custom instance templates, see our vm-images.md documentation page.

GPU Support

More information on GPU support in Slurm on GCP and other Cluster Toolkit modules can be found at docs/gpu-support.md

Reservation for Scheduled Maintenance

A maintenance event is when a compute engine stops a VM to perform a hardware or software update which is determined by the host maintenance policy. This can also affect the running jobs if the maintenance kicks in. Now, Customers can protect jobs from getting terminated due to maintenance using the cluster toolkit. You can enable creation of reservation for scheduled maintenance for your compute nodeset and Slurm will reserve your node for maintenance during the maintenance window. If you try to schedule any jobs which overlap with the maintenance reservation, Slurm would not schedule any job.

You can specify in your blueprint like

  - id: compute_nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      enable_maintenance_reservation: true

To enable creation of reservation for maintenance.

While running job on slurm cluster, you can specify total run time of the job using -t flag.This would only run the job outside of the maintenance window.

srun -n1 -pcompute -t 10:00 <job.sh>

Currently upcoming maintenance notification is supported in ALPHA version of compute API. You can update the API version from your blueprint,

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    settings:
      endpoint_versions:
        compute: "alpha"

Opportunistic GCP maintenance in Slurm

Customers can also enable running GCP maintenance as Slurm job opportunistically to perform early maintenance. If a node is detected for maintenance, Slurm will create a job to perform maintenance and put it in the job queue.

If backfill scheduler is used, Slurm will backfill maintenance job if it can find any empty time window.

Customer can also choose builtin scheduler type. In this case, Slurm would run maintenance job in strictly priority order. If the maintenance job doesn't kick in, then forced maintenance will take place at scheduled window.

Customer can enable this feature at nodeset level by,

  - id: debug_nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      enable_opportunistic_maintenance: true

Placement Max Distance

When using enable_placement with Slurm, Google Compute Engine will attempt to place VMs as physically close together as possible. Capacity constraints at the time of VM creation may still force VMs to be spread across multiple racks. Google provides the max-distance flag which can used to control the maximum spreading allowed. Read more about max-distance in the official docs.

You can use the placement_max_distance setting on the nodeset module to control the max-distance behavior. See the following example:

  - id: nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [ network ]
    settings:
      machine_type: c2-standard-4
      node_count_dynamic_max: 30
      enable_placement: true
      placement_max_distance: 1

> [!NOTE]
> `schedmd-slurm-gcp-v6-nodeset.settings.enable_placement: true` must also be
> set for placement_max_distance to take effect.

In the above case using a value of 1 will restrict VM to be placed on the same
rack. You can confirm that the `max-distance` was applied by calling the
following command while jobs are running:

```shell
gcloud beta compute resource-policies list \
  --format='yaml(name,groupPlacementPolicy.maxDistance)'

[!WARNING] If a zone lacks capacity, using a lower max-distance value (such as 1) is more likely to cause VMs creation to fail.

TreeWidth and Node Communication

Slurm uses a fan out mechanism to communicate large groups of nodes. The shape of this fan out tree is determined by the TreeWidth configuration variable.

In the cloud, this fan out mechanism can become unstable when nodes restart with new IP addresses. You can enforce that all nodes communicate directly with the controller by setting TreeWidth to a value >= largest partition.

If the largest partition was 200 nodes, configure the blueprint as follows:

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    ...
    settings:
      cloud_parameters:
        tree_width: 200

The default has been set to 128. Values above this have not been fully tested and may cause congestion on the controller. A more scalable solution is under way.

ResumeRate and Node Resumption

The ResumeRate parameter in slurm.conf controls the maximum number of nodes that Slurm attempts to resume (power up) per minute. This is particularly important in cloud environments where auto-scaling can lead to a large number of nodes starting concurrently.

When many nodes start simultaneously, they can place a heavy load on shared resources, especially shared filesystems, as they all try to mount filesystems and access configuration files at the same time. By limiting the ResumeRate, you can stagger the node startup process, reducing the peak load on these shared resources and improving overall cluster stability during scaling events.

For example, to limit the node resumption rate to 100 nodes per minute, configure the blueprint as follows:

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    ...
    settings:
      cloud_parameters:
        resume_rate: 100

Adjust this value based on the capabilities of your shared filesystem and the expected scaling behavior of your cluster.

Support

The Cluster Toolkit team maintains the wrapper around the slurm-on-gcp terraform modules. For support with the underlying modules, see the instructions in the slurm-gcp README.

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name Version
terraform >= 1.12.2
google >= 6.41
google-beta >= 6.0.0
local ~> 2.0

Providers

Name Version
google >= 6.41
google-beta >= 6.0.0
local ~> 2.0

Modules

Name Source Version
bucket terraform-google-modules/cloud-storage/google >= 6.1
daos_network_storage_scripts ../../../../modules/scripts/startup-script n/a
gpu ../../../../modules/internal/gpu-definition n/a
login ../../internal/slurm-gcp/login n/a
nodeset_cleanup ./modules/cleanup_compute n/a
nodeset_cleanup_tpu ./modules/cleanup_tpu n/a
slurm_controller_template ../../internal/slurm-gcp/instance_template n/a
slurm_files ./modules/slurm_files n/a
slurm_nodeset_template ../../internal/slurm-gcp/instance_template n/a
slurm_nodeset_tpu ../../internal/slurm-gcp/nodeset_tpu n/a

Resources

Name Type
google-beta_google_compute_instance_from_template.controller resource
google_compute_address.controller_ips resource
google_compute_disk.controller_disk resource
google_compute_firewall.health_check_firewall_rule resource
google_compute_forwarding_rule.slurm_controller_vip resource
google_compute_health_check.controller_health_check resource
google_compute_instance_group_manager.controller_zonal_mig resource
google_compute_per_instance_config.controller_zonal_stateful_ips resource
google_compute_region_backend_service.slurm_controller_backend resource
google_compute_region_health_check.slurm_health_check resource
google_compute_region_instance_group_manager.controller_regional_mig resource
google_compute_region_instance_group_manager.nodeset_mig resource
google_compute_region_per_instance_config.controller_regional_stateful_ips resource
google_compute_resource_policy.nodeset_workload_policy resource
google_secret_manager_secret.cloudsql resource
google_secret_manager_secret_iam_member.cloudsql_secret_accessor resource
google_secret_manager_secret_version.cloudsql_version resource
google_storage_bucket_iam_member.legacy_readers resource
google_storage_bucket_iam_member.viewers resource
google_storage_bucket_object.partition_config resource
local_file.hybrid_install resource
google_compute_subnetwork.controller_subnetwork data source
google_compute_zones.available data source
google_project.controller_project data source

Inputs

Name Description Type Default Required
additional_disks List of maps of disks. 
list(object({
    disk_name                           = optional(string)
    device_name                         = optional(string)
    disk_type                           = optional(string)
    disk_storage_pool                   = optional(string)
    disk_size_gb                        = optional(number)
    disk_labels                         = optional(map(string), {})
    auto_delete                         = optional(bool, true)
    boot                                = optional(bool, false)
    disk_resource_manager_tags          = optional(map(string), {})
    disk_encryption_key                 = optional(string)
    disk_encryption_key_service_account = optional(string)
  }))
 [] no
additional_networks Additional network interface details for the controller, if any. For Private Service Connect interfaces, 'network_attachment' must be the full resource URI: projects/{project}/regions/{region}/networkAttachments/{name}. 
list(object({
    access_config = optional(list(object({
      nat_ip       = string
      network_tier = string
    })), [])
    alias_ip_range = optional(list(object({
      ip_cidr_range         = string
      subnetwork_range_name = string
    })), [])
    ipv6_access_config = optional(list(object({
      network_tier = string
    })), [])
    network            = optional(string)
    network_attachment = optional(string)
    network_ip         = optional(string, "")
    nic_type           = optional(string)
    queue_count        = optional(number)
    stack_type         = optional(string)
    subnetwork         = optional(string)
    subnetwork_project = optional(string)
  }))
 [] no
advanced_machine_features See https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance_template#nested_advanced_machine_features 
object({
    enable_nested_virtualization = optional(bool)
    threads_per_core             = optional(number)
    turbo_mode                   = optional(string)
    visible_core_count           = optional(number)
    performance_monitoring_unit  = optional(string)
    enable_uefi_networking       = optional(bool)
  })
 
{
  "threads_per_core": 1
}
 no
allow_automatic_updates If false, disables automatic system package updates on the created instances. This feature is
only available on supported images (or images derived from them). For more details, see
https://cloud.google.com/compute/docs/instances/create-hpc-vm#disable_automatic_updates		 bool true no
backup_zone Zone for the backup controller. If null, it will be placed in the same region, potentially different zone. string null no
bandwidth_tier Configures the network interface card and the maximum egress bandwidth for VMs.
- Setting platform_default respects the Google Cloud Platform API default values for networking.
- Setting virtio_enabled explicitly selects the VirtioNet network adapter.
- Setting gvnic_enabled selects the gVNIC network adapter (without Tier 1 high bandwidth).
- Setting tier_1_enabled selects both the gVNIC adapter and Tier 1 high bandwidth networking.
- Note: both gVNIC and Tier 1 networking require a VM image with gVNIC support as well as specific VM families and shapes.
- See official docs for more details.		 string "platform_default" no
bucket_dir Bucket directory for cluster files to be put into. If not specified, then one will be chosen based on slurm_cluster_name. string null no
bucket_name Name of GCS bucket.
Ignored when 'create_bucket' is true.		 string null no
can_ip_forward Enable IP forwarding, for NAT instances for example. bool false no
cgroup_conf_tpl Slurm cgroup.conf template file path. string null no
cloud_parameters cloud.conf options. Defaults inherited from Slurm GCP repo 
object({
    no_comma_params         = optional(bool, false)
    private_data            = optional(list(string))
    scheduler_parameters    = optional(list(string))
    resume_rate             = optional(number)
    resume_timeout          = optional(number)
    suspend_rate            = optional(number)
    suspend_timeout         = optional(number)
    slurmd_timeout          = optional(number)
    unkillable_step_timeout = optional(number)
    topology_plugin         = optional(string)
    topology_param          = optional(string)
    tree_width              = optional(number)
    prolog_flags            = optional(string)
    switch_type             = optional(string)
  })
 {} no
cloudsql Use this database instead of the one on the controller.
server_ip : Address of the database server.
user : The user to access the database as.
password : The password, given the user, to access the given database. (sensitive)
db_name : The database to access.
user_managed_replication : The list of location and (optional) kms_key_name for secret		 
object({
    server_ip = string
    user      = string
    password  = string # sensitive
    db_name   = string
    user_managed_replication = optional(list(object({
      location     = string
      kms_key_name = optional(string)
    })), [])
  })
 null no
compute_startup_script DEPRECATED: compute_startup_script has been deprecated.
Use startup_script of nodeset module instead.		 any null no
compute_startup_scripts_timeout The timeout (seconds) applied to each startup script in compute nodes. If
any script exceeds this timeout, then the instance setup process is considered
failed and handled accordingly.

NOTE: When set to 0, the timeout is considered infinite and thus disabled.		 number 300 no
controller_ha_type Type of Managed Instance Group for controllers: 'zonal' or 'regional'. string "zonal" no
controller_load_balancer_ip Optional static IP address to assign to the controller Internal Load Balancer (VIP). If null, one will be dynamically assigned from the subnetwork. string null no
controller_network_attachment SelfLink for NetworkAttachment to be attached to the controller, if any. string null no
controller_project_id Optionally. Provision controller and config bucket in the different project string null no
controller_startup_script Startup script used by the controller VM. string "# no-op" no
controller_startup_scripts_timeout The timeout (seconds) applied to each script in controller_startup_scripts. If
any script exceeds this timeout, then the instance setup process is considered
failed and handled accordingly.

NOTE: When set to 0, the timeout is considered infinite and thus disabled.		 number 300 no
controller_state_disk A disk that will be attached to the controller instance template to save state of slurm. The disk is created and used by default.
To disable this feature, set this variable to null.

NOTE: This will not save the contents at /opt/apps and /home. To preserve those, they must be saved externally.		 
object({
    type = string
    size = number
  })
 
{
  "size": 50,
  "type": "pd-ssd"
}
 no
create_bucket Create GCS bucket instead of using an existing one. bool true no
deployment_name Name of the deployment. string n/a yes
disable_controller_public_ips DEPRECATED: Use enable_controller_public_ips instead. bool null no
disable_default_mounts DEPRECATED: Use enable_default_mounts instead. bool null no
disable_smt DEPRECATED: Use advanced_machine_features.threads_per_core instead. bool null no
disk_auto_delete Whether or not the boot disk should be auto-deleted. bool true no
disk_encryption_key The id of the encryption key that is stored in Google Cloud KMS to use to encrypt all the disks on this instance string null no
disk_encryption_key_service_account The service account being used for the encryption request for the given KMS key. If absent, the Compute Engine default service account is used. string null no
disk_labels Labels specific to the boot disk. These will be merged with var.labels. map(string) {} no
disk_resource_manager_tags (Optional) A set of key/value resource manager tag pairs to bind to the instance disks. Keys must be in the format tagKeys/{tag_key_id}, and values are in the format tagValues/456. map(string) {} no
disk_size_gb Boot disk size in GB. number 50 no
disk_storage_pool Storage pool to use for the boot disk. Note that storage pools are only supported with Hyperdisk types. For boot disks, only hyperdisk-balanced is supported. You must provide an existing storage pool, as this module does not create new ones. string null no
disk_type Boot disk type, can be either hyperdisk-balanced, pd-ssd, pd-standard, pd-balanced, or pd-extreme. string "pd-ssd" no
enable_backup_controller Enables a secondary backup controller for High Availability. bool false no
enable_bigquery_load Enables loading of cluster job usage into big query.

NOTE: Requires Google Bigquery API.		 bool false no
enable_chs_gpu_health_check_epilog Enable a Cluster Health Sacnner(CHS) GPU health check that slurmd executes as an epilog script after completing a job step from a new job allocation.
Compute nodes that fail GPU health check during epilog will be marked as drained. Find more details at:
https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/main/docs/CHS-Slurm.md		 bool false no
enable_chs_gpu_health_check_prolog Enable a Cluster Health Sacnner(CHS) GPU health check that slurmd executes as a prolog script whenever it is asked to run a job step from a new job allocation. Compute nodes that fail GPU health check during prolog will be marked as drained. Find more details at:
https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/main/docs/CHS-Slurm.md		 bool false no
enable_cleanup_compute Enables automatic cleanup of compute nodes and resource policies (e.g.
placement groups) managed by this module, when cluster is destroyed.

WARNING: Toggling this off will impact the running workload.
Deployed compute nodes will be destroyed.		 bool true no
enable_confidential_vm Enable the Confidential VM configuration. Note: the instance image must support option. bool false no
enable_controller_load_balancer Enables an Internal Load Balancer (ILB) in front of the controllers for stable Virtual IP and network-level failover. bool false no
enable_controller_public_ips If set to true. The controller will have a random public IP assigned to it. Ignored if access_config is set. bool false no
enable_debug_logging Enables debug logging mode. bool false no
enable_default_mounts Enable default global network storage from the controller
- /home
- /opt/apps		 bool null no
enable_devel DEPRECATED: enable_devel is always on. bool null no
enable_expedited_requeue Enables Expedited Requeue, which automatically requeues eligible jobs and grants them the highest priority upon node failure. (Usage: sbatch --requeue=expedite) bool true no
enable_external_prolog_epilog Automatically enable a script that will execute prolog and epilog scripts
shared by NFS from the controller to compute nodes. Find more details at:
https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/tools/prologs-epilogs/README.md		 bool null no
enable_health_check_start_only Adjusts the Slurm HealthCheckNodeState behavior to run health checks solely upon node initialization. This prevents continuous health check polling. bool false no
enable_hybrid Enables use of hybrid controller mode. When true, controller_hybrid_config will
be used instead of controller_instance_config and will disable login instances.		 bool false no
enable_openmetrics Enable native Prometheus OpenMetrics telemetry via Slurm and Google Cloud Ops Agent bool false no
enable_oslogin Enables Google Cloud os-login for user login and authentication for VMs.
See https://cloud.google.com/compute/docs/oslogin		 bool true no
enable_shielded_vm Enable the Shielded VM configuration. Note: the instance image must support option. bool false no
enable_slurm_auth Enables slurm authentication instead of munge. bool true no
enable_slurm_gcp_plugins DEPRECATED: Slurm GCP plugins have been deprecated.
Instead of 'max_hops' plugin please use the 'placement_max_distance' nodeset property.
Instead of 'enable_vpmu' plugin please use 'advanced_machine_features.performance_monitoring_unit' nodeset property.		 any null no
enable_smt DEPRECATED: Use advanced_machine_features.threads_per_core instead. bool null no
endpoint_versions Version of the API to use (The compute service is the only API currently supported) 
object({
    compute = string
  })
 
{
  "compute": "beta"
}
 no
epilog_scripts List of scripts to be used for Epilog. Programs for the slurmd to execute
on every node when a user's job completes.
See https://slurm.schedmd.com/slurm.conf.html#OPT_Epilog.		 
list(object({
    filename = string
    content  = optional(string)
    source   = optional(string)
  }))
 [] no
experimental Experimental Slurm settings. These features are subject to change and may be modified in future releases. 
object({
    enable_async_reply = optional(bool, false)
  })
 {} no
extra_logging_flags The only available flag is trace_api map(bool) {} no
gcloud_path_override Directory of the gcloud executable to be used during cleanup string "" no
guest_accelerator List of the type and count of accelerator cards attached to the instance. 
list(object({
    type  = string,
    count = number
  }))
 [] no
health_check Health check and autohealing configuration for controller HA instance groups. 
object({
    type                 = optional(string, "tcp")
    port                 = optional(number, 6818)
    initial_delay_sec    = optional(number, 900)
    check_interval_sec   = optional(number, 60)
    timeout_sec          = optional(number, 10)
    healthy_threshold    = optional(number, 2)
    unhealthy_threshold  = optional(number, 5)
    enable_logging       = optional(bool, true)
    request_path         = optional(string, "/healthz")
    create_firewall_rule = optional(bool, true)
    port_name            = optional(string, "slurmctld")
  })
 {} no
hybrid_conf The hybrid configuration 
object({
    slurm_bin_dir           = optional(string)
    slurm_log_dir           = optional(string)
    slurm_control_host      = string
    slurm_control_host_port = optional(string)
    slurm_control_addr      = optional(string)
    slurm_uid               = optional(number)
    slurm_gid               = optional(number)
    service_account_email   = optional(string)
    google_app_cred_path    = optional(string)
    slurm_version           = optional(string)
  })
 null no
install_dir Directory where the hybrid configuration directory will be installed on the
on-premise controller (e.g. /etc/slurm/hybrid). This updates the prefix path
for the resume and suspend scripts in the generated cloud.conf file.

This variable should be used when the TerraformHost and the SlurmctldHost
are different.

This will default to var.output_dir if null.		 string null no
instance_image Defines the image that will be used in the Slurm controller VM instance.

Expected Fields:
name: The name of the image. Mutually exclusive with family.
family: The image family to use. Mutually exclusive with name.
project: The project where the image is hosted.

For more information on creating custom images that comply with Slurm on GCP
see the "Slurm on GCP Custom Images" section in docs/vm-images.md.		 map(string) 
{
  "family": "slurm-gcp-6-12-hpc-rocky-linux-9",
  "project": "schedmd-slurm-public"
}
 no
instance_image_custom A flag that designates that the user is aware that they are requesting
to use a custom and potentially incompatible image for this Slurm on
GCP module.

If the field is set to false, only the compatible families and project
names will be accepted. The deployment will fail with any other image
family or name. If set to true, no checks will be done.

See: https://goo.gle/hpc-slurm-images		 bool false no
instance_template DEPRECATED: Instance template can not be specified for controller. string null no
labels Labels, provided as a map. map(string) {} no
login_network_storage An array of network attached storage mounts to be configured on all login nodes. 
list(object({
    server_ip               = string,
    remote_mount            = string,
    local_mount             = string,
    local_mount_owner       = optional(string)
    local_mount_permissions = optional(string)
    fs_type                 = string,
    mount_options           = string,
  }))
 [] no
login_nodes List of slurm login instance definitions. 
list(object({
    group_name = string
    access_config = optional(list(object({
      nat_ip       = string
      network_tier = string
    })))
    additional_disks = optional(list(object({
      disk_name                           = optional(string)
      device_name                         = optional(string)
      disk_size_gb                        = optional(number)
      disk_type                           = optional(string)
      disk_storage_pool                   = optional(string)
      disk_labels                         = optional(map(string), {})
      auto_delete                         = optional(bool, true)
      boot                                = optional(bool, false)
      disk_resource_manager_tags          = optional(map(string), {})
      disk_encryption_key                 = optional(string)
      disk_encryption_key_service_account = optional(string)
    })), [])
    additional_networks = optional(list(object({
      access_config = optional(list(object({
        nat_ip       = string
        network_tier = string
      })), [])
      alias_ip_range = optional(list(object({
        ip_cidr_range         = string
        subnetwork_range_name = string
      })), [])
      ipv6_access_config = optional(list(object({
        network_tier = string
      })), [])
      network            = optional(string)
      network_attachment = optional(string)
      network_ip         = optional(string, "")
      nic_type           = optional(string)
      queue_count        = optional(number)
      stack_type         = optional(string)
      subnetwork         = optional(string)
      subnetwork_project = optional(string)
    })), [])
    bandwidth_tier             = optional(string, "platform_default")
    can_ip_forward             = optional(bool, false)
    disk_auto_delete           = optional(bool, true)
    disk_labels                = optional(map(string), {})
    disk_resource_manager_tags = optional(map(string), {})
    disk_size_gb               = optional(number)
    disk_type                  = optional(string, "n1-standard-1")
    disk_storage_pool          = optional(string)
    enable_confidential_vm     = optional(bool, false)
    enable_oslogin             = optional(bool, true)
    enable_shielded_vm         = optional(bool, false)
    gpu = optional(object({
      count = number
      type  = string
    }))
    labels       = optional(map(string), {})
    machine_type = optional(string)
    advanced_machine_features = object({
      enable_nested_virtualization = optional(bool)
      threads_per_core             = optional(number)
      turbo_mode                   = optional(string)
      visible_core_count           = optional(number)
      performance_monitoring_unit  = optional(string)
      enable_uefi_networking       = optional(bool)
    })
    metadata              = optional(map(string), {})
    min_cpu_platform      = optional(string)
    num_instances         = optional(number, 1)
    on_host_maintenance   = optional(string)
    preemptible           = optional(bool, false)
    region                = optional(string)
    resource_manager_tags = optional(map(string), {})
    service_account = optional(object({
      email  = optional(string)
      scopes = optional(list(string), ["https://www.googleapis.com/auth/cloud-platform"])
    }))
    shielded_instance_config = optional(object({
      enable_integrity_monitoring = optional(bool, true)
      enable_secure_boot          = optional(bool, true)
      enable_vtpm                 = optional(bool, true)
    }))
    source_image_family                 = optional(string)
    source_image_project                = optional(string)
    source_image                        = optional(string)
    static_ips                          = optional(list(string), [])
    subnetwork                          = string
    spot                                = optional(bool, false)
    tags                                = optional(list(string), [])
    zone                                = optional(string)
    termination_action                  = optional(string)
    disk_encryption_key                 = optional(string)
    disk_encryption_key_service_account = optional(string)
    startup_script = optional(list(object({
      filename = string
      content  = string
    })), [])
  }))
 [] no
login_startup_script Startup script used by the login VMs. string "# no-op" no
login_startup_scripts_timeout The timeout (seconds) applied to each script in login_startup_scripts. If
any script exceeds this timeout, then the instance setup process is considered
failed and handled accordingly.

NOTE: When set to 0, the timeout is considered infinite and thus disabled.		 number 300 no
machine_type Machine type to create. string "c2-standard-4" no
metadata Metadata, provided as a map. map(string) {} no
min_cpu_platform Specifies a minimum CPU platform. Applicable values are the friendly names of
CPU platforms, such as Intel Haswell or Intel Skylake. See the complete list:
https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform		 string null no
munge_mount Remote munge mount for compute and login nodes to acquire the munge.key.
By default, the munge mount server will be assumed to be the
var.slurm_control_host (or var.slurm_control_addr if non-null) when
server_ip=null.		 
object({
    server_ip     = string
    remote_mount  = string
    fs_type       = string
    mount_options = string
  })
 
{
  "fs_type": "nfs",
  "mount_options": "",
  "remote_mount": "/etc/munge/",
  "server_ip": null
}
 no
named_ports Named ports for the controller instance group. 
list(object({
    name = string
    port = number
  }))
 [] no
network_storage An array of network attached storage mounts to be configured on all instances. 
list(object({
    server_ip               = string,
    remote_mount            = string,
    local_mount             = string,
    local_mount_owner       = optional(string)
    local_mount_permissions = optional(string)
    fs_type                 = string,
    mount_options           = string,
    client_install_runner   = optional(map(string))
    mount_runner            = optional(map(string))
  }))
 [] no
nodeset Define nodesets, as a list. 
list(object({
    node_count_static      = optional(number, 0)
    node_count_dynamic_max = optional(number, 1)
    node_conf              = optional(map(string), {})
    nodeset_name           = string
    additional_disks = optional(list(object({
      disk_name                  = optional(string)
      device_name                = optional(string)
      disk_size_gb               = optional(number)
      disk_type                  = optional(string)
      disk_storage_pool          = optional(string)
      disk_labels                = optional(map(string), {})
      auto_delete                = optional(bool, true)
      boot                       = optional(bool, false)
      disk_resource_manager_tags = optional(map(string), {})
    })), [])
    bandwidth_tier                      = optional(string, "platform_default")
    can_ip_forward                      = optional(bool, false)
    disk_auto_delete                    = optional(bool, true)
    disk_labels                         = optional(map(string), {})
    disk_resource_manager_tags          = optional(map(string), {})
    disk_size_gb                        = optional(number)
    disk_type                           = optional(string)
    disk_storage_pool                   = optional(string)
    disk_encryption_key                 = optional(string)
    disk_encryption_key_service_account = optional(string)
    enable_confidential_vm              = optional(bool, false)
    confidential_instance_type          = optional(string)
    enable_placement                    = optional(bool, false)
    placement_max_distance              = optional(number, null)
    enable_oslogin                      = optional(bool, true)
    enable_shielded_vm                  = optional(bool, false)
    enable_maintenance_reservation      = optional(bool, false)
    enable_opportunistic_maintenance    = optional(bool, false)
    gpu = optional(object({
      count = number
      type  = string
    }))
    # Resolved GPU count per VM, used to derive accelerator topology slice sizes.
    # Declared explicitly so it is not dropped during object type conversion.
    gpu_count            = optional(number)
    accelerator_topology = optional(string, null)
    dws_flex = object({
      enabled          = bool
      max_run_duration = number
      use_job_duration = bool
      use_bulk_insert  = bool
    })
    provisioning_engine = optional(string, "AUTO")
    labels              = optional(map(string), {})
    machine_type        = optional(string)
    advanced_machine_features = object({
      enable_nested_virtualization = optional(bool)
      threads_per_core             = optional(number)
      turbo_mode                   = optional(string)
      visible_core_count           = optional(number)
      performance_monitoring_unit  = optional(string)
      enable_uefi_networking       = optional(bool)
    })
    maintenance_interval     = optional(string)
    instance_properties_json = string
    metadata                 = optional(map(string), {})
    min_cpu_platform         = optional(string)
    network_tier             = optional(string, "STANDARD")
    network_storage = optional(list(object({
      server_ip               = string
      remote_mount            = string
      local_mount             = string
      local_mount_owner       = optional(string)
      local_mount_permissions = optional(string)
      fs_type                 = string
      mount_options           = string
      client_install_runner   = optional(map(string))
      mount_runner            = optional(map(string))
    })), [])
    on_host_maintenance   = optional(string)
    preemptible           = optional(bool, false)
    region                = optional(string)
    resource_manager_tags = optional(map(string), {})
    service_account = optional(object({
      email  = optional(string)
      scopes = optional(list(string), ["https://www.googleapis.com/auth/cloud-platform"])
    }))
    shielded_instance_config = optional(object({
      enable_integrity_monitoring = optional(bool, true)
      enable_secure_boot          = optional(bool, true)
      enable_vtpm                 = optional(bool, true)
    }))
    source_image_family  = optional(string)
    source_image_project = optional(string)
    source_image         = optional(string)
    subnetwork_self_link = string
    additional_networks = optional(list(object({
      network            = optional(string)
      subnetwork         = optional(string)
      subnetwork_project = optional(string)
      network_attachment = optional(string)
      network_ip         = optional(string, "")
      nic_type           = optional(string)
      stack_type         = optional(string)
      queue_count        = optional(number)
      access_config = optional(list(object({
        nat_ip       = string
        network_tier = string
      })), [])
      ipv6_access_config = optional(list(object({
        network_tier = string
      })), [])
      alias_ip_range = optional(list(object({
        ip_cidr_range         = string
        subnetwork_range_name = string
      })), [])
    })), [])
    access_config = optional(list(object({
      nat_ip       = string
      network_tier = string
    })))
    spot               = optional(bool, false)
    tags               = optional(list(string), [])
    termination_action = optional(string)
    reservation_name   = optional(string)
    future_reservation = string
    startup_script = optional(list(object({
      filename = string
    content = string })), [])

    zone_target_shape = string
    zone_policy_allow = set(string)
    zone_policy_deny  = set(string)

    instance_flexibility_policy = optional(object({
      instance_selections = list(object({
        name          = optional(string)
        rank          = optional(number, 1)
        machine_types = set(string)
      }))
    }))
  }))
 [] no
nodeset_dyn Defines dynamic nodesets, as a list. 
list(object({
    nodeset_name    = string
    nodeset_feature = string
  }))
 [] no
nodeset_tpu Define TPU nodesets, as a list. 
list(object({
    node_count_static      = optional(number, 0)
    node_count_dynamic_max = optional(number, 5)
    nodeset_name           = string
    enable_public_ip       = optional(bool, false)
    node_type              = string
    accelerator_config = optional(object({
      topology = string
      version  = string
      }), {
      topology = ""
      version  = ""
    })
    tf_version   = string
    preemptible  = optional(bool, false)
    preserve_tpu = optional(bool, false)
    zone         = string
    data_disks   = optional(list(string), [])
    docker_image = optional(string, "")
    network_storage = optional(list(object({
      server_ip               = string
      remote_mount            = string
      local_mount             = string
      local_mount_owner       = optional(string)
      local_mount_permissions = optional(string)
      fs_type                 = string
      mount_options           = string
      client_install_runner   = optional(map(string))
      mount_runner            = optional(map(string))
    })), [])
    subnetwork = string
    service_account = optional(object({
      email  = optional(string)
      scopes = optional(list(string), ["https://www.googleapis.com/auth/cloud-platform"])
    }))
    project_id = string
    reserved   = optional(string, false)
  }))
 [] no
on_host_maintenance Instance availability Policy. string "MIGRATE" no
output_dir Directory where this module will write its files to. These files include:
cloud.conf; cloud_gres.conf; config.yaml; resume.py; suspend.py; and util.py.		 string null no
partitions Cluster partitions as a list. See module slurm_partition. 
list(object({
    partition_name        = string
    partition_conf        = optional(map(string), {})
    partition_nodeset     = optional(list(string), [])
    partition_nodeset_dyn = optional(list(string), [])
    partition_nodeset_tpu = optional(list(string), [])
    enable_job_exclusive  = optional(bool, false)
  }))
 [] no
preemptible Allow the instance to be preempted. bool false no
project_id Project ID to create resources in. string n/a yes
prolog_scripts List of scripts to be used for Prolog. Programs for the slurmd to execute
whenever it is asked to run a job step from a new job allocation.
See https://slurm.schedmd.com/slurm.conf.html#OPT_Prolog.		 
list(object({
    filename = string
    content  = optional(string)
    source   = optional(string)
  }))
 [] no
region The default region to place resources in. string n/a yes
resource_manager_tags (Optional) A set of key/value resource manager tag pairs to bind to the instances. Keys must be in the format tagKeys/{tag_key_id}, and values are in the format tagValues/456. map(string) {} no
service_account DEPRECATED: Use service_account_email and service_account_scopes instead. 
object({
    email  = string
    scopes = set(string)
  })
 null no
service_account_email Service account e-mail address to attach to the controller instance. string null no
service_account_scopes Scopes to attach to the controller instance. set(string) 
[
  "https://www.googleapis.com/auth/cloud-platform"
]
 no
shielded_instance_config Shielded VM configuration for the instance. Note: not used unless
enable_shielded_vm is 'true'.
enable_integrity_monitoring : Compare the most recent boot measurements to the
integrity policy baseline and return a pair of pass/fail results depending on
whether they match or not.
enable_secure_boot : Verify the digital signature of all boot components, and
halt the boot process if signature verification fails.
enable_vtpm : Use a virtualized trusted platform module, which is a
specialized computer chip you can use to encrypt objects like keys and
certificates.		 
object({
    enable_integrity_monitoring = bool
    enable_secure_boot          = bool
    enable_vtpm                 = bool
  })
 
{
  "enable_integrity_monitoring": true,
  "enable_secure_boot": true,
  "enable_vtpm": true
}
 no
slurm_bucket_kms_key Customer-managed encryption key self-link to use for the Slurm configuration bucket. string null no
slurm_cluster_name Cluster name, used for resource naming and slurm accounting.
If not provided it will default to the first 8 characters of the deployment name (removing any invalid characters).		 string null no
slurm_conf_template Slurm slurm.conf template. Content of the file in 'slurm_conf_tpl' is used if this is not set. string null no
slurm_conf_tpl Slurm slurm.conf template file path. This path is used only if raw content is not provided in 'slurm_conf_template'. string null no
slurm_control_host_port The port number that the Slurm controller, slurmctld, listens to for work. string "6818" no
slurm_key_mount Remote mount for compute and login nodes to acquire the slurm.key. 
object({
    server_ip     = string
    remote_mount  = string
    fs_type       = string
    mount_options = string
  })
 null no
slurmdbd_conf_tpl Slurm slurmdbd.conf template file path. string null no
static_ips List of static IPs for VM instances. list(string) [] no
subnetwork_self_link Subnet to deploy to. string n/a yes
subnetwork_stack_type Stack type of the subnet to deploy to. string "IPV4_ONLY" no
tags Network tag list. list(string) [] no
task_epilog_scripts List of scripts to be used for TaskEpilog. Programs for the slurmd to execute
as the slurm job's owner after termination of each task.
See https://slurm.schedmd.com/slurm.conf.html#OPT_TaskEpilog.		 
list(object({
    filename = string
    content  = optional(string)
    source   = optional(string)
  }))
 [] no
task_prolog_scripts List of scripts to be used for TaskProlog. Programs for the slurmd to execute
as the slurm job's owner prior to initiation of each task.
See https://slurm.schedmd.com/slurm.conf.html#OPT_TaskProlog.		 
list(object({
    filename = string
    content  = optional(string)
    source   = optional(string)
  }))
 [] no
universe_domain Domain address for alternate API universe string "googleapis.com" no
zone Zone where the instances should be created. If not specified, instances will be
spread across available zones in the region.		 string null no

Outputs

Name Description
controller_instance_group Self-link of the controller instance group (zonal or regional) if HA is enabled.
controller_instance_names Names of the controller instances when HA is enabled.
controller_mig_id Fully qualified group manager id (zonal or regional).
controller_mig_name Name of the controller Managed Instance Group.
instructions Post deployment instructions.
munge_deprecation_warning Deprecation warning for legacy MUNGE authentication.
slurm_bucket GCS Bucket of Slurm cluster file storage.
slurm_bucket_dir Path directory within bucket_name for Slurm cluster file storage.
slurm_bucket_name GCS Bucket name of Slurm cluster file storage.
slurm_bucket_path Bucket path used by cluster.
slurm_cluster_name Slurm cluster name.
slurm_control_host_port The port number that the Slurm controller, slurmctld, listens to for work.
slurm_controller_instance Compute instance of controller node
slurm_login_instances Compute instances of login nodes