Description

This module creates a Google Kubernetes Engine (GKE) node pool.

NOTE: This is an experimental module and the functionality and documentation will likely be updated in the near future. This module has only been tested in limited capacity.

Example

The following example creates a GKE node group.

  - id: compute_pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]

Also see a full GKE example blueprint.

Taints and Tolerations

By default node pools created with this module will be tainted with user-workload=true:NoSchedule to prevent system pods from being scheduled. User jobs targeting the node pool should include this toleration. This behavior can be overridden using the taints setting. See docs for more info.

Local SSD Storage

GKE offers two options for managing locally attached SSDs.

The first, and recommended, option is for GKE to manage the ephemeral storage space on the node, which will then be automatically attached to pods which request an emptyDir volume. This can be accomplished using the local_ssd_count_ephemeral_storage variable.

The second, more complex, option is for GCP to attach these nodes as raw block storage. In this case, the cluster administrator is responsible for software RAID settings, partitioning, formatting and mounting these disks on the host OS. Still, this may be desired behavior in use cases which aren't supported by an emptyDir volume (for example, a ReadOnlyMany or ReadWriteMany PV). This can be accomplished using the local_ssd_count_nvme_block variable.

The local_ssd_count_ephemeral_storage and local_ssd_count_nvme_block variables are mutually exclusive and cannot be mixed together.

Also, the number of SSDs which can be attached to a node depends on the machine type.

See docs for more info.

Considerations with GPUs

When a GPU is attached to a node an additional taint is automatically added: nvidia.com/gpu=present:NoSchedule. For jobs to get placed on these nodes, the equivalent toleration is required. The gke-job-template module will automatically apply this toleration when using a node pool with GPUs.

Nvidia GPU drivers must be installed. The recommended approach for GKE to install GPU dirvers is by applying a DaemonSet to the cluster. See these instructions.

However, in some cases it may be desired to compile a different driver (such as a desire to install a newer version, compatibility with the Nvidia GPU-operator or other use-cases). In this case, ensure that you turn off the enable_secure_boot option to allow unsigned kernel modules to be loaded.

GPUs Examples

There are several ways to add GPUs to a GKE node pool. See docs for more info on GPUs.

The following is a node pool that uses a2 or g2 machine types which has a fixed number of attached GPUs:

  - id: simple-a2-pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g

Note: It is not necessary to define the guest_accelerator setting when using a2 or g2 machines as information about GPUs, such as type and count, is automatically inferred from the machine type.

The following scenarios require the guest_accelerator block is specified:

To partition an A100 GPU into multiple GPUs on an A2 family machine.
To specify a time sharing configuration on a GPUs.
To attach a GPU to an N1 family machine.

The following is an example of partitioning an A100 GPU:

  - id: multi-instance-gpu-pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g
      guest_accelerator:
      - type: nvidia-tesla-a100
        count: 1
        gpu_partition_size: 1g.5gb
        gpu_sharing_config: null
        gpu_driver_installation_config: null

Note: Once we define the guest_accelerator block, all fields must be defined. Use null for optional fields.

The following is an example of GPU time sharing (with partitioned GPUs):

  - id: time-sharing-gpu-pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: a2-highgpu-1g
      guest_accelerator:
      - type: nvidia-tesla-a100
        count: 1
        gpu_partition_size: 1g.5gb
        gpu_sharing_config:
        - gpu_sharing_strategy: TIME_SHARING
          max_shared_clients_per_gpu: 3
        gpu_driver_installation_config: null

Finally, the following is an example of using a GPU attached to an n1 machine:

  - id: t4-pool
    source: community/modules/compute/gke-node-pool
    use: [gke_cluster]
    settings:
      machine_type: n1-standard-16
      guest_accelerator:
      - type: nvidia-tesla-t4
        count: 2
        gpu_partition_size: null
        gpu_sharing_config: null
        gpu_driver_installation_config: null

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name	Version
terraform	>= 1.2
google	>= 4.75.1, < 5.0
google-beta	>= 4.75.1, < 5.0

Providers

Name	Version
google	>= 4.75.1, < 5.0
google-beta	>= 4.75.1, < 5.0

Modules

No modules.

Resources

Name	Type
google-beta_google_container_node_pool.node_pool	resource
google_project_iam_member.node_service_account_artifact_registry	resource
google_project_iam_member.node_service_account_gcr	resource
google_project_iam_member.node_service_account_log_writer	resource
google_project_iam_member.node_service_account_metric_writer	resource
google_project_iam_member.node_service_account_monitoring_viewer	resource
google_project_iam_member.node_service_account_resource_metadata_writer	resource
google_compute_default_service_account.default_sa	data source

Inputs

Name	Description	Type	Default	Required
auto_upgrade	Whether the nodes will be automatically upgraded.	`bool`	`false`	no
autoscaling_total_max_nodes	Total maximum number of nodes in the NodePool.	`number`	`1000`	no
autoscaling_total_min_nodes	Total minimum number of nodes in the NodePool.	`number`	`0`	no
cluster_id	projects/{{project}}/locations/{{location}}/clusters/{{cluster}}	`string`	n/a	yes
compact_placement	Places node pool's nodes in a closer physical proximity in order to reduce network latency between nodes.	`bool`	`false`	no
disk_size_gb	Size of disk for each node.	`number`	`100`	no
disk_type	Disk type for each node.	`string`	`"pd-standard"`	no
enable_gcfs	Enable the Google Container Filesystem (GCFS). See restrictions.	`bool`	`false`	no
enable_secure_boot	Enable secure boot for the nodes. Keep enabled unless custom kernel modules need to be loaded. See here for more info.	`bool`	`true`	no
guest_accelerator	List of the type and count of accelerator cards attached to the instance.	list(object({ type = string count = number gpu_driver_installation_config = list(object({ gpu_driver_version = string })) gpu_partition_size = string gpu_sharing_config = list(object({ gpu_sharing_strategy = string max_shared_clients_per_gpu = number })) }))	`null`	no
image_type	The default image type used by NAP once a new node pool is being created. Use either COS_CONTAINERD or UBUNTU_CONTAINERD.	`string`	`"COS_CONTAINERD"`	no
kubernetes_labels	Kubernetes labels to be applied to each node in the node group. Key-value pairs. (The `kubernetes.io/` and `k8s.io/` prefixes are reserved by Kubernetes Core components and cannot be specified)	`map(string)`	`null`	no
labels	GCE resource labels to be applied to resources. Key-value pairs.	`map(string)`	n/a	yes
local_ssd_count_ephemeral_storage	The number of local SSDs to attach to each node to back ephemeral storage. Uses NVMe interfaces. Must be supported by `machine_type`. See above for more info.	`number`	`0`	no
local_ssd_count_nvme_block	The number of local SSDs to attach to each node to back block storage. Uses NVMe interfaces. Must be supported by `machine_type`. See above for more info.	`number`	`0`	no
machine_type	The name of a Google Compute Engine machine type.	`string`	`"c2-standard-60"`	no
name	The name of the node pool. If left blank, will default to the machine type.	`string`	`null`	no
project_id	The project ID to host the cluster in.	`string`	n/a	yes
service_account	DEPRECATED: use service_account_email and scopes.	object({ email = string, scopes = set(string) })	`null`	no
service_account_email	Service account e-mail address to use with the node pool	`string`	`null`	no
service_account_scopes	Scopes to to use with the node pool.	`set(string)`	[ "https://www.googleapis.com/auth/cloud-platform" ]	no
spot	Provision VMs using discounted Spot pricing, allowing for preemption	`bool`	`false`	no
static_node_count	The static number of nodes in the node pool. If set, autoscaling will be disabled.	`number`	`null`	no
taints	Taints to be applied to the system node pool.	list(object({ key = string value = any effect = string }))	[ { "effect": "NO_SCHEDULE", "key": "user-workload", "value": true } ]	no
threads_per_core	Sets the number of threads per physical core. By setting threads_per_core to 2, Simultaneous Multithreading (SMT) is enabled extending the total number of virtual cores. For example, a machine of type c2-standard-60 will have 60 virtual cores with threads_per_core equal to 2. With threads_per_core equal to 1 (SMT turned off), only the 30 physical cores will be available on the VM. The default value of "0" will turn off SMT for supported machine types, and will fall back to GCE defaults for unsupported machine types (t2d, shared-core instances, or instances with less than 2 vCPU). Disabling SMT can be more performant in many HPC workloads, therefore it is disabled by default where compatible. null = SMT configuration will use the GCE defaults for the machine type 0 = SMT will be disabled where compatible (default) 1 = SMT will always be disabled (will fail on incompatible machine types) 2 = SMT will always be enabled (will fail on incompatible machine types)	`number`	`0`	no
timeout_create	Timeout for creating a node pool	`string`	`null`	no
timeout_update	Timeout for updating a node pool	`string`	`null`	no
total_max_nodes	DEPRECATED: Use autoscaling_total_max_nodes.	`number`	`null`	no
total_min_nodes	DEPRECATED: Use autoscaling_total_min_nodes.	`number`	`null`	no
zones	A list of zones to be used. Zones must be in region of cluster. If null, cluster zones will be inherited. Note `zones` not `zone`; does not work with `zone` deployment variable.	`list(string)`	`null`	no

Outputs

Name	Description
allocatable_cpu_per_node	Number of CPUs available for scheduling pods on each node.
has_gpu	Boolean value indicating whether nodes in the pool are configured with GPUs.
node_pool_name	Name of the node pool.
tolerations	Tolerations needed for a pod to be scheduled on this node pool.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Description

Example

Taints and Tolerations

Local SSD Storage

Considerations with GPUs

GPUs Examples

License

Requirements

Providers

Modules

Resources

Inputs

Outputs

Files

README.md

Latest commit

History

README.md

File metadata and controls

Description

Example

Taints and Tolerations

Local SSD Storage

Considerations with GPUs

GPUs Examples

License

Requirements

Providers

Modules

Resources

Inputs

Outputs