Skip to content

Commit

Permalink
add a default EFS CSI driver taint to node pools (#235)
Browse files Browse the repository at this point in the history 
https://github.com/kubernetes-sigs/aws-efs-csi-driver?tab=readme-ov-file#configure-node-startup-taint
  • Loading branch information
@steved
steved committed Jul 30, 2024
1 parent cd2c348 commit 8d6bd0c
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 3 deletions.
4 changes: 2 additions & 2 deletions modules/nodes/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ No modules.

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number)<br> desired_per_az = number<br> availability_zone_ids = list(string)<br> labels = map(string)<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = object({<br> size = string<br> type = string<br> iops = optional(number)<br> throughput = optional(number, 500)<br> })<br> }))</pre> | `{}` | no |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m6i.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number, null)<br> desired_per_az = optional(number, 0)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 1000)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number, 500)<br> }), {<br> size = 1000<br> type = "gp3"<br> iops = null<br> throughput = 500<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m7i-flex.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 1)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, null)<br> max_unavailable = optional(number, 1)<br> desired_per_az = optional(number, 1)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), []<br> )<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number)<br> }), {<br> size = 100<br> type = "gp3"<br> iops = null<br> throughput = null<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["g5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number, null)<br> desired_per_az = optional(number, 0)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [{<br> key = "nvidia.com/gpu"<br> value = "true"<br> effect = "NO_SCHEDULE"<br> },<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br><br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 1000)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number, 500)<br> }), {<br> size = 1000<br> type = "gp3"<br> iops = null<br> throughput = 500<br> }<br> )<br> })<br> })</pre> | n/a | yes |
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number)<br> desired_per_az = number<br> availability_zone_ids = list(string)<br> labels = map(string)<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> },<br> {<br> key = "efs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br><br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = object({<br> size = string<br> type = string<br> iops = optional(number)<br> throughput = optional(number, 500)<br> })<br> }))</pre> | `{}` | no |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m6i.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number, null)<br> desired_per_az = optional(number, 0)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> },<br> {<br> key = "efs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 1000)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number, 500)<br> }), {<br> size = 1000<br> type = "gp3"<br> iops = null<br> throughput = 500<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m7i-flex.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 1)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, null)<br> max_unavailable = optional(number, 1)<br> desired_per_az = optional(number, 1)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), []<br> )<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number)<br> }), {<br> size = 100<br> type = "gp3"<br> iops = null<br> throughput = null<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string, null)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["g5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> max_unavailable_percentage = optional(number, 50)<br> max_unavailable = optional(number, null)<br> desired_per_az = optional(number, 0)<br> availability_zone_ids = list(string)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({<br> key = string<br> value = optional(string)<br> effect = string<br> })), [{<br> key = "nvidia.com/gpu"<br> value = "true"<br> effect = "NO_SCHEDULE"<br> },<br> {<br> key = "ebs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> },<br> {<br> key = "efs.csi.aws.com/agent-not-ready",<br> value = "true",<br> effect = "NO_EXECUTE"<br> }<br> ])<br> tags = optional(map(string), {})<br> gpu = optional(bool, null)<br> volume = optional(object({<br> size = optional(number, 1000)<br> type = optional(string, "gp3")<br> iops = optional(number)<br> throughput = optional(number, 500)<br> }), {<br> size = 1000<br> type = "gp3"<br> iops = null<br> throughput = 500<br> }<br> )<br> })<br> })</pre> | n/a | yes |
| <a name="input_eks_info"></a> [eks\_info](#input\_eks\_info) | cluster = {<br> addons = List of addons<br> specs = Cluster spes. {<br> name = Cluster name.<br> endpoint = Cluster endpont.<br> kubernetes\_network\_config = Cluster k8s nw config.<br> }<br> version = K8s version.<br> arn = EKS Cluster arn.<br> security\_group\_id = EKS Cluster security group id.<br> endpoint = EKS Cluster API endpoint.<br> roles = Default IAM Roles associated with the EKS cluster. {<br> name = string<br> arn = string<br> }<br> custom\_roles = Custom IAM Roles associated with the EKS cluster. {<br> rolearn = string<br> username = string<br> groups = list(string)<br> }<br> oidc = {<br> arn = OIDC provider ARN.<br> url = OIDC provider url.<br> }<br> }<br> nodes = {<br> security\_group\_id = EKS Nodes security group id.<br> roles = IAM Roles associated with the EKS Nodes.{<br> name = string<br> arn = string<br> }<br> }<br> kubeconfig = Kubeconfig details.{<br> path = string<br> extra\_args = string<br> } | <pre>object({<br> k8s_pre_setup_sh_file = string<br> cluster = object({<br> addons = optional(list(string), ["kube-proxy", "coredns", "vpc-cni"])<br> vpc_cni = optional(object({<br> prefix_delegation = optional(bool, false)<br> annotate_pod_ip = optional(bool, true)<br> }))<br> specs = object({<br> name = string<br> endpoint = string<br> kubernetes_network_config = list(map(any))<br> certificate_authority = list(map(any))<br> })<br> version = string<br> arn = string<br> security_group_id = string<br> endpoint = string<br> roles = list(object({<br> name = string<br> arn = string<br> }))<br> custom_roles = list(object({<br> rolearn = string<br> username = string<br> groups = list(string)<br> }))<br> oidc = object({<br> arn = string<br> url = string<br> })<br> })<br> nodes = object({<br> security_group_id = string<br> roles = list(object({<br> name = string<br> arn = string<br> }))<br> })<br> kubeconfig = object({<br> path = string<br> extra_args = string<br> })<br> })</pre> | n/a | yes |
| <a name="input_ignore_tags"></a> [ignore\_tags](#input\_ignore\_tags) | Tag keys to be ignored by the aws provider. | `list(string)` | `[]` | no |
| <a name="input_kms_info"></a> [kms\_info](#input\_kms\_info) | key\_id = KMS key id.<br> key\_arn = KMS key arn.<br> enabled = KMS key is enabled | <pre>object({<br> key_id = string<br> key_arn = string<br> enabled = bool<br> })</pre> | n/a | yes |
Expand Down
17 changes: 16 additions & 1 deletion modules/nodes/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,11 @@ variable "default_node_groups" {
key = "ebs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
},
{
key = "efs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
}
])
tags = optional(map(string), {})
Expand Down Expand Up @@ -256,8 +261,12 @@ variable "default_node_groups" {
key = "ebs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
},
{
key = "efs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
}

])
tags = optional(map(string), {})
gpu = optional(bool, null)
Expand Down Expand Up @@ -300,7 +309,13 @@ variable "additional_node_groups" {
key = "ebs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
},
{
key = "efs.csi.aws.com/agent-not-ready",
value = "true",
effect = "NO_EXECUTE"
}

])
tags = optional(map(string), {})
gpu = optional(bool, null)
Expand Down

0 comments on commit 8d6bd0c

Please sign in to comment.