Skip to content

Commit

Permalink
feat: Add worker groups submodules
Browse files Browse the repository at this point in the history
  • Loading branch information
@js-timbirkett
js-timbirkett committed Mar 10, 2020
1 parent dc5cd8b commit 1db1d2a
Show file tree
Hide file tree
Showing 11 changed files with 785 additions and 9 deletions.
45 changes: 42 additions & 3 deletions modules/worker_groups/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,55 @@
# eks `worker_groups` submodule

This submodule is designed for use by both the parent `eks` module and by the user.

> :warning: **Launch Configuration driven worker groups have been superceded by Launch Template driven worker groups**
`worker_groups` is a map of maps. Key of first level will be used as unique value for `for_each` resources and in the `aws_autoscaling_group` and `aws_launch_template` name. Inner map can take the below values.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Providers

No provider.
| Name | Version |
|------|---------|
| aws | >= 2.44.0 |
| random | >= 2.1 |
| template | >= 2.1 |

## Inputs

No input.
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:-----:|
| attach\_worker\_cni\_policy | Whether to attach the Amazon managed `AmazonEKS_CNI_Policy` IAM policy to the default worker IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster. | `bool` | `true` | no |
| attach\_worker\_groups\_cni\_policy | Whether to attach the Amazon managed `AmazonEKS_CNI_Policy` IAM policy to the default node groups IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster. | `bool` | `true` | no |
| cluster\_name | Name of parent cluster. | `string` | n/a | yes |
| cluster\_security\_group\_id | If provided, the EKS cluster will be attached to this security group. If not given, a security group will be created with necessary ingress/egress to work with the workers | `string` | n/a | yes |
| create\_eks | Controls if EKS resources should be created (it affects almost all resources). | `bool` | `true` | no |
| iam\_path | If provided, all IAM roles will be created on this path. | `string` | `"/"` | no |
| manage\_worker\_groups\_iam\_resources | Whether to let the module manage node group IAM resources. If set to false, iam\_instance\_profile\_name must be specified for workers. | `bool` | `true` | no |
| manage\_worker\_iam\_resources | Whether to let the module manage worker IAM resources. If set to false, iam\_instance\_profile\_name must be specified for workers. | `bool` | `true` | no |
| permissions\_boundary | If provided, all IAM roles will be created with this permissions boundary attached. | `string` | n/a | yes |
| subnets | A list of subnets to place the EKS cluster and workers within. | `list(string)` | n/a | yes |
| tags | A map of tags to add to all resources. | `map(string)` | n/a | yes |
| vpc\_id | VPC where the cluster and workers will be deployed. | `string` | n/a | yes |
| worker\_additional\_security\_group\_ids | A list of additional security group ids to attach to worker instances | `list(string)` | `[]` | no |
| worker\_ami\_name\_filter | Name filter for AWS EKS worker AMI. If not provided, the latest official AMI for the specified 'cluster\_version' is used. | `string` | `""` | no |
| worker\_ami\_name\_filter\_windows | Name filter for AWS EKS Windows worker AMI. If not provided, the latest official AMI for the specified 'cluster\_version' is used. | `string` | `""` | no |
| worker\_ami\_owner\_id | The ID of the owner for the AMI to use for the AWS EKS workers. Valid values are an AWS account ID, 'self' (the current account), or an AWS owner alias (e.g. 'amazon', 'aws-marketplace', 'microsoft'). | `string` | `"602401143452"` | no |
| worker\_ami\_owner\_id\_windows | The ID of the owner for the AMI to use for the AWS EKS Windows workers. Valid values are an AWS account ID, 'self' (the current account), or an AWS owner alias (e.g. 'amazon', 'aws-marketplace', 'microsoft'). | `string` | `"801119661308"` | no |
| worker\_create\_initial\_lifecycle\_hooks | Whether to create initial lifecycle hooks provided in worker groups. | `bool` | `false` | no |
| worker\_create\_security\_group | Whether to create a security group for the workers or attach the workers to `worker_security_group_id`. | `bool` | `true` | no |
| worker\_groups | Map of map of worker groups to create. See documentation above for more details. | `any` | `{}` | no |
| worker\_groups\_additional\_policies | Additional policies to be added to node groups. | `list(string)` | `[]` | no |
| worker\_groups\_defaults | Map of values to be applied to all node groups. See documentation above for more details. | `any` | `{}` | no |
| worker\_groups\_role\_name | User defined node groups role name. | `string` | `""` | no |
| worker\_sg\_ingress\_from\_port | Minimum port number from which pods will accept communication. Must be changed to a lower value if some pods in your cluster will expose a port lower than 1025 (e.g. 22, 80, or 443). | `number` | `1025` | no |
| workers\_additional\_policies | Additional policies to be added to workers | `list(string)` | `[]` | no |
| workers\_role\_name | User defined workers role name. | `string` | `""` | no |

## Outputs

No output.
| Name | Description |
|------|-------------|
| aws\_auth\_roles | Roles for use in aws-auth ConfigMap |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
88 changes: 88 additions & 0 deletions modules/worker_groups/data.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
locals {
worker_ami_name_filter = var.worker_ami_name_filter != "" ? var.worker_ami_name_filter : "amazon-eks-node-${data.aws_eks_cluster.cluster.version}-v*"

# Windows nodes are available from k8s 1.14. If cluster version is less than 1.14, fix ami filter to some constant to not fail on 'terraform plan'.
worker_ami_name_filter_windows = (var.worker_ami_name_filter_windows != "" ?
var.worker_ami_name_filter_windows : "Windows_Server-2019-English-Core-EKS_Optimized-${tonumber(data.aws_eks_cluster.cluster.version) >= 1.14 ? data.aws_eks_cluster.cluster.version : 1.14}-*"
)
}

data "aws_eks_cluster" "cluster" {
name = var.cluster_name
}

data "aws_iam_policy_document" "workers_assume_role_policy" {
statement {
sid = "EKSWorkerAssumeRole"

actions = [
"sts:AssumeRole",
]

principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}

data "aws_ami" "eks_worker" {
filter {
name = "name"
values = [local.worker_ami_name_filter]
}

most_recent = true

owners = [var.worker_ami_owner_id]
}

data "aws_ami" "eks_worker_windows" {
filter {
name = "name"
values = [local.worker_ami_name_filter_windows]
}

filter {
name = "platform"
values = ["windows"]
}

most_recent = true

owners = [var.worker_ami_owner_id_windows]
}

data "template_file" "launch_template_userdata" {
for_each = local.worker_groups_expanded

template = coalesce(
each.value["userdata_template_file"],
file(
each.value["platform"] == "windows"
? "${path.module}/templates/userdata_windows.tpl"
: "${path.module}/templates/userdata.sh.tpl"
)
)

vars = merge({
platform = each.value["platform"]
cluster_name = var.cluster_name
endpoint = data.aws_eks_cluster.cluster.endpoint
cluster_auth_base64 = data.aws_eks_cluster.cluster.certificate_authority.0.data
pre_userdata = each.value["pre_userdata"]
additional_userdata = each.value["additional_userdata"]
bootstrap_extra_args = each.value["bootstrap_extra_args"]
kubelet_extra_args = each.value["kubelet_extra_args"]
},
each.value["userdata_template_extra_args"]
)
}

data "aws_iam_instance_profile" "custom_worker_group_launch_template_iam_instance_profile" {
for_each = local.worker_groups_expanded_for_iam_data

name = each.value["iam_instance_profile_name"]
}

data "aws_region" "current" {}
125 changes: 125 additions & 0 deletions modules/worker_groups/locals.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
locals {
asg_tags = [
for item in keys(var.tags) :
map(
"key", item,
"value", element(values(var.tags), index(keys(var.tags), item)),
"propagate_at_launch", "true"
)
]

default_iam_role_id = concat(aws_iam_role.workers.*.id, [""])[0]

default_ami_id_linux = data.aws_ami.eks_worker.id
default_ami_id_windows = data.aws_ami.eks_worker_windows.id

worker_groups_defaults = {
name = "" # Name of the worker group. Literal count.index will never be used but if name is not set, the count.index interpolation will be used.
tags = [] # A list of map defining extra tags to be applied to the worker group autoscaling group.
ami_id = "" # AMI ID for the eks workers. If none is provided, Terraform will search for the latest version of their EKS optimized worker AMI based on platform.
desired_capacity = "1" # Desired worker capacity in the autoscaling group and changing its value will not affect the autoscaling group's desired capacity because the cluster-autoscaler manages up and down scaling of the nodes. Cluster-autoscaler add nodes when pods are in pending state and remove the nodes when they are not required by modifying the desirec_capacity of the autoscaling group. Although an issue exists in which if the value of the min_size is changed it modifies the value of desired_capacity.
max_size = "3" # Maximum worker capacity in the autoscaling group.
min_size = "1" # Minimum worker capacity in the autoscaling group. NOTE: Change in this paramater will affect the desired_capacity, like changing its value to 2 will change desired_capacity value to 2 but bringing back it to 1 will not affect the desired_capacity.
force_delete = false # Enable forced deletion for the autoscaling group.
initial_lifecycle_hooks = [] # Initital lifecycle hook for the autoscaling group.
recreate_on_change = false # Recreate the autoscaling group when the Launch Template or Launch Configuration change.
instance_type = "m4.large" # Size of the workers instances.
spot_price = "" # Cost of spot instance.
placement_tenancy = "" # The tenancy of the instance. Valid values are "default" or "dedicated".
root_volume_size = "100" # root volume size of workers instances.
root_volume_type = "gp2" # root volume type of workers instances, can be 'standard', 'gp2', or 'io1'
root_iops = "0" # The amount of provisioned IOPS. This must be set with a volume_type of "io1".
key_name = "" # The key name that should be used for the instances in the autoscaling group
pre_userdata = "" # userdata to pre-append to the default userdata.
userdata_template_file = "" # alternate template to use for userdata
userdata_template_extra_args = {} # Additional arguments to use when expanding the userdata template file
bootstrap_extra_args = "" # Extra arguments passed to the bootstrap.sh script from the EKS AMI (Amazon Machine Image).
additional_userdata = "" # userdata to append to the default userdata.
ebs_optimized = true # sets whether to use ebs optimization on supported types.
enable_monitoring = true # Enables/disables detailed monitoring.
public_ip = false # Associate a public ip address with a worker
kubelet_extra_args = "" # This string is passed directly to kubelet if set. Useful for adding labels or taints.
subnets = var.subnets # A list of subnets to place the worker nodes in. i.e. ["subnet-123", "subnet-456", "subnet-789"]
additional_security_group_ids = [] # A list of additional security group ids to include in worker launch config
protect_from_scale_in = false # Prevent AWS from scaling in, so that cluster-autoscaler is solely responsible.
iam_instance_profile_name = "" # A custom IAM instance profile name. Used when manage_worker_iam_resources is set to false. Incompatible with iam_role_id.
iam_role_id = local.default_iam_role_id # A custom IAM role id. Incompatible with iam_instance_profile_name. Literal local.default_iam_role_id will never be used but if iam_role_id is not set, the local.default_iam_role_id interpolation will be used.
suspended_processes = ["AZRebalance"] # A list of processes to suspend. i.e. ["AZRebalance", "HealthCheck", "ReplaceUnhealthy"]
target_group_arns = null # A list of Application LoadBalancer (ALB) target group ARNs to be associated to the autoscaling group
enabled_metrics = [] # A list of metrics to be collected i.e. ["GroupMinSize", "GroupMaxSize", "GroupDesiredCapacity"]
placement_group = "" # The name of the placement group into which to launch the instances, if any.
service_linked_role_arn = "" # Arn of custom service linked role that Auto Scaling group will use. Useful when you have encrypted EBS
termination_policies = [] # A list of policies to decide how the instances in the auto scale group should be terminated.
platform = "linux" # Platform of workers. either "linux" or "windows"
# Settings for launch templates
root_block_device_name = data.aws_ami.eks_worker.root_device_name # Root device name for workers. If non is provided, will assume default AMI was used.
root_kms_key_id = "" # The KMS key to use when encrypting the root storage device
launch_template_version = "$Latest" # The lastest version of the launch template to use in the autoscaling group
launch_template_placement_tenancy = "default" # The placement tenancy for instances
launch_template_placement_group = "" # The name of the placement group into which to launch the instances, if any.
root_encrypted = "" # Whether the volume should be encrypted or not
eni_delete = true # Delete the Elastic Network Interface (ENI) on termination (if set to false you will have to manually delete before destroying)
cpu_credits = "standard" # T2/T3 unlimited mode, can be 'standard' or 'unlimited'. Used 'standard' mode as default to avoid paying higher costs
market_type = null
# Settings for launch templates with mixed instances policy
override_instance_types = ["m5.large", "m5a.large", "m5d.large", "m5ad.large"] # A list of override instance types for mixed instances policy
on_demand_allocation_strategy = null # Strategy to use when launching on-demand instances. Valid values: prioritized.
on_demand_base_capacity = "0" # Absolute minimum amount of desired capacity that must be fulfilled by on-demand instances
on_demand_percentage_above_base_capacity = "0" # Percentage split between on-demand and Spot instances above the base on-demand capacity
spot_allocation_strategy = "lowest-price" # Valid options are 'lowest-price' and 'capacity-optimized'. If 'lowest-price', the Auto Scaling group launches instances using the Spot pools with the lowest price, and evenly allocates your instances across the number of Spot pools. If 'capacity-optimized', the Auto Scaling group launches instances using Spot pools that are optimally chosen based on the available Spot capacity.
spot_instance_pools = 10 # "Number of Spot pools per availability zone to allocate capacity. EC2 Auto Scaling selects the cheapest Spot pools and evenly allocates Spot capacity across the number of Spot pools that you specify."
spot_max_price = "" # Maximum price per unit hour that the user is willing to pay for the Spot instances. Default is the on-demand price
}

# Merge defaults and per-group values to make code cleaner
worker_groups_expanded = { for k, v in var.worker_groups : k => merge(
local.worker_groups_defaults,
var.worker_groups_defaults,
v,
) if var.create_eks }

worker_groups_expanded_for_sg = var.worker_create_security_group ? local.worker_groups_expanded : {}

worker_groups_expanded_for_iam_resource = var.manage_worker_iam_resources ? local.worker_groups_expanded : {}
worker_groups_expanded_for_iam_data = var.manage_worker_iam_resources ? {} : local.worker_groups_expanded

policy_arn_prefix = contains(["cn-northwest-1", "cn-north-1"], data.aws_region.current.name) ? "arn:aws-cn:iam::aws:policy" : "arn:aws:iam::aws:policy"

ebs_optimized_not_supported = [
"c1.medium",
"c3.8xlarge",
"c3.large",
"c5d.12xlarge",
"c5d.24xlarge",
"c5d.metal",
"cc2.8xlarge",
"cr1.8xlarge",
"g2.8xlarge",
"g4dn.metal",
"hs1.8xlarge",
"i2.8xlarge",
"m1.medium",
"m1.small",
"m2.xlarge",
"m3.large",
"m3.medium",
"m5ad.16xlarge",
"m5ad.8xlarge",
"m5dn.metal",
"m5n.metal",
"r3.8xlarge",
"r3.large",
"r5ad.16xlarge",
"r5ad.8xlarge",
"r5dn.metal",
"r5n.metal",
"t1.micro",
"t2.2xlarge",
"t2.large",
"t2.medium",
"t2.micro",
"t2.nano",
"t2.small",
"t2.xlarge"
]
}
Empty file removed modules/worker_groups/main.tf
View file
Empty file.
9 changes: 9 additions & 0 deletions modules/worker_groups/outputs.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
output "aws_auth_roles" {
description = "Roles for use in aws-auth ConfigMap"
value = [
for k, v in local.worker_groups_expanded : {
instance_role_arn = lookup(v, "iam_role_arn", aws_iam_role.workers[0].arn)
platform = v["platform"]
}
]
}
26 changes: 26 additions & 0 deletions modules/worker_groups/random.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
resource "random_pet" "worker_groups" {
for_each = local.worker_groups_expanded

separator = "-"
length = 2

keepers = {
ami_id = lookup(each.value, "ami_id", null)
root_volume_size = lookup(each.value, "root_volume_size", null)
instance_type = each.value["instance_type"]

override_instance_types = join("|", compact(
lookup(each.value, "override_instance_types", [])
))

iam_role_id = each.value["iam_role_id"]
key_name = each.value["key_name"]

source_security_group_ids = join("|", compact(
lookup(each.value, "source_security_group_ids", [])
))

subnet_ids = join("|", each.value["subnets"])
worker_group_name = join("-", [var.cluster_name, each.key])
}
}
10 changes: 10 additions & 0 deletions modules/worker_groups/templates/userdata.sh.tpl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash -xe

# Allow user supplied pre userdata code
${pre_userdata}

# Bootstrap and join the cluster
/etc/eks/bootstrap.sh --b64-cluster-ca '${cluster_auth_base64}' --apiserver-endpoint '${endpoint}' ${bootstrap_extra_args} --kubelet-extra-args "${kubelet_extra_args}" '${cluster_name}'

# Allow user supplied userdata code
${additional_userdata}
Loading

0 comments on commit 1db1d2a

Please sign in to comment.