Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Add support for Outposts, remove node security group, add support for addon preserve and most_recent configurations #2250

Merged
merged 36 commits into from
Dec 5, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
862fd61
refactor: Change default NTP CIDR blocks to use those provided by Amazon
bryantbiggs Aug 20, 2022
136d7ed
feat: Add support for specifying `most_recent` and `preserve` argumen…
bryantbiggs Aug 20, 2022
ad645c4
refactor: Allow both static and computed additional policies to be at…
bryantbiggs Aug 20, 2022
ba892fa
refactor: Remove empty, default security group created per node group
bryantbiggs Aug 20, 2022
fc80020
chore: Remove Karpenter example from examples
bryantbiggs Aug 20, 2022
aec1532
chore: Update local variable name to align with sub-modules
bryantbiggs Aug 20, 2022
6b8c4ec
feat: Change default behavior of KMS key creation to true
bryantbiggs Aug 20, 2022
780a737
feat: Update documentation for v19.x changes
bryantbiggs Aug 20, 2022
002be2f
feat: Update defaults, add variable for custom launch template usage,…
bryantbiggs Aug 25, 2022
c03f67c
docs: Update docs for changes around security groups
bryantbiggs Sep 26, 2022
8b176db
feat: Add support for provisioning on outposts and addon timeout conf…
bryantbiggs Sep 26, 2022
911fe42
fix: Move additional IAM policies to separate attachment resource, up…
bryantbiggs Sep 27, 2022
d3d4f38
chore: Remove VPC `"shared"` subnet tags which are no longer required
bryantbiggs Sep 28, 2022
b1615ae
fix: Correct use of iterating over maps with `for_each` and computed …
bryantbiggs Sep 28, 2022
291e0b7
chore: Update formatting and example content
bryantbiggs Sep 29, 2022
385d758
docs: Update upgrade guide for v19.x
bryantbiggs Sep 29, 2022
c304b08
Merge branch 'master' of github.com:bryantbiggs/terraform-aws-eks int…
bryantbiggs Oct 14, 2022
ed0c336
fix: Remove all references of `aws_default_tags` to avoid update conf…
bryantbiggs Oct 18, 2022
4f651e1
feat: Add support for `service_ipv6_cidr` within the EKS cluster netw…
bryantbiggs Oct 18, 2022
0738569
chore: Update upgrade documentation and fix CI checks
bryantbiggs Oct 18, 2022
87ced51
Fixed IAM policy cluster_encryption when create_kms_key is false
antonbabenko Oct 25, 2022
5fa0e33
chore: Updates from PR review feedback
bryantbiggs Oct 25, 2022
e782ce5
docs: Fix v18 diff format for additional IAM policies
bryantbiggs Oct 25, 2022
cea7595
feat: Update `self-managed-node-group` to use latest settings provide…
bryantbiggs Oct 26, 2022
4754012
feat: Update `eks-managed-node-group` to use latest settings provided…
bryantbiggs Oct 26, 2022
2596ea3
feat: Add example for EKS on Outposts with local cluster
bryantbiggs Oct 26, 2022
2b69482
fix: Updates from testing with Outposts, still need final remote vali…
bryantbiggs Oct 28, 2022
15d2836
feat: Update defaults; simplify security group settings and forward c…
bryantbiggs Nov 1, 2022
63f2d89
chore: Clean up and simplify examples
bryantbiggs Nov 8, 2022
a4b83b7
Merge branch 'master' of github.com:bryantbiggs/terraform-aws-eks int…
bryantbiggs Nov 8, 2022
43e57dd
Update main.tf
bryantbiggs Dec 3, 2022
701c9c5
Merge branch 'master' of github.com:bryantbiggs/terraform-aws-eks int…
bryantbiggs Dec 3, 2022
0ed15dc
chore: Update AWS provider to support changes required by Outposts
bryantbiggs Dec 3, 2022
f6fb069
fix: Correct `cluster_id` output value for recent provider changes
bryantbiggs Dec 5, 2022
90b6614
fix: Updates from testing on Outpost lab
bryantbiggs Dec 5, 2022
eff1df7
fix: Correct Karpenter example for auth, update Karpenter module for …
bryantbiggs Dec 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/images/security_groups.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.76.0
rev: v1.77.0
hooks:
- id: terraform_fmt
- id: terraform_validate
Expand All @@ -23,7 +23,7 @@ repos:
- '--args=--only=terraform_standard_module_structure'
- '--args=--only=terraform_workspace_remote'
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
rev: v4.4.0
hooks:
- id: check-merge-conflict
- id: end-of-file-fixer
93 changes: 55 additions & 38 deletions README.md

Large diffs are not rendered by default.

466 changes: 466 additions & 0 deletions docs/UPGRADE-19.0.md

Large diffs are not rendered by default.

48 changes: 23 additions & 25 deletions docs/compute_resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
```hcl
eks_managed_node_groups = {
default = {
create_launch_template = false
launch_template_name = ""
use_custom_launch_template = false
}
}
```
Expand All @@ -29,8 +28,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
```hcl
eks_managed_node_groups = {
bottlerocket_default = {
create_launch_template = false
launch_template_name = ""
use_custom_launch_template = false

ami_type = "BOTTLEROCKET_x86_64"
platform = "bottlerocket"
Expand All @@ -45,15 +43,15 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
prepend_userdata = {
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
EOT
}
}
Expand All @@ -68,9 +66,9 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
platform = "bottlerocket"

bootstrap_extra_args = <<-EOT
# extra args added
[settings.kernel]
lockdown = "integrity"
# extra args added
[settings.kernel]
lockdown = "integrity"
EOT
}
}
Expand Down Expand Up @@ -116,17 +114,17 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
enable_bootstrap_user_data = true
# this will get added to the template
bootstrap_extra_args = <<-EOT
# extra args added
[settings.kernel]
lockdown = "integrity"
# extra args added
[settings.kernel]
lockdown = "integrity"

[settings.kubernetes.node-labels]
"label1" = "foo"
"label2" = "bar"
[settings.kubernetes.node-labels]
"label1" = "foo"
"label2" = "bar"

[settings.kubernetes.node-taints]
"dedicated" = "experimental:PreferNoSchedule"
"special" = "true:NoSchedule"
[settings.kubernetes.node-taints]
"dedicated" = "experimental:PreferNoSchedule"
"special" = "true:NoSchedule"
EOT
}
}
Expand Down
36 changes: 0 additions & 36 deletions docs/faq.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Frequently Asked Questions

- [I received an error: `expect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME> ...`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#i-received-an-error-expect-exactly-one-securitygroup-tagged-with-kubernetesioclustername-)
- [I received an error: `Error: Invalid for_each argument ...`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#i-received-an-error-error-invalid-for_each-argument-)
- [Why are nodes not being registered?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-nodes-not-being-registered)
- [Why are there no changes when a node group's `desired_size` is modified?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-there-no-changes-when-a-node-groups-desired_size-is-modified)
- [How can I deploy Windows based nodes?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#how-can-i-deploy-windows-based-nodes)
Expand Down Expand Up @@ -48,41 +47,6 @@ By default, EKS creates a cluster primary security group that is created outside

In theory, if you are attaching the cluster primary security group, you shouldn't need to use the shared node security group created by the module. However, this is left up to users to decide for their requirements and use case.

### I received an error: `Error: Invalid for_each argument ...`

Users may encounter an error such as `Error: Invalid for_each argument - The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply ...`

This error is due to an upstream issue with [Terraform core](https://github.com/hashicorp/terraform/issues/4149). There are two potential options you can take to help mitigate this issue:

1. Create the dependent resources before the cluster => `terraform apply -target <your policy or your security group>` and then `terraform apply` for the cluster (or other similar means to just ensure the referenced resources exist before creating the cluster)

- Note: this is the route users will have to take for adding additional security groups to nodes since there isn't a separate "security group attachment" resource

2. For additional IAM policies, users can attach the policies outside of the cluster definition as demonstrated below

```hcl
resource "aws_iam_role_policy_attachment" "additional" {
for_each = module.eks.eks_managed_node_groups
# you could also do the following or any combination:
# for_each = merge(
# module.eks.eks_managed_node_groups,
# module.eks.self_managed_node_group,
# module.eks.fargate_profile,
# )

# This policy does not have to exist at the time of cluster creation. Terraform can
# deduce the proper order of its creation to avoid errors during creation
policy_arn = aws_iam_policy.node_additional.arn
role = each.value.iam_role_name
}
```

TL;DR - Terraform resource passed into the modules map definition _must_ be known before you can apply the EKS module. The variables this potentially affects are:

- `cluster_security_group_additional_rules` (i.e. - referencing an external security group resource in a rule)
- `node_security_group_additional_rules` (i.e. - referencing an external security group resource in a rule)
- `iam_role_additional_policies` (i.e. - referencing an external policy resource)

### Why are nodes not being registered?

Nodes not being able to register with the EKS control plane is generally due to networking mis-configurations.
Expand Down
3 changes: 1 addition & 2 deletions docs/network_connectivity.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ Please refer to the [AWS documentation](https://docs.aws.amazon.com/eks/latest/u
- Lastly, users are able to opt in to attaching the primary security group automatically created by the EKS service by setting `attach_cluster_primary_security_group` = `true` from the root module for the respective node group (or set it within the node group defaults). This security group is not managed by the module; it is created by the EKS service. It permits all traffic within the domain of the security group as well as all egress traffic to the internet.

- Node Group Security Group(s)
- Each node group (EKS Managed Node Group and Self Managed Node Group) by default creates its own security group. By default, this security group does not contain any additional security group rules. It is merely an "empty container" that offers users the ability to opt into any addition inbound our outbound rules as necessary
- Users also have the option to supply their own, and/or additional, externally created security group(s) to the node group as well via the `vpc_security_group_ids` variable
- Users have the option to assign their own externally created security group(s) to the node group via the `vpc_security_group_ids` variable

See the example snippet below which adds additional security group rules to the cluster security group as well as the shared node security group (for node-to-node access). Users can use this extensibility to open up network access as they see fit using the security groups provided by the module:

Expand Down
18 changes: 9 additions & 9 deletions docs/user_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,15 @@ Since the EKS Managed Node Group service provides the necessary bootstrap user d
```hcl
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
EOT
```

Expand Down
12 changes: 7 additions & 5 deletions examples/complete/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,15 @@ Note that this example may create resources which cost money. Run `terraform des

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.45 |
| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.72 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.45 |

## Modules

Expand All @@ -61,7 +61,9 @@ Note that this example may create resources which cost money. Run `terraform des

| Name | Type |
|------|------|
| [aws_iam_policy.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_security_group.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource |
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |

## Inputs

Expand All @@ -81,9 +83,9 @@ No inputs.
| <a name="output_cluster_iam_role_arn"></a> [cluster\_iam\_role\_arn](#output\_cluster\_iam\_role\_arn) | IAM role ARN of the EKS cluster |
| <a name="output_cluster_iam_role_name"></a> [cluster\_iam\_role\_name](#output\_cluster\_iam\_role\_name) | IAM role name of the EKS cluster |
| <a name="output_cluster_iam_role_unique_id"></a> [cluster\_iam\_role\_unique\_id](#output\_cluster\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| <a name="output_cluster_id"></a> [cluster\_id](#output\_cluster\_id) | The id of the EKS cluster. Will block on cluster creation until the cluster is really ready |
| <a name="output_cluster_id"></a> [cluster\_id](#output\_cluster\_id) | The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts |
| <a name="output_cluster_identity_providers"></a> [cluster\_identity\_providers](#output\_cluster\_identity\_providers) | Map of attribute maps for all EKS identity providers enabled |
| <a name="output_cluster_name"></a> [cluster\_name](#output\_cluster\_name) | The name of the EKS cluster. Will block on cluster creation until the cluster is really ready |
| <a name="output_cluster_name"></a> [cluster\_name](#output\_cluster\_name) | The name of the EKS cluster |
| <a name="output_cluster_oidc_issuer_url"></a> [cluster\_oidc\_issuer\_url](#output\_cluster\_oidc\_issuer\_url) | The URL on the EKS cluster for the OpenID Connect identity provider |
| <a name="output_cluster_platform_version"></a> [cluster\_platform\_version](#output\_cluster\_platform\_version) | Platform version for the cluster |
| <a name="output_cluster_security_group_arn"></a> [cluster\_security\_group\_arn](#output\_cluster\_security\_group\_arn) | Amazon Resource Name (ARN) of the cluster security group |
Expand Down
Loading