feat: Add karpenter on fargate example #1187

micbegin · 2022-11-17T05:06:37Z

What does this PR do?

Add an example that demonstrates how to create a cluster using Fargate profiles in which to run Karpenter and other add-ons. More specifically how to implement the Run the Karpenter controller on EKS Fargate best practice and provide a demo application so customers can test this use case using the EKS blueprints.

This example also uses spot instances, SQS for the interruption queue and an ALB deployed by the AWS Load Balancer Controller (running on Fargate) to access the demo app (running on worker nodes deployed by Karpenter).

The demo app deployment is also using nodeSelector, topologySpreadConstraints, and podAntiAffinity to demonstrate how Karpenter handles these types of constraints.

Motivation

Resolves Example - Karpenter on Fargate #1159

More

Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
Yes, I have added a new example under examples to support my PR
Yes, I have created another PR for add-ons under add-ons repo (if applicable)
Yes, I have updated the docs for this feature
Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs require a new example and/or doc page. In general:

Use an existing example when possible to demonstrate a new addons usage
A new docs page under docs/add-ons/* is required for new a new addon

For Moderators

E2E Test successfully complete before merge?

Additional Notes

bryantbiggs · 2022-11-17T12:32:00Z

examples/karpenter-fargate/README.md

+- AWS EKS Fargate Profiles for the `kube-system` namespace with Pod Labels used by the `coredns`, `karpenter`, and `aws-load-balancer-controller` addons, while additional profiles can be added as needed.
+- AWS EKS managed addons `vpc-cni` and `kube-proxy`.
+- Karpenter Helm Chart.
+- AWS Node Termination Handler Helm Chart. Karpenter does not handle the Spot Interruption Termination Notice (ITN) two-minute warning. In lieu of this, you can install AWS Node Termination Handler to gracefully cordon and drain your spot nodes when they are interrupted. Pods that require checkpointing or other forms of graceful draining, requiring the 2-mins before shutdown, will need NTH.


it does now in v0.19

So should we wait for #1175 and #1145 to be complete an use it in this example?

i expect to have the PR ready in a couple of days, so feel free to go ahead

I see two options while we wait for the blueprint changes for Karpenter 0.19.0.

We specify that prior to v0.19.0, Karpenter requires NTH to handle the Spot ITN

We remove NTH from this example since it's not using Spot for the sample app

Since Karpenter 0.20.0 is now available in the blueprints I removed NTH and configured Karpenter to use a SQS Interruption Queue.

bryantbiggs · 2022-11-17T12:33:55Z

examples/karpenter-fargate/README.md

@@ -0,0 +1,210 @@
+# EKS Cluster with Karpenter Cluster Autoscaler running on Fargate


We should start with https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/main/examples/fargate-serverless as the baseline for this example and update to show the relevant aspects that we are trying to highlight that are different

"Cluster Autoscaler" is not something the official docs mention.
plus is not about the cluster, but about the worker nodes

I reused what's in https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/main/examples/karpenter/README.md?plain=1#L1.

I will update the README.md file following @bryantbiggs recommendation.

I've updated the readme, but I don't think I can reuse much of what's in https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/main/examples/fargate-serverless after all. It is not the same use case since we have EC2 compute that can be provided by Karpenter in this example. In this example, addons are placed on Fargate for another reason. I explained the reasons why I'm putting the addons on Fargate in #1187 (comment).

FernandoMiguel · 2022-11-17T15:05:04Z

examples/karpenter-fargate/main.tf

+  # We filter the kube-system pods with labels since not all add-ons can run on Fargate (e.g. aws-node-termination-handler)
+  fargate_profiles = {
+    # Providing compute for the kube-system namespace where addons that can run on Fargate reside
+    coredns = {


since karpenter does not depend on coredns, it seems best to have it running on karpenter nodes than the lead time to create and maintain more fargate profiles, roles, etc

For this example, I decided to put the coredns and also the aws-load-balancer-controller add-ons on Fargate. Not because Karpenter depends on them, but because I think Fargate might be a better choice to run these add-ons since they don't need specific hardware support or a particular instance type. For these reasons, my guiding principle was to put addons on Fargate unless they were not compatible with it, like NTH.

i just found that creating and destroying multiple fargate profiles to take far too long, since they are done in serial rather than parallel, and karpenter nodes are faster than that.

I created this example while thinking about how AWS customers could use it in the real world. I don't think the time it takes to create and destroy the resources provided by this example should be taken into consideration.

@FernandoMiguel have you tried profile wildcards to work the problem?

If we remove NTH or deploy in another namespace we could use only one Fargate profile for kube-system pods and wildcards would not be required. I created multiple profiles for kube-system since the NTH namespace is hardcoded to kube-system in https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/main/modules/kubernetes-addons/aws-node-termination-handler/locals.tf#L2 and I didn't want to change that in this PR. Since NTH cannot run on Fargate (it needs to access IMDS) and since it will not be required with Karpenter v.0.19.0, we could just remove it from that PR as I proposed in #1187 (comment).

I also think the question we need to answer for this example is do we want to run the add-ons other than Karpenter on Fargate or not. Because other add-ons that run inside the kube-system might not be compatible with Fargate and we would still need a solution to not schedule them on Fargate nodes. Another thing to consider is that Fargate profile wildcards do not offer much flexibility when selecting which pods can be placed on a Fargate profile for one namespace and when they have different labels. I tried it and did not find any good pattern that could allow filtering out NTH pods from the kube-system Fargate profile.

@askulkarni2 those were not a thing when i tried. havent tried since

Since Karpenter 0.20.0 is now available in the blueprints I removed NTH and this example now uses only two Fargate profiles.

…blueprints into karpenter-fargate

askulkarni2 · 2022-12-17T00:23:43Z

examples/karpenter-fargate/provisioners/sample_deployment.yaml

@@ -0,0 +1,71 @@
+apiVersion: apps/v1


Can we use the existing app instead?

I thought about using it at first, but finally decided to use a deployment with nodeSelector, topologySpreadConstraints, and podAntiAffinity to demonstrate how Karpenter handles these types of constraints. I explain it in the README. I understand that this may look like code duplication, so I can replace it with a simpler deployment using pause pods like this inflate example with the same constraints if you prefer.

@askulkarni2 I just realized that I may not have been clear in what I'm trying to achieve with this sample deployment. I want to deploy one application to demonstrate two things:

Deploy workloads with contraints to see how Karpenter will manage them

Deploy a web application accessible through an ALB provisioned by the AWS Load Balancer Controller running on Fargate

I thought that using the same application to demonstrate these two points was a good idea so that's why I didn't use the app-2048 addon.

What are we trying to highlight or demonstrate?

Why is this beneficial to customers?

Is this specific to Karpenter on Fargate?

We are trying to demonstrate how to create a cluster using Fargate profiles in which to run Karpenter and other add-ons. More specifically how to implement the Run the Karpenter controller on EKS Fargate best practice and provide a demo application so customers can test this use case using the EKS blueprints.

This example also uses spot instances, SQS for the interruption queue and an ALB deployed by the AWS Load Balancer Controller (running on Fargate) to access the demo app (running on worker nodes deployed by Karpenter).

The demo app deployment is also using nodeSelector, topologySpreadConstraints, and podAntiAffinity to demonstrate how Karpenter handles these types of constraints.

allamand · 2023-01-06T16:22:43Z

examples/karpenter-fargate/main.tf

+# # Allow ingress from the worker nodes security group (Karpenter nodes)
+# # to the cluster primary security group (Fargate nodes)
+# resource "aws_security_group_rule" "cluster_primary_ingress_all" {
+#   description              = "Allow All ingress from the worker nodes security group"
+#   type                     = "ingress"
+#   to_port                  = 0
+#   protocol                 = "-1"
+#   from_port                = 0
+#   security_group_id        = module.eks_blueprints.cluster_primary_security_group_id
+#   source_security_group_id = module.eks_blueprints.worker_node_security_group_id
+# }


Are thoses still needed ?

No. I removed them.

allamand · 2023-01-06T16:24:30Z

examples/karpenter-fargate/main.tf

+  }
+}
+
+resource "kubectl_manifest" "karpenter_provisioner" {


Warning here, if you update a provisioner in futur terraform apply, it will first delete, then recreate it.
But when it delete it, it will delete all karpenter nodes associated to it..

Yes you're right. The provisioner that was deployed in https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/karpenter was also using this method but it was removed in 8915911.

bryantbiggs · 2023-01-07T02:08:03Z

please see the updated https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/karpenter

…blueprints into karpenter-fargate

micbegin · 2023-01-09T14:20:19Z

please see the updated https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/karpenter

What do you want to with this PR then? My goal was to provide a more complete example to customers who wants to deploy workloads on EKS cluster that are using Karpenter and Fargate. If you think this repo is not the right place to provide that kind of example I can close this PR.

bryantbiggs · 2023-01-21T15:08:26Z

the current example for Karpenter is already configured for running on Fargate. If there are additional features/questions related to this pattern, please feel free to open a new issue describing the solution you would like to see or have answered

Ref: https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/karpenter

micbegin · 2023-01-22T21:41:29Z

I'm not sure why you are closing this PR since https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/karpenter does not provide the following items:

A default Karpenter provisioner using Bottlerocket AMI and a mix of spot and on-demand instances
A demo application to demonstrates how Karpenter scales nodes based on workload constraints like nodeSelector, topologySpreadConstraints, and podAntiAffinity
Other add-ons like aws-load-balancer-controller and external-dns running on fargate or nodes provisionned by Karpenter to demonstrate how a cluster can operate without any managed node groups

Should I assume that you are not interested in providing this type of example to customers within this repo?

bryantbiggs · 2023-01-22T21:49:49Z

I would recommend checking out this draft PR that lays out some upcoming changes to EKS Blueprints and specifically this section

micbegin added 2 commits November 16, 2022 23:50

Add karpenter on fargate example

0f34850

Remove unused local.node_group_name

d1e8275

micbegin had a problem deploying to EKS Blueprints Test November 17, 2022 05:06 Error

bryantbiggs reviewed Nov 17, 2022

View reviewed changes

Add description to cluster_primary_ingress_all security group rule

75aa64f

micbegin had a problem deploying to EKS Blueprints Test November 17, 2022 14:25 Error

FernandoMiguel reviewed Nov 17, 2022

View reviewed changes

Update README

69b4d2f

micbegin had a problem deploying to EKS Blueprints Test November 17, 2022 18:17 Error

micbegin marked this pull request as ready for review November 17, 2022 18:31

micbegin requested review from askulkarni2, csantanapr, vara-bonthu and Zvikan as code owners November 17, 2022 18:31

micbegin marked this pull request as draft November 28, 2022 15:45

Merge branch 'main' of https://github.com/micbegin/terraform-aws-eks-…

d805da1

…blueprints into karpenter-fargate

micbegin had a problem deploying to EKS Blueprints Test December 14, 2022 23:36 — with GitHub Actions Error

Remove node termination handler

1686ff1

micbegin had a problem deploying to EKS Blueprints Test December 15, 2022 01:28 — with GitHub Actions Error

micbegin marked this pull request as ready for review December 15, 2022 01:35

micbegin requested a review from a team as a code owner December 15, 2022 01:35

Remove nth from readme

7931ca5

micbegin temporarily deployed to EKS Blueprints Test December 15, 2022 13:42 — with GitHub Actions Inactive

askulkarni2 reviewed Dec 17, 2022

View reviewed changes

allamand reviewed Jan 6, 2023

View reviewed changes

micbegin added 2 commits January 9, 2023 09:11

Merge branch 'main' of https://github.com/micbegin/terraform-aws-eks-…

c54c376

…blueprints into karpenter-fargate

Remove unneeded security group

782b98d

bryantbiggs closed this Jan 21, 2023

micbegin mentioned this pull request Jan 22, 2023

Example - Karpenter on Fargate #1159

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add karpenter on fargate example #1187

feat: Add karpenter on fargate example #1187

micbegin commented Nov 17, 2022 •

edited

Loading

bryantbiggs Nov 17, 2022

micbegin Nov 17, 2022 •

edited

Loading

FernandoMiguel Nov 17, 2022

micbegin Nov 17, 2022

micbegin Dec 15, 2022

bryantbiggs Nov 17, 2022 •

edited

Loading

FernandoMiguel Nov 17, 2022

micbegin Nov 17, 2022

micbegin Nov 17, 2022

FernandoMiguel Nov 17, 2022

micbegin Nov 17, 2022 •

edited

Loading

FernandoMiguel Nov 17, 2022

micbegin Nov 17, 2022

askulkarni2 Nov 17, 2022

micbegin Nov 17, 2022 •

edited

Loading

FernandoMiguel Nov 18, 2022

micbegin Dec 15, 2022

askulkarni2 Dec 17, 2022

micbegin Dec 18, 2022

micbegin Jan 4, 2023

bryantbiggs Jan 5, 2023

micbegin Jan 6, 2023

allamand Jan 6, 2023

micbegin Jan 9, 2023

allamand Jan 6, 2023

micbegin Jan 9, 2023

bryantbiggs commented Jan 7, 2023

micbegin commented Jan 9, 2023

bryantbiggs commented Jan 21, 2023

micbegin commented Jan 22, 2023

bryantbiggs commented Jan 22, 2023

		@@ -0,0 +1,210 @@
		# EKS Cluster with Karpenter Cluster Autoscaler running on Fargate

feat: Add karpenter on fargate example #1187

feat: Add karpenter on fargate example #1187

Conversation

micbegin commented Nov 17, 2022 • edited Loading

What does this PR do?

Motivation

More

For Moderators

Additional Notes

Choose a reason for hiding this comment

micbegin Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micbegin Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micbegin Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs commented Jan 7, 2023

micbegin commented Jan 9, 2023

bryantbiggs commented Jan 21, 2023

micbegin commented Jan 22, 2023

bryantbiggs commented Jan 22, 2023

micbegin commented Nov 17, 2022 •

edited

Loading

micbegin Nov 17, 2022 •

edited

Loading

bryantbiggs Nov 17, 2022 •

edited

Loading

micbegin Nov 17, 2022 •

edited

Loading

micbegin Nov 17, 2022 •

edited

Loading