[EKS]: Managed Cluster Add-ons

[tabern]

Core and optional addons are managed by Amazon EKS. Specify which addons you would like to run when provisioning the cluster and Amazon EKS will update addons to their latest versions in context of the recommended addon version for the Kubernetes version of the cluster.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Today, certain core Kubernetes addons (DNS, CNI, etc) are automatically provisioned by Amazon EKS during cluster provisioning but must be manually updated to new versions as they are available and when updating the Kubernetes version of the cluster. EKS managed cluster addons would give additional control over what addons start on the cluster and provide automation around keeping addons up to date.

Are you currently working around this issue?
Mannualy start additional addons for your cluster and manually update (through kubectl apply) addons to the latest versions when required for new functionality or new Kubernetes versions.

Additional context
This helps to avoid issues such as #24 (comment)

Helps to solve:

• [EKS] Install AWS Controllers for Kubernetes (ACK) on control plane #47
• [EKS] [request]: Independent control plane and cluster addons management #217
• [EKS] EKS add-ons support for Amazon EBS CSI Driver #247

As always, feedback welcome.

[errordeveloper]

There certain benefits to current approach, i.e. user can easily customise some of the add-ons.

Aside from that, I wanted to point out that there is a new sig-cluster-lifecycle working group around this: https://github.com/kubernetes-sigs/addon-operators.

[ld-singh]

Does this also means choosing CNI of choice as add-ons?

[ld-singh]

@tabern Could you please confirm if this includes ability to choose any CNI as well like weave, calico etc?

[tabern]

@ld-singh this means that the user would be able to choose from a set of add ons that EKS automatically starts on the cluster or de-select so they have a 'blank' cluster as desired. Per #398 we are not currently planning to include other CNI plugins.

[tabern]

Starting today, you can use EKS to manage the add-ons for your cluster, starting with the Amazon VPC CNI plugin.

You can:

• choose the version during cluster setup and updathttps://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#update-cluster-add-onse at any time
• use the EKS apis to assign a IRSA role to the add-on

EKS add-ons is available for EKS clusters running Kubernetes version 1.18 eks.3 and higher.

You can learn more here:

• Announcement
• Blog - Introducing EKS Add-Ons

We'll be closing this issue and cross-linking to issues for support of individual add-ons. If there's a particular add-on you would like us to support, please open an issue!

[jwenz723]

Starting today, you can use EKS to manage the add-ons for your cluster, starting with the Amazon VPC CNI plugin.

You can:

• choose the version during cluster setup and updathttps://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#update-cluster-add-onse at any time
• use the EKS apis to assign a IRSA role to the add-on

EKS add-ons is available for EKS clusters running Kubernetes version 1.18 eks.3 and higher.

You can learn more here:

• Announcement
• Blog - Introducing EKS Add-Ons

We'll be closing this issue and cross-linking to issues for support of individual add-ons. If there's a particular add-on you would like us to support, please open an issue!

@tabern Please add support for kube-proxy and coreDNS. I'm assuming this is planned since these 2 add-ons are the only ones remaining out of the add-ons that ship out of the box with a new EKS cluster, but I don't see any issues currently open to track these.

[tabern]

@jwenz723 yes! these are at the top of our list. I've created #1159 to track this.

[errordeveloper]

It's not clear from the blog post if one can opt-out from installing the VPC CNI add-on, or perhaps if there maybe alternative CNI add-ons available in the future. Would you mind to clarify the direction. The term "add-ons" usually refers to something that is either optional, removable or swappable.

[tabern]

@errordeveloper good question. Until now, EKS has added in these add-ons (vpc-cni, kube-proxy, coredns) by default with every cluster. That has not changed.

That said, it is in our plans to allow opt-out from all add-ons (including these default add-ons) during cluster creation. Our goal is to make EKS a bit more modular with this release, giving you options to add more things to the cluster easily, but also not have things you may not want.

[errordeveloper]

@tabern thanks for clarifying regarding the roadmap for this. Does the way the new feature works allows user to delete e.g. VPC CNI daemonset, and it won't be reinstated? What happens in the UI, will users get warnings saying "your cluster is missing a critical addon, you must click here to re-instate it"?
My main concern is wether Cilium installation instructions need some adjustments or not, we currently recommend replacing VPC CNI by deleting the daemonset, and installing Cilium instead. I hope this use-cases is still supported, but just wanted to double-check with you.

[foriequal0]

I tried to add vpc-cni plugin to a existing cluster, and deleted it.
The console says: "The cluster will no longer have this add-on's features. You can re-add the add-on later if needed.", but I couldn't re-instate it with the console. I've fixed it by applying this config https://github.com/aws/amazon-vpc-cni-k8s/blob/master/config/v1.7/aws-k8s-cni.yaml with kubectl

[3oris]

@errordeveloper Currently different things happen when you create a new cluster via web console or via CloudFormation/CDK:

• In the web console you are required to select the add-on and add-on version. This will launch the cluster with the managed version of the add-on. You can then delete it via web console or API and replace it with your own.
• When you create a cluster via CloudFormation or CDK, it will come with the unmanaged, still current version of the add-on like before. You can then patch it to make it helm-manageable or remove/replace it like before.

So, currently, this behavior is free of interference for existing clusters and clusters built with IaC tools, and it should leave a migration path from un-managed to managed versions of the add-ons once things have settled. At least it should't break things with existing tooling right now.

[errordeveloper]

@3oris thanks a lot, this very helpful!

[3oris]

@foriequal0 have you tried re-installing with force override? See https://d2908q01vomqb2.cloudfront.net/fe2ef495a1152561572949784c16bf23abb28057/2020/12/01/3.-update-version.png

[foriequal0]

@3oris IIRC, it didn't work for me.