(aws-eks): Make sure kubectl is compatible with the k8s version

[otaviomacedo]

Currently, to update Kubernetes resources, we use kubectl version 1.20, hardcoded in a lambda layer. For now, this works fine, but eventually, as new Kubernetes versions are released, they won't be compatible with the client anymore. Also, it's not an option to always use the latest version of the client, as we know that kubectl version 1.21.0 is not compatible with version 1.20 (and lower) of the server.

[robertd]

Hi @otaviomacedo,

Based on my Slack conversation w/ @rix0rrr on #20000 here:

This commit reverts addition of the latest version of EKS (1.22) to CDK as there are some incompatibilities between the current lambda-layer-kubectl and v1.22 EKS cluster. Preferably we would have multiple versions fo kubectl lambda layer, but we are currently struggling with package size issues. kubectl is large, and our package is larger, and adding multiple kubectls to it will make it exceed size limits of package managers. So we can't do this right now, until we have invested significant engineering effort into lifting those limitations. In order to not break the backward compatibility with <=1.20 versions, it was decided to revert this addition PR until all the underlying issues around package size issues are resolved.

EKS v1.22 just got released and I bet v1.23 (and possibly v1.24) is around the corner, and CDK users will be looking into migrating over very soon (myself included 😅).

Would it be feasible for CDK to create an official kubectl lambda layer(s) and deploy them in each region (or reference the official ones if they exist), and then reference them based on the EKS cluster version selection. Just brainstorming here... Would this be an overkill? Would this even fall under CDK to manage, or standalone project? Is this too much to ask for from CDK itself? Or CDK should not go in that direction? ... etc

LMKWYT

[robertd]

cc-ing @pahud to see if he has any thoughts on this. Thanks!

[scottbisker]

Is there a status update on this? Any reason why there can't be a different layer version per EKS version? AWS only supports a given EKS version for 16 months. 1.20 is end of support in approximately 3 months. Due to the rolling nature of EKS, CDK needs to support matching versions for kubectl. Once 1.20 is EOS, the control plane will automatically migrate to the next supported version breaking the use of kubectl either way.

[pahud]

I think we probably have to dynamically create the kubectl-lambda-layer with exactly the same k8s version of the cluster. In this case:

1. we need expose the k8s version of the cluster
2. we need allow the lambda-layer-kubectl construct to accept the version as an optional property and dynamically build the layer accordingly.

And according to this versions mapping, if we select cluster version 1.22, the kubectl lambda layer of 1.22.6 should be provisioned under the covers.

That being said, we probably should extend the lambda-layer-kubectl construct first.

any thoughts?

[wtibbitts]

Really seems crazy there's nothing on the roadmap for this yet.

[zhelyan]

I think we probably have to dynamically create the kubectl-lambda-layer with exactly the same k8s version of the cluster. In this case:

1. we need expose the k8s version of the cluster
2. we need allow the lambda-layer-kubectl construct to accept the version as an optional property and dynamically build the layer accordingly.

And according to this versions mapping, if we select cluster version 1.22, the kubectl lambda layer of 1.22.6 should be provisioned under the covers.

That being said, we probably should extend the lambda-layer-kubectl construct first.

any thoughts?

Def 2 first to unblock people and give yourselves some breathing space to rethink 1

[michelesr]

Isn't possible to retrieve the version of the passed cluster using AWS API and use the appropriate kubectl version for the cluster?

[Saberos]

If/when this is picked up is there any chance this will also be implemented in CDK v1?

[shcherbak]

release date of eks 1.22 is April 4, 2022, 1.23 is planned for August, but we still can not upgrade using cdk
eks 1.22 is long awaited release for which people were impatiently waiting for

[robertd]

We've been brainstorming a possible solution in #20596, but no definitive PR yet.

[memark]

With Kubernetes 1.24 recently released, 1.21 is no longer a supported Kubernetes release!

https://kubernetes.io/releases/patch-releases/#non-active-branch-history

This has now turned into a serious (security) problem for anyone wanting to provision production grade clusters via CDK.

[otterley]

@memark Amazon EKS 1.21 is supported through February 2023 per the official documentation.

[memark]

@otterley Thanks, that's good information! I was not aware that AWS backported security patches. Then we're safe until February.

[shcherbak]

AWS EKS 1.23 is available, but we can not upgrade even to 1.22 using CDK :(

[Obirah]

AWS EKS 1.23 is available, but we can not upgrade even 1.21 using CDK :(

I was so sick and tired of this problem, that I adapted this workaround for our environment and have been running and maintaining several 1.22 Clusters with this approach. So far, I can recommend it, it's not a lot of overhead imho.

Will try 1.23 soon.

[tutunak]

Have solved this problem in the best way. Imported EKS to Terraform.

[pahud]

Now that kubectlLayer is supported, I believe this issue should be closed.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.