Rewrite EKS L2 construct (EKSv2)

[evgenyka]

Description

When an EKS cluster is created, the only role that has access to the cluster itself (e.g running kubectl commands) is the role that created the cluster. When using CloudFormation, this would be the CloudFormation execution role. Since this role isn’t assumable by anyone, it effectively means it is impossible to connect to the cluster post creation.
In CDK, we workaround this issue by implementing the L2 using custom resources, instead of L1s. This allows us to create the role that creates the cluster (i.e invokes eks.CreateCluster API), and subsequently use this role to grant additional (user defined) roles permissions on the cluster.
The EKS team added a new feature that allows more control over cluster access. Now it would be possible for CloudFormation to specify a list of roles to be granted access to the cluster, in addition to the role that creates the cluster.
The RFC is to create a new EKS L2 construct and drop the custom resource implementation in favor of the native L1.

This is going to incur a breaking change that will require cluster replacement (because type will change from Custom::AWSCDK-EKS-Cluster to AWS::EKS::Cluster). Given a breaking change is inevitable, we can decide also to make some additional breaking changes in the API that make it more ergonomic and aligned with the new cluster implementation.

Roles

Role	User
Proposed by	@evgenyka
Author(s)	@xazhao
API Bar Raiser	@iliapolo
Stakeholders	@alias, @alias, @alias

See RFC Process for details

Workflow

• Tracking issue created (label: status/proposed)
• API bar raiser assigned (ping us at #aws-cdk-rfcs if needed)
• Kick off meeting
• RFC pull request submitted (label: status/review)
• Community reach out (via Slack and/or Twitter)
• API signed-off (label status/api-approved applied to pull request)
• Final comments period (label: status/final-comments-period)
• Approved and merged (label: status/approved)
• Execution plan submitted (label: status/planning)
• Plan approved and merged (label: status/implementing)
• Implementation complete (label: status/done)

---

Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.

[diranged]

Yes please. Also given how many people have created their clusters via the custom resource, please think about how to introduce an import-system so that we can transition from the old model to the new model seamlessly (I know that's not trivial, but it should be possible).

[kromanow94]

Hey, this would be really great.

Also, would this other approach drop the need to create two Nested Stacks when creating the EKS Cluster and one Nested Stack when importing to other Stack?

[xazhao]

Hi @kromanow94 yes nested stack will be removed in the new EKS L2

[dsilbergleithcu-godaddy]

I know that in Dynamo we have Table and TableV2. It might be a wise idea to approach this problem in a similar manner. Have eks.Cluster and eks.ClusterV2. That way we don't have to worry about rolling out the new features and magically breaking anybody who has an EKS cluster. Cluster could then be deprecated and marked as such, and eventually obsoleted. Support could be defined as ending at a particular point in time related to deprecation.

This also facilitates API updates as we really would be independent of the existing "v1" api.

[andreprawira]

Any chance we can implement eksctl since "eksctl is now fully maintained by AWS." in the L2 construct rewrite? It would even be better if we can implement this EKS blueprint too. Support in Python would be appreciated

Lastly, my coworkers and i notice that some objects creation (Deployment for example) might get stuck (the CFN stack that gets stuck) for hours and requires manual intervention by entering the cluster and modifying the yaml manifests, its very time consuming and painful. The result of this have us migrate to github actions (not exactly what i want as i want to keep using CDK, but its a fix)