[aws-eks] wrong order of cluster.awsAuth.addMastersRole & cluster.addNodegroup breaks awsAuth

[arjanschaaf]

If you execute the cluster.awsAuth.addMastersRole & cluster.addNodegroup calls in the wrong order you end up with an awsAuth configmap in your cluster that misses the rolemapping for the node group role. The result of this is that the nodes in the nodegroup can't connect to the apiserver and the cluster is broken.

Reproduction Steps

1. create an EKS cluster

    const cluster = new eks.Cluster(this, 'eks-cloud-infra', {
      clusterName: 'eks-cloud-infra',
      defaultCapacity: 0,
      version: '1.15',
      vpc: vpc,
    });

2. add a nodegroup to the new cluster

    cluster.addNodegroup('node-group-cloud-infra', {
      instanceType: new ec2.InstanceType('t3.large'),
      minSize: 1,
      maxSize: 10
    });

3. add additional master roles (we do this because we want to give 2 groups of users control of the EKS cluster that authenticate using an IAM Role created by Control Tower

    cluster.awsAuth.addMastersRole(iam.Role.fromRoleArn(this, 'administrators', 'arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_234d7d11d88198f7'));
    cluster.awsAuth.addMastersRole(iam.Role.fromRoleArn(this, 'powerusers', 'arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_432fa27f4402ce4e'));

Error Log

expected result from kubectl -n kube-system get configmap aws-auth -o yaml

apiVersion: v1
data:
  mapAccounts: '[]'
  mapRoles: |
    - groups:
      - system:masters
      rolearn: arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_437d7d11d88198f7
      username: arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_437d7d11d88198f7
    - groups:
      - system:masters
      rolearn: arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_ebffa27f4402ce4e
      username: arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_ebffa27f4402ce4e
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::12345:role/CloudInfrastructureStack-ekscloudinfraNodegroupnod-GKQNHZRL0EM6
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: '[]'
kind: ConfigMap

actual result from kubectl -n kube-system get configmap aws-auth -o yaml

  apiVersion: v1
  data:
    mapAccounts: '[]'
    mapRoles: '[{"rolearn":"arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_437d7d11d88198f7","username":"arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_437d7d11d88198f7","groups":["system:masters"]},{"rolearn":"arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_ebffa27f4402ce4e","username":"arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_ebffa27f4402ce4e","groups":["system:masters"]}]'
    mapUsers: '[]'
  kind: ConfigMap

Please note that the formatting looks strange, but more importantly the system:bootstrappers & system:nodes groups are missing for the nodegroup role

Environment

• CLI Version : aws-cli/2.0.5
• Framework Version: 1.34.1 (build 7b21aa0)
• OS : OSX
• Language : Typescript

Other

Changing the order of executing the cluster.awsAuth.addMastersRole & cluster.addNodegroup methods leads to the desired result:

    cluster.awsAuth.addMastersRole(iam.Role.fromRoleArn(this, 'administrators', 'arn:aws:iam::12345:role/AWSReservedSSO_AWSAdministratorAccess_234d7d11d88198f7'));
    cluster.awsAuth.addMastersRole(iam.Role.fromRoleArn(this, 'powerusers', 'arn:aws:iam::12345:role/AWSReservedSSO_PowerUserAccess_432fa27f4402ce4e'));

    cluster.addNodegroup('node-group-cloud-infra', {
      instanceType: new ec2.InstanceType('t3.large'),
      minSize: 1,
      maxSize: 10
    });

---

This is 🐛 Bug Report

[eladb]

Thanks for reporting!

[pahud]

@eladb

I believe this is because we don't do the AwsAuth for managed ng and let the eks control plane do it instead. So the state will be inconsistent.

aws-cdk/packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts

Lines 266 to 267 in a732d14

	// As managed nodegroup will auto map the instance role to RBAC behind the scene and users don't have to manually
	// do it anymore. We don't need to print out the instance role arn now.

Maybe we still should do it for manage nodegroups? Let me try add this logic?

[eladb]

@eladb

I believe this is because we don't do the AwsAuth for managed ng and let the eks control plane do it instead. So the state will be inconsistent.

aws-cdk/packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts

Lines 266 to 267 in a732d14

	// As managed nodegroup will auto map the instance role to RBAC behind the scene and users don't have to manually
	// do it anymore. We don't need to print out the instance role arn now.

Maybe we still should do it for manage nodegroups? Let me try add this logic?

I see. Yes. I think we must also do it for Managed Node Groups or otherwise we will lose track of the state.

[pahud]

@eladb OK. Let me create the PR.