[EKS] [request]: Managed Node Groups - Add ability to set the underlying ASG name

[shelby-moore]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Right now, when using Managed Node Groups, the underlying ASG name is generated by AWS and set to something like eks-40bbb26b-8679-eb64-d33a-4ba475413529. I would like the ability to set the name of the underlying ASG, eg cluster-1-us-east-1a, or set a prefix for AWS to use in the name of the underlying ASG, eg cluster-1-us-east-1a-AWS_GENERATED_HASH_HERE, or have the ASG name incorporate the node group name. Basically, I want the underlying ASG to have a meaningful name.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I would like to use multiple node groups in my cluster, some of which will use SPOT instances, others will be ON_DEMAND. I want to set up the cluster autoscaler to prefer scaling out the node groups with SPOT instances when possible. To accomplish this, you configure the cluster autoscaler to use a priority expander. The expander is configured with weights for node groups matching certain regexes, for example:

apiVersion: v1
kind: ConfigMap
data:
  priorities: |-
    20:
      - .*spot.*
    10:
      - .*on-demand.*

The regexes here match on the node group name - this needs to match the AWS ASG name. Because I have no control over the name of the underlying ASGs created by Managed Node Groups, I am unable to use the priority expander. I've looked through the source code for the cluster autoscaler, and I don't think it's appropriate to add logic in there to try to map Managed Node Group names to the underlying ASGs in a way that could be used in the priority expander. In fact, it is a design decision of the cluster autoscaler that the ASGs should have meaningful names. From the docs on the priority expander:

Priority based expander selects an expansion option based on priorities assigned by a user to scaling groups. The assignment is based on matching of the scaling group's name to regular expressions. The correct and meaningful naming of scaling groups is left up to the user.

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md

Are you currently working around this issue?
I am not using Managed Node Groups that use SPOT instances, as it is not possible to use the Cluster Autoscaler with the priority expander.

[cablespaghetti]

I've built a tool to work around this issue: https://github.com/cablespaghetti/priority-expander-eks-managed-nodegroup-configurer

It's very much an MVP at the moment and could use some more testing/error handling. However it works!

[mikestef9]

Would it help if we just added something like "spot" or "on-demand" to the autoscaling group name? Depending on the type of node group

[cablespaghetti]

Yes that would help. In my case I would like to have spot-amd64 spot-arm64 for example as well. Basically if we can add a prefix or suffix to the ASG name that would solve the problem. 🙂

edit: Having an intermittent issue. See: kubernetes/autoscaler#3956

[abierbaum]

I just ran into this use AWS CDK to create a cluster with two managed nodegroups. Having some way for the nodegroup name to propogate into the ASG name would be very helpful. I can't setup the autoscaler to prioritize nodegroups without it.

[mikestef9]

One more suggestion, what if we simply put the node group name as a prefix to the ASG name?

[elementalvoid]

One more suggestion, what if we simply put the node group name as a prefix to the ASG name?

This would certainly solve the problem on my side.

It would also be a really nice convenience feature. With Managed NodeGroups the NG Names no longer match the ASG name so it's a minor pain to lookup the AWS resources in observability tools (AWS Console, CloudWatch, Datadog).

Something to keep in mind ASG name character limits (max 255 chars). My team uses rather expressive NodeGroup names -- our biggest is 44 chars so not anywhere near the limit, but it is something to keep in mind.

---

Edit:

Would it help if we just added something like "spot" or "on-demand" to the autoscaling group name?

This too would help but I think it would provide less overall benefit than prefixing with the NG name.

As mentioned above, my team uses expressive NG Names (eg: worker-managed-spot-16vcpu-32gb-ib2). We do this because we have a large number of NGs to spread across AZs as well as for dedicated purposes -- in one cluster alone we have 14 NGs! Being able to easily grok the NG's purpose by it's name is super helpful. Thus, including only the spot vs. on-demand type is not as helpful; though it would solve the expander priority problem.

[mikestef9]

We are working on this. Plan is once EKS launches v1.21 support and you upgrade your cluster to v1.21, newly created node groups will have the backing ASG name prefixed with the node group name. We will not be changing the name of ASGs backing existing node groups.

So in the example above, newly created node group with name worker-managed-spot-16vcpu-32gb-ib2 would get ASG name worker-managed-spot-16vcpu-32gb-ib2-eks-{UUID}

[jaimehrubiks]

Why will this only work on kube 1.21, is it relared to any new feature? Will there not be other manual workaround for lower versions?

[trobert2]

hello @mikestef9,
Will the ASG name be retrievable when defining a new nodegroup?
If I want to create a new lifecyclehook the cdk construct requires the ASG to be passed in.

[mikestef9]

The ASG name is already returned as part of the DescribeNodegroup API. Is that sufficient?

[trobert2]

hey @mikestef9,
thanks for the quick reply.

That would be enough, but it's not available in CDK.
given this nodegroup definition:

import * as eks from '@aws-cdk/aws-eks'
...
    const nodeGroup = new eks.Nodegroup(this, props.nodeGroupName, {
      cluster: cluster,
      nodegroupName: props.nodeGroupName,

      minSize: 1,
      launchTemplateSpec: {
        id: lt.ref,
        version: lt.attrLatestVersionNumber
      }
    })

I don't think the ASG details are available. I know by default there are a couple of lifecycle hooks being defined as well. Those could be helpful, but I can't seem to find a way to retrieve them.

Given that the ASG would end with {UUID} it would not be possible to use autoscaling.AutoScalingGroup.fromAutoScalingGroupName. In order to import the ASG that way, the name would need to be deterministic.

EDIT (Solved):
For my usecase I was able to fetch the required information using the API call mentioned by @mikestef9.
I did this by making use of a Custom Resource (@aws-cdk/custom-resources).
Code:

    const nodeGroupASG = new cr.AwsCustomResource(this, 'nodeGroupASG', {
      onCreate: {
        service: 'EKS',
        action: 'describeNodegroup',
        parameters: {
          clusterName: cluster.clusterName,
          nodegroupName: props.nodeGroupName
        },
        physicalResourceId: cr.PhysicalResourceId.of(
          `${props.nodeGroupName}-asg-name`
        )
      },
      onUpdate: {
        service: 'EKS',
        action: 'describeNodegroup',
        parameters: {
          clusterName: cluster.clusterName,
          nodegroupName: props.nodeGroupName
        },
        physicalResourceId: cr.PhysicalResourceId.of(
          `${props.nodeGroupName}-asg-name`
        )
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
        resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE
      })
    })

where props are the values passed in to my construct and cluster is my EKS cluster object I have already declared.
NOTE: if you declare a nodegroup and give it a name, the nodegroupName field on the created object will not reflect the correct value you have passed in. The api call will not work using this. For this reason I am using the props directly.

[stevehipwell]

Why will this only work on kube 1.21, is it relared to any new feature? Will there not be other manual workaround for lower versions?

@mikestef9 is there a reason for this? I get why existing node groups can't be upgraded, but backporting this functionality would be very useful.

[mikestef9]

While not technically part of the MNG API contract, we are erring on the side of caution in case customers have written scripts that rely on the existing format of the autoscaling group name.

[stevehipwell]

Could it be enabled for existing versions behind a flag? Otherwise MNGs are pretty much off the table for older EKS versions.

[mikestef9]

This is now available with the release of EKS v1.21, see release blog for more details.