Cannot scale up nodegroup after adding an ASG tag node-template label

[andrewwdye]

Which component are you using?:

cluster-autoscaler

What version of the component are you using?:

1.24.0 and 1.26.2

Component version:

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.22.17-eks-48e63af
WARNING: version difference between client (1.25) and server (1.22) exceeds the supported minor version skew of +/-1

What environment is this in?:

Using EKS

What did you expect to happen?:

I'm using ASG tags for node-template labels to support scaling nodegroups from 0 as described here. Specifically k8s.io/cluster-autoscaler/node-template/label/<label-name>: <label-value>.

When using a NodeSelector in my pod spec, I expect the nodegroup to scale up from 0.

What happened instead?:

Nodegroup did not scale up with error

...node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=

How to reproduce it (as minimally and precisely as possible):

I don't have a great repro at this point, but I suspect it's along the lines of

1. Run autoscaler
2. Allow target nodegroup YYY to scale down to 0
3. Add ASG tag w/ label L to YYY
4. Attempt to run a pod with node selector that matches label L

Anything else we need to know?:

Restarting autoscaler resolves the issue and allows the nodegroup to scale up from 0. Based on the discussion here, I suspect the nodegroup info is getting cached and autoscaler isn't able to discover the newly applied labels.

[andrewwdye]

Kudos to @jeevb for tracking this down to the node-info-cache-expire-time flag, which defaults to 10 years. Without changing this, cluster-autoscaler effectively ignores tag changes for any nodegroups that have scaled to 0.

I don't see node-info-cache-expire-time mentioned in the cluster-autoscaler on AWS README or really anywhere searching around.

Looks like this flag was introduced in #4669 and defaulted to 10 years for back compat reasons. Given the implications of not setting it, I'm curious what the risks are of defaulting to a more sane (but still performant) value of ~24h.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.