Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating nodegroup min/max size without desired capacity resets desired capacity to CF value #3532

Closed
msven opened this issue Apr 6, 2021 · 5 comments
Assignees
Labels

Comments

@msven
Copy link

msven commented Apr 6, 2021

What were you trying to accomplish?
I was testing the newly merged functionality provided in #3511 which was trying to fix #2332

Specifically I was trying out

eksctl scale nodegroup --nodes-max 5 --cluster myCluster --name myNg

What happened?
My nodegroups had previously scaled up nodes via the cluster autoscaler, so the current desired capacity differed from what was set in the cloudformation template. However, the desired capacity was modified back to what was set in the cloudformation template.

I would expect the desired capacity to not change.

How to reproduce it?
Manually modify the desired capacity of the autoscaling group to differ from what was set in cloudformation (this would represent some sort of cluster autoscaler adding / removing nodes).

Alter the min or max setting without giving the desired capacity, e.g.

eksctl scale nodegroup --nodes-max 5 --cluster myCluster --name myNg

Logs

Anything else we need to know?
Tagging @aclevername since he was working on this feature

I was using a local build of eksctl. The version is shown in the versions section.

Versions

./eksctl version
0.45.0-dev+b4e84cca.2021-04-06T14:17:16Z

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
@msven msven added the kind/bug label Apr 6, 2021
@aclevername aclevername self-assigned this Apr 6, 2021
@aclevername
Copy link
Contributor

Hi @msven thanks for opening the issue.

What were you trying to accomplish?
I was testing the newly merged functionality provided in #3511 which was trying to fix #2332

Specifically I was trying out

eksctl scale nodegroup --nodes-max 5 --cluster myCluster --name myNg

What happened?
My nodegroups had previously scaled up nodes via the cluster autoscaler, so the current desired capacity differed from what was set in the cloudformation template. However, the desired capacity was modified back to what was set in the cloudformation template.

I would expect the desired capacity to not change.

Mhmm interesting. The eksctl scale command was always just updating the cloudformation stack, the PR mentioned just reduces the number of mandatory fields to be passed in,min/max/desired default to the existing fields in the stack when the value is not provided. This sounds like a feature request to have eksctl scale automatically detect the current amount in the ASG and set the desired to be equal to it when scaling (if desired it not provided on the cli), if I'm understanding you correctly?

@msven
Copy link
Author

msven commented Apr 6, 2021

The issue it closed had the following use case described:

Why do you want this feature?

I use the Cluster Autoscaler to manage the number of nodes in my nodegroups. However, I want to be able to change the limits (min and max) without changing the number of currently active nodes.

It sounds like to achieve that, eksctl scale as you say may need to detect the current capacity and update the cloudformation stack with that value in order to avoid cloudformation from rescaling the nodegroup to match the old desired capacity.

It could be the PR mentioned it only changed the number of mandatory fields, but if that was the only intention, I don't think it should've closed issue #2332 . By closing that issue, I assumed the use case described there was fixed and that feature added, which was why I classified this as a bug.

@aclevername
Copy link
Contributor

The issue it closed had the following use case described:

Why do you want this feature?
I use the Cluster Autoscaler to manage the number of nodes in my nodegroups. However, I want to be able to change the limits (min and max) without changing the number of currently active nodes.

It sounds like to achieve that, eksctl scale as you say may need to detect the current capacity and update the cloudformation stack with that value in order to avoid cloudformation from rescaling the nodegroup to match the old desired capacity.

It could be the PR mentioned it only changed the number of mandatory fields, but if that was the only intention, I don't think it should've closed issue #2332 . By closing that issue, I assumed the use case described there was fixed and that feature added, which was why I classified this as a bug.

Ah your absolutely right, I will re-open the issue and address. Thanks!

@msven
Copy link
Author

msven commented Apr 6, 2021

With #2332 being re-opened I am closing this issue.

Thanks!

@aclevername
Copy link
Contributor

fix should be out in todays 0.45.0-rc.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants