Ensure pods fit on mixed instance pools

[nielsole]

We use launch templates with mixed instance types with varying CPU and memory for spot instance pools in AWS.

This poses some challenges for the estimation of available CPU and memory on an ASG.
In our use-case we would like to use the smallest resources of each type across all instances (not the smallest instance, but that smallest value for each resource type).
This ensures that that if the autoscaler scales up an ASG, the pods are guaranteed to actually fit on it.

IIUC as soon as an instance was created, that instance is used for estimation of the size that can be expected of that ASG. But this assumption is not necessarily true.
Let's assume m5.xlarge and m5.2xlarge are in the same ASG and the autoscaler randomly brings up a m5.2xlarge it will use this for estimation. But the next instance might be a m5.xlarge. If the autoscaler chose this ASG for scale-up because it would fit an especially large pod, scheduling it would fail.

We would like to fix this. I am wondering whether this minimum resources across the ASG is a good default for everyone. If it is not, I would like to be able to override the CPU and memory available in an ASG using manually set tags on the ASG.

Manual tagging has already been pondered in #1650 (specifically #1656 (comment)) for a different use-case and could be extended for CPU and memory. One possible blocker for this that IIRC the autoscaler wouldn't use the buildNodeFromTemplate if there are existing nodes. It would then use the existing nodes as a template (with all the problems described above).

Which of the two approaches would you favor? Or do you see a better solution alltogether?

[Jeffwan]

/sig aws

[Jeffwan]

Still working on a doc in internal review.. This takes some time. I checked several limitation with our EC2 teams and seems no perfect solution right now. We can make it work for some cases but won't cover all use case. Currently, the best practice here is to use similar node with same resources as much as possible.

The assumption here is container can alway run on the template node. Otherwise, there's always possibility pod can not schedule on the node since ASG is backed by several instance type. Then let's consider how to build this template.

1. If we have m5.xlarge (4CPU, 16Memory) and m5.2xlarge (8CPU, 32Memory) in the same ASG Mixed Instance Pool, we should use Min (instances.CPU) and Min(instances.Memory) which is (4GPU, 16 Memory) to do estimation. No matter scale up from existing node or scale up from 0 (template). I have a PR ongoing to honor override LaunchTemplate values.

2. Manual tagging brings some management efforts. I am not sure if users like this way. Using m5.xlarge and m5.2xlarge, what's the right CPU/Memory you want to use? For users understand their workloads, they may come with some good combination with high cost efficiency. There're also lots of users don't know what the limit range of their workloads. (like cluster admin). That would be hard to give right value. If there's one perfect formula or algorithm, we can also make it work in Fix imports in cluster autoscaler after migrating it from contrib #1.

[nielsole]

1. would currently be our preferred approach. Do you have a link to your PR?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.