Support Weight in localQueue for better resource partitioning

[kerthcet]

What would you like to be added:

Currently, for queues inside one CQ, we can submit queues as much as we want until the bottleneck of CQ, and one queue can occupy the whole resources with higher priorities, considering localQueue represents the tenant, this is not reasonable. We should have hard restrictions on how much resources each tenant can have. Instead of defining the resources requests inconveniently, we can have a weight field.

One think for better resource utilization consideration is when resources are sufficient, how to breakthrough the hard restriction of weight and also preemption when resources are not enough.

Why is this needed:

Multi-tenant requirement and better resource utilization.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[KunWuLuan]

If no one working on this, I will try to create a kep to track this issue and work on this
/assign

[KunWuLuan]

Due to the fact that each cluster queue can borrow resources from other cluster queues, the total amount of resources for each queue is uncertain. So it is not easy to design with weight.

I think we can use 'max and min' or 'nominal and borrowingLimit' to define the limit for local queue.

[alculquicondor]

Ideally we should be using just weight so that new LocalQueues can be easily added without changing the configuration for other queues.

In system that use weights, the calculations of fair sharing are always dynamic and depend on how many queues have running or pending jobs and the total resources from the parent queue.
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
https://karthiksharma1227.medium.com/deep-dive-into-yarn-scheduler-options-cf3f29e1d20d

[kerthcet]

The weight is a relative value, unlike max or min, it's an absolute value. So as Aldo shares, fair sharing is dynamic. Like queueA's weight is a, queueB's weight is b, then queueA can get a/(a+b) of the resources. One concern is if a queue get the resources, then another queue joined with a hight weight, then will the original queue be preempted?

[KunWuLuan]

If we just use weight, how this weight work with borrow? Should we use nominal+borrowing limit as total resources? Or we can just calculate the proportion of each queue's admitted workloads and only pop workloads from those queues whose proportion is under theirs weights.
If this is not enough, we can add min and max to guarantee min resources for each queue, this can be another question.

We can achieve the goals by two steps.
In first step, we add an interface in kueue scheduler to determine whether a queue has used too much resources and reject the next workload if yes. In this way we can make each queue get their desired resources when queues don't change.
In second step, we add a controller in kueue to preempt workload if a queue used too much resources. In this way we can make each queue get their desired resources when queues change.

[alculquicondor]

Or we can just calculate the proportion of each queue's admitted workloads and only pop workloads from those queues whose proportion is under theirs weights.

This sounds like a better idea, and closer to what I've seen other systems do. Another thing to consider is that if a queue is 100% empty and has no running workloads, there is no need to consider it in the calculations.

One concern is if a queue get the resources, then another queue joined with a high weight, then will the original queue be preempted?

That is a very important question. Maybe it should be configurable whether or not to preempt.

I think the best step at this point is to do a review of existing systems. Then we can start collaborating in a google doc or KEP PR.

[alculquicondor]

cc @mwielgus

[KunWuLuan]

@alculquicondor Hi, I have submit a new kep for this, we can discuss details there.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/remove-lifecycle stale

[alculquicondor]

/close
in favor of #1714

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close
in favor of #1714

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.