Reclaim consumed resources once quit from cohort

[kerthcet]

What happened:

When clusterQueue quits from the cohort, the resources consumed will not be reclaimed.

1. rayjob-high and rayjob-low both submitted to the cluster-queue-1
2. cluster-queue-1 and cluster-queue-2 belong to the same cohort cohort-pool at first
3. resources(rayjob-high) + resources(rayjob-low) > capacity(cluster-queue-1)
4. then remove the cohort from cluster-queue-1
5. rayjob-high and rayjob-low will keep running

What you expected to happen:

high priority rayjob will preempt the lower one.

How to reproduce it (as minimally and precisely as possible):

The process described above.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always): main
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[alculquicondor]

This is somewhat analogous to scheduler vs descheduler. We don't have a process that checks that the decisions taken before still hold.
So far, the effect is that no new workloads would be admitted in this scenario.
I would label this a feature request instead.

[kerthcet]

Agree, more like a feature.
/remove-kind bug
/kind feature

[tenzen-y]

I think that this feature request isn't only for RayJob.

/retitle Reclaim consumed resources once quit from cohort

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/lifecycle frozen
Wait for more feedbacks.