Managing raw pods

[ahg-g]

What would you like to be added:
Management of raw pods.

kube-scheduler added support for a feature named scheduling gates, which allows an external controller to control when kube-scheduler should consider the pod for scheduling, so in a sense providing pod-level pause capability. This can be used by Kueue to manage raw pods.

Why is this needed:
Some workloads are better managed at the pod level, like Spark. This will also make it easier to support any job type without explicit integration.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

/assign

I want to take a stab at the design first.

[alculquicondor]

Also related to #77

[tenzen-y]

@alculquicondor @achernevskii Do we try to implement all features designed in KEP by release v0.5?

[alculquicondor]

No, we'll just go with single pod support in 0.5 and follow up on 0.6 with the rest.

[tenzen-y]

No, we'll just go with single pod support in 0.5 and follow up on 0.6 with the rest.

SGTM
I was concerned that this feature blocks the v0.5 release.

[kerthcet]

I guess this is completed, right?

[tenzen-y]

I guess this is completed, right?

This is not completed. The kueue doesn't support https://github.com/kubernetes-sigs/kueue/blob/main/keps/976-plain-pods/README.md#groups-of-pods-created-beforehand yet.

[achernevskii]

/assign

[kerthcet]

One potential issue here is pod is both watched by kueue and scheduler, and scheduler may complete the scheduling cycle in prior than kueue, then patching gates to pod will be failed.

[tenzen-y]

One potential issue here is pod is both watched by kueue and scheduler, and scheduler may complete the scheduling cycle in prior than kueue, then patching gates to pod will be failed.

@kerthcet IIUC, there isn't such a potential issue since the webhook will work before the kube-scheduler will do anything to the pod, right?

kueue/pkg/controller/jobs/pod/pod_webhook.go

Line 186 in 1579d75

pod.pod.Spec.SchedulingGates = append(pod.pod.Spec.SchedulingGates, corev1.PodSchedulingGate{Name: SchedulingGateName})

[alculquicondor]

Keep in mind that it's not possible to add scheduling gates to an existing pod. They always have to be set on creation or by webhooks.

[kerthcet]

there isn't such a potential issue since the webhook will work before the kube-scheduler will do anything to the pod

Yes, I found it, I'm in an environment with kubernetes 1.24 which doesn't support scheduling gates. sorry for that.

[alculquicondor]

/close

Splitting remaining use case into a different issue: #1656

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close

Splitting remaining use case into a different issue: #1656

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.