Pod scheduling readiness support in Cluster Autoscaler

[x13n]

Per the suggestion on last SIG meeting, opening an issue to start a discussion.

Which component are you using?:

cluster-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

This isn't exactly a specific feature I have in mind, I want to start a discussion on what to do in CA about Pod Scheduling Readiness KEP (kubernetes/enhancements#3522). It introduces a way for kube-scheduler to postpone scheduling until certain additional criteria are met for the pod. I think CA has two options on how to approach this feature:

• Pretend these pods do not exist (probably a no-op in CA code, but it would be good to check)
• Detect such pods and potentially trigger scale-up to ensure they can schedule once ready

Describe the solution you'd like.:

I'm leaning towards just ignoring such pods in CA. The downside of detecting is that they may never actually become ready, which would lead to waste of cluster resources.

Describe any alternative solutions you've considered.:

Described both above.

Additional context.:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3521-pod-scheduling-readiness

CC @Huang-Wei @ahg-g in case you have an opinion.

[ahg-g]

By design we wanted pods with schedulingGates set to not trigger autoscaling. The idea is that such pods still need to go through some validation logic (like if those pods have quota to schedule) before deciding if/when to consider them for placement.

So I strongly support option 1, otherwise this negates some of this feature's purpose.

[ahg-g]

We will add the condition PodScheduled=false to those pods, but with a different reason than the one the scheduler sets when it actually fails to find a node, so one thing we need to do on the CA side is to ignore pods with this new reason value.

Otherwise, we can use PodScheduled=unknown, which means no changes at all on CA. But I prefer the first approach.

[x13n]

Thanks, that makes sense to me. I guess the only thing to do would be to verify CA works as expected when such pods are ready.

As far as I can tell without actually testing, CA already filters by reason, so no changes should be required:

autoscaler/cluster-autoscaler/utils/kubernetes/listers.go

Lines 170 to 172 in c127763

	if condition != nil && condition.Status == apiv1.ConditionFalse && condition.Reason == apiv1.PodReasonUnschedulable {
	unschedulablePods = append(unschedulablePods, pod)
	}

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Huang-Wei]

This can be closed. Now a scheduled gated pod would carry a condition like:

- type: PodScheduled
  status: False
  reason: SchedulingGated
  message: Scheduling is blocked due to non-empty scheduling gates

and it won't trigger autoscaling.

[x13n]

Ack, thanks!

/close

[k8s-ci-robot]

@x13n: Closing this issue.

In response to this:

Ack, thanks!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.