Move away from encouraging Deployment for additional schedulers

[Adirio]

This is a...
/sig scheduling
/kind feature

Problem:
Currently it is suggested in Tasks/Administer a Cluster/Configure multiple schedulers (source) to deploy schedulers with a Deployment and disabling leader election.

Rolling updates of Deployments may imply that multiple replicas of the scheduler are running concurrently and thus they can interfere each other as leader election was turned off.

Proposed Solution:
StatefulSets or Deployments with Recreate deployment strategy type would solve this issue. I personally prefer a StatefulSet as it:

1. Doesn't allow to swap the deployment strategy type afterwards.
2. Avoids creating an additional abstraction layer (ReplicaSet).
3. Enforces to use persistent volumes in case some volume wants to be used, which I think should be encouraged.

Page to Update:
https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#define-a-kubernetes-deployment-for-the-scheduler

[sftim]

If this guide were to recommend enabling leader election, and explained why, would that make using a Deployment suitable / acceptable?

[Adirio]

In my opinion, leader election being false with a single replica is the right approach as the other way round would only mean additional overhead without any gain. In this case I don't see a Deployment fitting.

On the other hand if multiple replicas were to be deployed of the scheduler, leader election should be set to true. Leader election would mean the Recreate strategy argument is no longer valid. The second and third argument are not that important and thus the difference from a StatefulSet and a Deployment would be greatly reduced.

So in a HA scheduler with multiple replicas both StatefulSet and Deployment would be valid while in single instance schedulers without leader election StatefulSet should be prefered. I would also like to mention that for a HA replicated scheduler, a PodAntiAffinity to itself should be set in order to avoid collocating multiple replicas.

[sftim]

It sounds like we could decide between:

• illustrating using multiple kinds of scheduler, leaving aside topics such as HA so as to keep the page simple
  • and maybe signpost the reader to the HA topic for further reading
• demonstrating a recommended / good practise approach: multiple replicas for each controller type, leader elections, PodAntiAffinity, etc

and then tweak the document based on which approach we favor. Does that sound reasonable?

[Adirio]

I was sticking to the simple approach as that document is usually the starting point to extra-scheduler setups. In the simplest case a single instance with leader election disabled as it is now is enough. But with leader election turned off deployment's default rolling updates are dangerous and thats why I suggested to change to StatefulSet.

We could actually also add a comment at the end mentioning that to achieve scheduler resillience multiple replicas with leader election turned on and pod anti affinity to itself. In this complex case StatefulSets do not offer a lot over Deployemnts but they are also a nice fit so I would not mention switching to Deployment in this case (KIS).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.