Improve docs of leaderelection configuration

[ChrsMark]

Follow up of #20512.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[masci]

Following up #20512 (comment)

I would say that the current approach is smoother for every case. I don't think we should remove the Deployment completely by now since in many cases it can be really helpful when it comes to scaling where someone may want to collect cluster-wide metrics from big clusters.

I'm not sure we should chase complex use cases but focus on onboarding instead.

With the current approach, users don't benefit from the simplified configuration provided by leader election out of the box: on the contrary, they have to learn about specifics of the feature through the comments and change the manifest in several points to simplify the setup, which is a bit of a paradox.

If I'm running a big, complex cluster requiring to fine-tune observability, chances are I won't even use the manifests as they are in the repo because the default resources provided to the standalone pods won't fit any possible case.

To sum up, I think with the current setup we're not providing a turnkey solution to anybody, while with leader election we have the chance to build a good user story to simplify onboarding.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #20601 updated]

• Start Time: 2020-09-02T08:51:51.369+0000

• Duration: 61 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2732
Skipped	722
Total	3454

Steps errors

Expand to view the steps failures

• Name: Install Go 1.14.7

  • Description: .ci/scripts/install-go.sh

  • Duration: 1 min 32 sec

  • Start Time: 2020-09-02T09:15:37.801+0000

  • log

• Name: Install docker-compose 1.21.0

  • Description: .ci/scripts/install-docker-compose.sh

  • Duration: 1 min 33 sec

  • Start Time: 2020-09-02T09:15:47.840+0000

  • log

[ChrsMark]

@exekias @jsoriano what do you think about @masci 's proposal of completely removing the old proposed manifests that include the Deployment and keep only the new approach with the leader election configuration within the Daemonset?

I'm ok with this, however I'm only concerned about users that already use Deployment manifests (breaking change ?).

[jsoriano]

@exekias @jsoriano what do you think about @masci 's proposal of completely removing the old proposed manifests that include the Deployment and keep only the new approach with the leader election configuration within the Daemonset?

Maybe we have to check how big a deployment needs to be to require a dedicated pod. There are two main reasons why this may be needed:

• Collection of cluster-level metrics require significantly more resources, and configuring these resources in the DaemonSet will make most of the pods to reserve much more resources than they need.
• Performance problems with leader election caused by having two many candidates.

If we are more or less sure that for most of the cases this is not going to be a problem, then I am ok with keeping the new approach only.

I'm ok with this, however I'm only concerned about users that already use Deployment manifests (breaking change ?).

We can keep a dummy deployment with the same name and zero replicas to replace the previous one avoiding a breaking change.

[ChrsMark]

@exekias @jsoriano what do you think about @masci 's proposal of completely removing the old proposed manifests that include the Deployment and keep only the new approach with the leader election configuration within the Daemonset?

Maybe we have to check how big a deployment needs to be to require a dedicated pod. There are two main reasons why this may be needed:

• Collection of cluster-level metrics require significantly more resources, and configuring these resources in the DaemonSet will make most of the pods to reserve much more resources than they need.
• Performance problems with leader election caused by having two many candidates.

If we are more or less sure that for most of the cases this is not going to be a problem, then I am ok with keeping the new approach only.

My feeling is that this kind of performance issues will hit users with real big clusters and if they have to fine tune their setup to resolve this issues it shouldn't be a big deal to evaluate the solution of the singleton Pod using a deployment. For most of our users keeping only the new approach from now on would improve their onboarding experience.

I'm ok with this, however I'm only concerned about users that already use Deployment manifests (breaking change ?).

We can keep a dummy deployment with the same name and zero replicas to replace the previous one avoiding a breaking change.

I think that having a dummy Deployment spec inside our manifests making things more complicated for users that are not so familiar with the whole setup. On the other hand I'm not sure if this can be considered a breaking change since docs are linked to the versions of the manifests and only old users that might want adopt the new approach might end up with "orphan" Deployment Pods on their clusters however this is something that not so unusual and it's not that bad.

In this, I'm leaning towards keeping only the new approach.
@masci @jsoriano @exekias let me know what you think.

[exekias]

It would sound like the general case would benefit from going only for the DaemonSet, and then we can document how to switch to a cluster-wide Deployment for big clusters.

This is not really a breaking change, as it won't break existing deployments, we are just changing how the new ones will happen.

[jsoriano]

Ok, should we mention something in the docs in case someone upgrades by using the new manifests?

"orphan" Deployment Pods on their clusters however this is something that not so unusual and it's not that bad.

Well, these orphan deployments will be collecting duplicated metrics, with an old version.

[ChrsMark]

Ok, should we mention something in the docs in case someone upgrades by using the new manifests?

New docs will only point to new manifests and only refer to the new approach. And wouldn't be an overkill to mention this in the old versions' docs?

"orphan" Deployment Pods on their clusters however this is something that not so unusual and it's not that bad.

Well, these orphan deployments will be collecting duplicated metrics, with an old version.

[jsoriano]

New docs will only point to new manifests and only refer to the new approach. And wouldn't be an overkill to mention this in the old versions' docs?

I meant to mention only something like "if you deployed these manifests before 7.10, you may need to manually remove the deployment". But we are not telling anything about upgrades so far, so maybe it is ok to don't say anything and users just change the versions in the manifests they are using.

[masci]

Left few wording suggestions for the docs but this LGTM!