[Agent] Implement leader election on k8s for Elastic Agent

[ChrsMark]

In Beats we have leader election feature for k8s(#19731), so as to make it possible to avoid deploying a singleton instance of Metricbeat via a k8s Deployment for cluster wide metrics collection. The implementation is based on go-client's leader election implementation: https://pkg.go.dev/k8s.io/client-go/tools/leaderelection

It would be nice to support the similar in Elastic Agent too. What comes first to my mind is a separate provider called kubernetes_leaderelection since its implementation currently is not related to kubernetes provider (no resource discovery is required).

An example configuration would look like this:

providers:
  kubernetes_leaderelection:
    leader_lease: leader-election-elastic-agent

leader_lease: This will be the name of the lock lease. One can monitor the status of the lease with kubectl describe lease leader-election-elastic-agent.

And then users can define inputs that would be enabled only by the leader Pod with:

- name: kube-state-metrics
  type: kubernetes/metrics
  use_output: default
  meta:
    package:
      name: kubernetes
      version: 0.2.8
  data_stream:
    namespace: default
  streams:
   - data_stream:
       dataset: kubernetes.state_node
       type: metrics
     metricsets:
       - state_node
     add_metadata: true
     hosts:
       - 'kube-state-metrics:8080'
     period: 10s
     condition: ${kubernetes_leaderelection.leader} == 'true'

The input will be enabled by the Pod that will acquire the lock and will set kubernetes_leaderelection.leader to true.
(condition can be set on input level too I guess so as to be the same for all state_* datastreams)

With this setup Managed version could be simplified since we will only have to deal with a Deamonset with the same configuration along the Pods.

@blakerouse @ph @ruflin @masci @exekias @mukeshelastic let me know what you think.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[mukeshelastic]

@ChrsMark I am trying to understand the benefits of this feature. Without this feature, users need to deploy daemonset to collect worker node metrics and logs and then a separate and just a single pod deployment for a cluster for collecting cluster level metrics, master node metrics and logs? So the only con is a separate deployment to track these metrics. Is that right or there are additional cons if we don't support this feature.

[blakerouse]

At first I was going to say this should really already be handled with the coordinator with Fleet Server. But then that would prevent this from working in standalone mode. Being we need standalone more to work, I think this is a good proposal and something we should do.

I think the need for a specific Agent running a specific configmap just to get this information, make the configuration more complicated. Having this would simplify this greatly.

[mukeshelastic]

@exekias Are you able to confirm the benefits of this feature? Is it more than what I stated in above comment?

[exekias]

Your statement is accurate @mukeshelastic. I would say leader election is not a blocker / must-have.

It's still a desirable feature to fit in when we have solved the main parts, as in our experience, the extra Deployment needed causes confusion and complicates the architecture.