Running Beats for k8s monitoring

[pebrc]

Relates to #2051

Should ECK support running Beats on k8s:

Two main options come to mind:

• create a Helm chart or similar YAML packaging that deploys for example Metricbeat and Filebeat alongside ECK
• support *beats as first class objects in ECK via a CRD

apiVersion: beats.k8s.elastic.co/v1beta1
kind: Filebeat
metadata:
  name: ""
  namespace: ""
spec:
  config:
    inline: {}   # we use inline config for all other resources but with structural schema around the corner
    configMap: "" # we might want to support just a ConfigMap reference
  deamonSet:
    template: {}  # give user the ability to template the spec for the DaemonSet similar to what we do for all other resources
  elasticsearchRef:
    name: ""
    namespace: ""
  version: 7.5.0
---
apiVersion: beats.k8s.elastic.co/v1beta1
kind: Metricbeat
metadata:
  name: ""
  namespace: ""
spec:
  config:
    inline: {}
    configMap: ""
  modules:
    inline: {}
    configMap: ""
  deployment:
    template: {}
  deamonSet:
    template: {}
  elasticsearchRef:
    name: ""
    namespace: ""
  version: 7.5.0

The main motivation for involving ECK in the deployment of Beats is that it can add value by automating tedious or error prone aspects of configuring Beats. For example:

• ECK could automatically configure the necessary certificates to access a monitoring Elasticsearch cluster
• ECK could automated user setup for the various beats
• ECK could make sure Beats are rotated if configuration changes happen that are not automatically picked up by Beats. It seems like this is the case for certificate rotation (to be verified)

With that said we could also explore a third option which would be to keep Beats separate from ECK but to facilitate integration for example with a MutatingAdmissionWebhook that injects user credentials and Elasticsearch certificates into arbitrary containers as needed, which could be used to inject these credentials into Beats containers.

[anyasabo]

I'm in favor of creating a proof of concept of a CRD. At least personally I struggle with how we could make this as useful as ES/Kibana/etc. The "hard" part IMO is the configuration (not so much the orchestration), but also for beats to be useful people need to apply a lot of their own configurations. As you said the biggest thing the operator can help with is hooking up the credentials/outputs. Maybe there is a way to provide useful defaults though that make the common case easy.

[david-kow]

+1 on a new CRD. I think there is value in being able to version and reason about the specification that we would have in a CRD.

[david-kow]

This is superseeded by #3040, hence closing.