[Stack Monitoring] Elasticsearch monitoring with Metricbeat and Filebeat as sidecars

[thbkrkr]

Adds a new monitoring field to the Elasticsearch resource to configure one or two different Elasticsearch references to set up stack monitoring with Metricbeat and log delivery with Filebeat. The referenced ES are used to send the data collected by the beats.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
spec:
  monitoring:
    metrics:
      elasticsearchRefs: 
        - name: m1
          namespace: m1
    logs:
      elasticsearchRefs:
        - name: m2
          namespace: m2

This is implemented with a multiple association of ES type (1 es <-> [1|2] es).

New stackmon packages contains functions to:

• set the xpack.monitoring.* settings in the ES config
• enable stack logging for the ES container using the environment variable ES_LOG_STYLE=file
• add volumes to the ES pod to mount the ES CA and the beats configurations in the beats sidecar containers
• inject Metricsbeat and Filebeat as sidecar container in the ES pod than that of ES container with volumeMounts for the ES CA
• reconcile the Metricsbeat and Filebeat configurations in ConfigMaps

The beats configuration is built from a base configuration merged with the output config section which defines es info to send data. For Metricsbeat, the base config is a template to inject the es info to collect data.

A hash of the two beats config files is added in a pod label to ensure pods are rotated when es user passwords are rotated.

YAML example for testing

apiVersion: v1
kind: Namespace
metadata:
  name: a
---
apiVersion: v1
kind: Namespace
metadata:
  name: b
---
apiVersion: v1
kind: Namespace
metadata:
  name: c
---
#######################################################################
# Monitored Elasticsearch
#######################################################################
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: test
  namespace: a
spec:
  version: 7.14.0-SNAPSHOT
  # --------------------- #
  monitoring:
    metrics:
      elasticsearchRefs: 
        - name: m1
          namespace: b
    logs:
      elasticsearchRefs:
        - name: m1
          namespace: b
        # - name: m2
        #   namespace: c
  # --------------------- #
  nodeSets:
  - name: master
    count: 2
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: test
  namespace: a
spec:
  version: 7.14.0-SNAPSHOT
  count: 1
  elasticsearchRef:
    name: test
    namespace: a
---
#######################################################################
# Monitoring clusters
#######################################################################
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: m1
  namespace: b
spec:
  version: 7.14.0-SNAPSHOT
  nodeSets:
  - name: master
    count: 2
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: m1
  namespace: b
spec:
  version: 7.14.0-SNAPSHOT
  count: 1
  elasticsearchRef:
    name: m1
    namespace: b
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: m2
  namespace: c
spec:
  version: 7.14.0-SNAPSHOT
  nodeSets:
  - name: master
    count: 2
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: m2
  namespace: c
spec:
  version: 7.14.0-SNAPSHOT
  count: 1
  elasticsearchRef:
    name: m2
    namespace: c

Limitations

• Minimum supported Stack version is 7.14.0 (to benefit from ES_LOG_STYLE=file)
• Custom Elasticsearch image that don't follow the Elastic scheme ($registry/elasticsearch/elasticsearch:$version) are not supported To use custom beat images, you have to override the podTemplate.
• The monitored Elasticsearch is not deployed until monitoring clusters are not ready yet or the association are not configured
• monitoring.[metrics|logs].elasticsearchRefs accepts only one Elasticsearch reference. It's a slice to future proof the API for Elastic agent.

Relates to #4183.

[pebrc]

Looks promising! I wonder however if we can model this with just one additional association type es->es instead of two separate ones for metricbeat and filebeat?

[thbkrkr]

I wonder however if we can model this with just one additional association type es->es instead of two separate ones for metricbeat and filebeat?

It was my first idea that I too quickly abandoned because I did not yet understand very well how associations work. I started again and you're right it's a little simpler and cleaner. I will update the PR.

[barkbay]

Out of curiosity I did a test with version 6.8 of the stack, Metricbeat containers do not start:

2021-06-03T08:59:11.655Z        INFO    instance/beat.go:280    Setup Beat: metricbeat; Version: 6.8.16
2021-06-03T08:59:11.657Z        INFO    elasticsearch/client.go:164     Elasticsearch url: https://monitoring-es-http.demo.svc:9200
2021-06-03T08:59:11.657Z        INFO    [publisher]     pipeline/module.go:110  Beat name: monitored-es-master-0
2021-06-03T08:59:11.657Z        INFO    instance/beat.go:359    metricbeat stopped.
2021-06-03T08:59:11.657Z        ERROR   instance/beat.go:906    Exiting: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr cluster_stats index index_recovery index_summary ml_job node_stats shard]
Exiting: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr cluster_stats index index_recovery index_summary ml_job node_stats shard]

I think it's because enrich is not supported before 7.5 (not sure about the version). It seems that we need to generate the Metricbeat configuration according to the stack version and plan to add an e2e test to validate version dependant behaviour (can be added in a subsequent PR).

Also, reading this compatibility matrix I'm wondering if we should prevent association between a monitored ES/Beat 6.x and a monitoring ES 7.x ?

[david-kow]

Couple small comments, I also played with it a bit and it works nicely.

Do we plan to include other apps (Kibana, Beats) monitoring as well?

[thbkrkr]

Remains to be done:

• min supported version (>= 7.14)
• dedicated role

[pebrc]

Filebeat keeps crash looping for me with one or more modules must be configured I haven't quite figured out what causes it but my guess is YAML indentation is to blame somewhere. Otherwise I think we are almost ready to merge 👍

[thbkrkr]

Filebeat keeps crash looping for me with one or more modules must be configured I haven't quite figured out what causes it but my guess is YAML indentation is to blame somewhere.

Bad copy-paste f4759fd 🤦‍♂️

[pebrc]

LGTM! Nice work @thbkrkr 👍 I have done a few tests, ran the e2e test you added, explored the validations. There is probably room for more in-depth testing of edge cases where a resource constrained Metricbeat or Filebeat bring down a whole Pod but I haven't tried that myself.

[thbkrkr]

Thanks Peter! I'm going to follow-up with Kibana monitoring.

There is probably room for more in-depth testing of edge cases where a resource constrained Metricbeat or Filebeat bring down a whole Pod but I haven't tried that myself.

This is a disadvantage of the sidecar pattern. If it happens, you will have to increase the compute resources.

I did a quick test be restricting memory for Metricbeat:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: alpha
  namespace: production
spec:
  version: 7.14.0-SNAPSHOT
  monitoring:
    metrics:
      elasticsearchRefs:
        - name: monitoring
          namespace: observability
    logs:
      elasticsearchRefs:
        - name: monitoring
          namespace: observability
  nodeSets:
  - name: master
    count: 2
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
          - name: metricbeat
            resources:
              limits:
                memory: 20M

Elasticsearch is green and reachable but from the operator's point of view, the ES resource is red and it blocks the deployment of the associated Kibana.

> k get elastic,sts,deploy,pods
NAME                                               HEALTH    NODES   VERSION           PHASE             AGE
elasticsearch.elasticsearch.k8s.elastic.co/alpha   unknown           7.14.0-SNAPSHOT   ApplyingChanges   14m

NAME                                 HEALTH   NODES   VERSION           AGE
kibana.kibana.k8s.elastic.co/alpha   red              7.14.0-SNAPSHOT   14m

NAME                               READY   AGE
statefulset.apps/alpha-es-master   0/2     13m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/alpha-kb   0/1     1            0           13m

NAME                            READY   STATUS             RESTARTS   AGE
pod/alpha-es-master-0           2/3     CrashLoopBackOff   7          13m
pod/alpha-es-master-1           2/3     CrashLoopBackOff   7          13m
pod/alpha-kb-6cf84594b9-rm4wp   0/1     Running            0          13m

> eckurl production alpha '/_cat/health' 
1625568204 10:43:24 alpha green 2 2 2 1 0 0 0 0 - 100.0%

To increase the memory limit, you need to apply the updated manifest with a new limit and kill the pod.