Support log and metrics delivery with restricted RBAC permissions

[pebrc]

We added support for Beats in ECK 1.2 and have published a YAML recipe that illustrates how to set up stack monitoring with Metricbeat and log delivery with Filebeat.

I order to run these recipes the user needs to be able to give Metricbeat access to a service account that allows queriying the k8s API and for Filebeat they need to be able to deploy a (privileged) DeamonSet.

We should investigate a simple way that allows users to extract log files and stack monitoring metrics without the need for elevated permission or the ability to deploy DaemonSets.

One approach that comes to mind is to revive the sidecar approach we discussed originally here #2415 for Metricbeat.

For Filebeat @david-kow has experimented with FIFOs to harvest output from the monitored applications std out

FIFO example

 working with stdin and pipe
apiVersion: v1
kind: Pod
metadata:
  name: test-pod8
spec:
  initContainers:
  - name: fifo
    image: docker.elastic.co/beats/filebeat:7.6.0
    command: [ 'bash', '-c', 'mkfifo -m 0666 /filebeat/sink' ]
    volumeMounts: 
    - name: sink
      mountPath: /filebeat
  containers:
  - name: app
    image: ubuntu:trusty
    command: [ 'bash', '-c', 'while [ true ]; do echo "$(date)"; sleep 1; done | tee $FILEBEAT_SINK' ]
    env:
    - name: FILEBEAT_SINK
      value: /filebeat/sink
    volumeMounts:
    - name: sink
      mountPath: /filebeat
  - name: filebeat
    image: docker.elastic.co/beats/filebeat:7.6.0
    command: [ 'bash', '-c', 'filebeat -e -c /etc/filebeat.yml <$FILEBEAT_SINK' ]
    env:
    - name: FILEBEAT_SINK
      value: /filebeat/sink
    volumeMounts:
    - name: sink
      mountPath: /filebeat
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml
  volumes:
  - name: sink
    emptyDir:
      medium: Memory
  - name: config
    configMap:
      defaultMode: 0744
      name: filebeat-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: stdin
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
    output.console:
      pretty: true

The goal of this feature would not only be to allow the setup in restricted k8s environments but also to make it easy to setup:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.11.0
  monitoring:
    logs:
      elasticsearchRef:
        namespace: monitoring-ns
        name: my-logging-cluster
    metrics:
      elasticsearchRef:
        namespace: monitoring-ns
        name: my-metrics-cluster
  nodeSets:
  - name: default
    count: 1

Of course there are a few assumptions and restrictions:

• we would not allow users to customize any aspect of the Metricbeat or Filebeat configuration
• it needs to be seen whether we can allow users to override the Pod template for these side car containers
• technically a separate stack monitoring sidecar for each Elasticsearch node is wasteful, but it is simpler to manage
• with access to the k8s APIs we would not be able to enrich the data with k8s specific metadata but maybe we compensate with additional operator generated metadata in the Filebeat and Metricbeat configs

[thbkrkr]

PoC with sidecars:

#########################
# Elastic apps to monitor
#########################
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: cluster01
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
      xpack.monitoring.collection.enabled: true
      xpack.monitoring.elasticsearch.collection.enabled: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          # enable rolling file appender
          env:
          - name: ES_LOG_STYLE
            value: file
        # metricbeat sidecar
        - name: metricbeat
          image: docker.elastic.co/beats/metricbeat:7.12.0
          args: [
            "-c", "/etc/metricbeat.yml",
            "-e"
          ]
          env:
          - name: ELASTICSEARCH_USERNAME
            value: elastic # TODO use a dedicated monitoring user
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: cluster01-es-elastic-user
          - name: MONITOR_HOST
            value: monitoring-es-http
          - name: MONITOR_PORT
            value: "9200"
          - name: MONITOR_USERNAME
            value: elastic
          - name: MONITOR_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: monitoring-es-elastic-user
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: metricbeat-config
            mountPath: /etc/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: monitoring-certs
            mountPath: /mnt/monitoring/tls.crt
            readOnly: true
            subPath: tls.crt
          - name: es-certs
            mountPath:  /mnt/elastic/tls.crt
            readOnly: true
            subPath: tls.crt
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.12.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e"
          ]
          env:
          - name: MONITOR_HOST
            value: monitoring-es-http
          - name: MONITOR_PORT
            value: "9200"
          - name: MONITOR_USERNAME
            value: elastic
          - name: MONITOR_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: monitoring-es-elastic-user
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: filebeat-config
            mountPath: /etc/filebeat.yml
            readOnly: true
            subPath: filebeat.yml
          - name: monitoring-certs
            mountPath: /mnt/monitoring/tls.crt
            readOnly: true
            subPath: tls.crt
          - name: elasticsearch-logs
            mountPath: /usr/share/elasticsearch/logs
        volumes:
        - name: metricbeat-config
          configMap:
            name: cluster01-metricbeat
        - name: monitoring-certs
          secret:
            secretName: monitoring-es-http-certs-public
        - name: es-certs
          secret:
            secretName: cluster01-es-http-certs-public
        - name: filebeat-config
          configMap:
            name: cluster01-filebeat
---          
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster01-metricbeat
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.modules:
    - module: elasticsearch
      metricsets:
        - ccr
        - cluster_stats
        - enrich
        - index
        - index_recovery
        - index_summary
        - ml_job
        - node_stats
        - shard
      period: 10s
      xpack.enabled: true
      hosts: ["https://localhost:9200"]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.verification_mode: "certificate"
      ssl.certificate_authorities: ["/mnt/elastic/tls.crt"]

    processors:
      - add_cloud_metadata: {}
    
    output.elasticsearch:
      hosts: ['https://${MONITOR_HOST}:${MONITOR_PORT}']
      username: ${MONITOR_USERNAME}
      password: ${MONITOR_PASSWORD}
      ssl.certificate_authorities: ["/mnt/monitoring/tls.crt"]
---          
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster01-filebeat
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
    - module: elasticsearch
      server:
        enabled: true
        var.paths:
          - /usr/share/elasticsearch/logs/*_server.json
      gc:
        enabled: true
        var.paths:
          - /usr/share/elasticsearch/logs/gc.log.[0-9]*
          - /usr/share/elasticsearch/logs/gc.log
          - /usr/share/elasticsearch/logs/gc.output.[0-9]*
          - /usr/share/elasticsearch/logs/gc.output
      audit:
        enabled: true
        var.paths:
          - /usr/share/elasticsearch/logs/*_audit.json
      slowlog:
        enabled: true
        var.paths:
          - /usr/share/elasticsearch/logs/*_index_search_slowlog.json
          - /usr/share/elasticsearch/logs/*_index_indexing_slowlog.json
      deprecation:
        enabled: true
        var.paths:
          - /usr/share/elasticsearch/logs/*_deprecation.json
    
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}

    setup.kibana:
      host: 'https://${MONITOR_HOST}:${MONITOR_PORT}'
      username: ${MONITOR_USERNAME}
      password: ${MONITOR_PASSWORD}

    output.elasticsearch:
      hosts: ['https://${MONITOR_HOST}:${MONITOR_PORT}']
      username: ${MONITOR_USERNAME}
      password: ${MONITOR_PASSWORD}
      ssl.certificate_authorities: ["/mnt/monitoring/tls.crt"]  
---
#########################
# Monitoring cluster
#########################
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitoring
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: monitoring
spec:
  version: 7.12.0
  count: 1
  elasticsearchRef:
    name: monitoring
  config:
    # enable the UI to reflect container level CPU usage, only displays info if CPU limits are set on the monitored ES cluster
    # https://www.elastic.co/guide/en/kibana/current/monitoring-settings-kb.html
    xpack.monitoring.ui.container.elasticsearch.enabled: true

[eedugon]

I'd like to share some thoughts around this feature, hope you don't mind.

I don't like much the approach of sidecar containers because potentially they could interfere with the main container performance and compete for resources with it.

For metrics my suggestion (a.k.a the solution I like :) ) would be to create one deployment per cluster to monitor, as it's a setup that I have used widely and it's a good balance between one process / container per instance (sidecar approach) and one process for multiple clusters. Of course we should find a way to measure the amount of resources (cpu & mem) needed depending on the amount of nodes of the cluster to monitor (i.e 50m cores per instance to monitor with a minimum request of 100m).

Using deployments with Metricbeat for Elasticsearch / Kibana (or even other Beats) metrics shouldn't require elevated permissions if i'm not mistaken.

For logs, due to the nature of the containers, it's true that it's more complicated if we don't want to rely on DaemonSets, and I don't have a simple solution to that. So probably a sidecar container tailing the logs (or the main container stdout/stderr) is the best approach here. But for metrics I would try to avoid it if possible.

My feeling is that Deployments (for metrics) and DaemonSets (for logs) will add less overhead to the running instances than implementing this with sidecar containers, but we definitely need elevated permissions if we want to get pods logs at kubernetes worker level by mounting the host logs...

[pebrc]

I don't like much the approach of sidecar containers because potentially they could interfere with the main container performance and compete for resources with it.

Sidecar containers on Kubernetes don't share resources with the main container (with the exception of the network) so there would be no resource contention between the two containers.

The reason we are still looking into the the sidecar approach for metrics is that it fits better into the existing CRD than a new deployment. The current CRD already contains a podTemplate that users can use to tweak the resources allocated to the individual containers for example. A separate deployment for Metricbeat/Agent would mean we would have to extend the API significantly to make this deployment configurable. For maximum efficiency and minimising resource consumption users can deploy a standalone Metricbeat Deployment already today.