Fix issue where autodiscover hints default configuration was not being copied.

[blakerouse]

What does this PR do?

FIxes issue where when a default configuration is provided to autodiscovery in filebeat it would cause the elasticsearch module to have log messages repeated and appear in different datasets.

Why is it important?

Without this change log messages would be repeated and reported in different datasets unless hints.default_config.enabled: false was set. Which also prevented filebeat from reading the logs of pods that didn't have the annotation of co.elastic.logs/enabled: true.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

1. Build the docker image cd filebeat && PACKAGES="linux/amd64" mage package
2. Push it to a registry (I used a personal private registry)
3. Use the image in the filebeat configurations:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    logging:
      level: debug
      metrics.enabled: false
      selectors:
        - "*"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
      annotations:
        co.elastic.logs/enabled: "false"
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      imagePullSecrets:
        - name: blake-creds
      containers:
      - name: filebeat
        image: blakerouse/elastic:filebeat-master
        args: [
          "run", "-c", "/etc/filebeat.yml", "-e"
        ]
        env:
        - name: ELASTIC_CLOUD_ID
          value: "***"
        - name: ELASTIC_CLOUD_AUTH
          value: "***"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---

4. Deploy elasticsearch with the elasticsearch module set in the hints.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
data:
  elasticsearch.yml: |
    discovery.type: single-node
    xpack.security.enabled: false
    xpack.monitoring.enabled: false
  ES_JAVA_OPTS: -Xms256m -Xmx256m
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
      annotations:
        co.elastic.logs/enabled: "true"
        co.elastic.logs/module: elasticsearch
    spec:
      securityContext:
        fsGroup: 1000
      containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 500Mi
          securityContext:
            privileged: true
            runAsUser: 1000
            capabilities:
              add:
                - IPC_LOCK
                - SYS_RESOURCE
          image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
          env:
            - name: ES_JAVA_OPTS
              valueFrom:
                configMapKeyRef:
                  name: elasticsearch-config
                  key: ES_JAVA_OPTS
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: /_cluster/health?local=true
              port: 9200
            initialDelaySeconds: 5
          ports:
            - containerPort: 9200
              name: es-http
            - containerPort: 9300
              name: es-transport
          volumeMounts:
            - name: es-data
              mountPath: /usr/share/elasticsearch/data
            - name: elasticsearch-config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              subPath: elasticsearch.yml
      volumes:
        - name: elasticsearch-config
          configMap:
            name: elasticsearch-config
            items:
              - key: elasticsearch.yml
                path: elasticsearch.yml
        - name: es-data
          persistentVolumeClaim:
            claimName: elasticsearch-data

5. Check that the log messages are not repeated and are in the correct event.dataset.

Related issues

• Closes [Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[ChrsMark]

It looks good @blakerouse ! I only left a question regarding the comments that were added, feel free to correct me!

[jsoriano]

Great investigation @blakerouse! Change looks good and safe.

Do you think we could add a test case that reproduces this issue and checks that multiple configs don't interfere one with each other?

[blakerouse]

@ChrsMark @jsoriano This is ready for another look, thank!

[jsoriano]

Thanks for adding a test!