Cherry-pick #20984 to 7.x: Add container ECS fields in kubernetes metadata

[ChrsMark]

Cherry-pick of PR #20984 to 7.x branch. Original message:

What does this PR do?

This PR adds container.id, container.runtime and container.image.name in kubernetes metadata to be compliant with https://www.elastic.co/guide/en/ecs/current/ecs-container.html

Why is it important?

In final events the metadata do not include container.id, container.runtime and container.image.name.

Related issues

• Closes Container id not included in Kubernetes events #20982
• Related Add ECS container fields for kubernetes #10926

How to test this PR

1. Configure and run Filebeat in k8s:

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

Make sure that container.id, container.runtime and container.image.name exist in the final documents in ES.

2. Configure and run Filebeat in k8s:

filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            indexers:
            - container:
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Screenshots

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

💔 Tests Failed

Expand to view the summary

Build stats

• Build Cause: [Pull request #21033 opened]

• Start Time: 2020-09-09T09:14:56.735+0000

• Duration: 66 min 2 sec

Test stats 🧪

Test	Results
Failed	2
Passed	19633
Skipped	1858
Total	21493

Test errors

Expand to view the tests failures

• Name: Build and Test / Libbeat / Libbeat oss / TestOutputReload – pipeline

  • Age: 1
  • Duration: 66.69
  • Error Details: Failed

• Name: Build and Test / Libbeat / Libbeat oss / TestOutputReload/client – pipeline

  • Age: 1
  • Duration: 36.79
  • Error Details: Failed

Steps errors

Expand to view the steps failures

• Name: Mage build test

  • Description: mage build test

  • Duration: 23 min 34 sec

  • Start Time: 2020-09-09T09:40:41.833+0000

  • log

• Name: Make test

  • Description: make -C generator/_templates/beat test

  • Duration: 7 min 20 sec

  • Start Time: 2020-09-09T10:00:38.203+0000

  • log

• Name: Recursively delete the current directory from the workspace

  • Description: script returned exit code 2

  • Duration: 1 min 20 sec

  • Start Time: 2020-09-09T10:08:18.793+0000

  • log

Log output

Expand to view the last 100 lines of log output

[2020-09-09T10:20:00.546Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21033-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21033/source.tgz
[2020-09-09T10:20:10.066Z] + tar --version
[2020-09-09T10:20:10.393Z] + tar -xpf source.tgz
[2020-09-09T10:20:23.021Z] + rm source.tgz
[2020-09-09T10:20:23.048Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats
[2020-09-09T10:20:23.100Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Lint
[2020-09-09T10:20:23.275Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Winlogbeat-oss
[2020-09-09T10:20:23.466Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-09-09T10:20:23.645Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-09-09T10:20:23.814Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/dockerlogbeat
[2020-09-09T10:20:23.982Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-09-09T10:20:24.163Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Journalbeat
[2020-09-09T10:20:24.347Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-09-09T10:20:24.548Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-09-09T10:20:24.748Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Packetbeat-Linux
[2020-09-09T10:20:25.011Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-09-09T10:20:25.207Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-09-09T10:20:25.400Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-09-09T10:20:25.590Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-09-09T10:20:25.784Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-09-09T10:20:25.979Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-09-09T10:20:26.174Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Heartbeat-oss
[2020-09-09T10:20:26.364Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Heartbeat-Windows
[2020-09-09T10:20:26.553Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-09T10:20:26.746Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-09-09T10:20:26.932Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-09-09T10:20:27.148Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-09-09T10:20:27.346Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-09-09T10:20:27.537Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Libbeat-x-pack
[2020-09-09T10:20:27.732Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-09T10:20:27.937Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-09-09T10:20:28.124Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-09-09T10:20:28.307Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-09-09T10:20:28.486Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-09-09T10:20:28.668Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-09T10:20:28.873Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-09-09T10:20:29.067Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Functionbeat-Windows
[2020-09-09T10:20:29.249Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Packetbeat-Windows
[2020-09-09T10:20:29.453Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-09T10:20:29.642Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-09-09T10:20:29.837Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-09-09T10:20:30.029Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-09-09T10:20:30.228Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-Windows
[2020-09-09T10:20:30.413Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-09-09T10:20:30.611Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Libbeat-oss
[2020-09-09T10:20:30.801Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-09-09T10:20:31.082Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-09T10:20:31.262Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-09-09T10:20:31.443Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Filebeat-oss
[2020-09-09T10:20:31.711Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-09-09T10:20:32.046Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-09-09T10:20:32.925Z] + cat
[2020-09-09T10:20:32.925Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-09T10:20:32.925Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-09T10:20:41.087Z] runbld>>> runbld started
[2020-09-09T10:20:41.087Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-09T10:20:42.032Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-09T10:20:42.032Z] runbld>>> Matches in the system config:
[2020-09-09T10:20:42.032Z] runbld>>> - Matched ^elastic\+beats
[2020-09-09T10:20:42.032Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-09T10:20:43.422Z] runbld>>> Debug logging enabled.
[2020-09-09T10:20:43.422Z] runbld>>> Storing result
[2020-09-09T10:20:43.684Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-09T10:20:43.684Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200909102043-5180E480
[2020-09-09T10:20:43.684Z] runbld>>> Adding system facts.
[2020-09-09T10:20:45.080Z] runbld>>> Adding vcs info for the latest commit:  7bdb522ceedc235264092ebfb9d0eed0cab50b2e
[2020-09-09T10:20:45.080Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-09T10:20:45.080Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-09T10:20:45.080Z] Processing JUnit reports with runbld...
[2020-09-09T10:20:45.080Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-09T10:20:45.342Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-09T10:20:45.342Z] runbld>>> DURATION: 38ms
[2020-09-09T10:20:45.342Z] runbld>>> STDOUT: 40 bytes
[2020-09-09T10:20:45.342Z] runbld>>> STDERR: 49 bytes
[2020-09-09T10:20:45.342Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-09T10:20:45.342Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21033
[2020-09-09T10:20:46.293Z] runbld>>> Storing build metadata: 
[2020-09-09T10:20:46.293Z] runbld>>> Adding test report.
[2020-09-09T10:20:46.293Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats
[2020-09-09T10:20:47.240Z] runbld>>> Found 139 test output files
[2020-09-09T10:20:49.796Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-09T10:20:49.796Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-09T10:20:49.796Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-09T10:20:49.796Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-09T10:20:49.796Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-09T10:20:50.370Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-09T10:20:50.370Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21033/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-09T10:20:50.945Z] runbld>>> Test output logs contained: Errors: 0 Failures: 2 Tests: 21342 Skipped: 1574
[2020-09-09T10:20:50.945Z] runbld>>> Storing result
[2020-09-09T10:20:50.945Z] runbld>>> FAILURES: 2
[2020-09-09T10:20:51.521Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-09T10:20:51.521Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200909102043-5180E480
[2020-09-09T10:20:51.782Z] runbld>>> Email notification disabled by environment variable.
[2020-09-09T10:20:51.782Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-09T10:20:57.523Z] Running on beats-ci-immutable-ubuntu-1404-i386-1599646695961912538 in /var/lib/jenkins/workspace/Beats_beats_PR-21033
[2020-09-09T10:20:57.818Z] [INFO] getVaultSecret: Getting secrets
[2020-09-09T10:20:57.921Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-09T10:21:00.394Z] + chmod 755 generate-build-data.sh
[2020-09-09T10:21:00.394Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21033/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21033/runs/1 FAILURE 3962154
[2020-09-09T10:21:00.395Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21033/runs/1/steps/?limit=10000 -o steps-info.json