Auditbeat under k8s fails to enrich short-lived processes

[adriansr]

Please include configurations and logs if available.

For confirmed bugs, please report:

• Version: n/a
• Operating System: Linux
• Discuss Forum URL: n/a
• Steps to Reproduce:

The sample manifest for k8s provided in deploy/kubernetes/auditbeat-kubernetes.yaml uses the add_process_metadata and add_kubernetes_metadata to enrich events based on the process.pid field:

beats/deploy/kubernetes/auditbeat-kubernetes.yaml

Lines 32 to 44 in 2474f5b

	processors:
	- add_cloud_metadata:
	- add_process_metadata:
	match_pids: ['process.pid']
	include_fields: ['container.id']
	- add_kubernetes_metadata:
	host: ${NODE_NAME}
	default_indexers.enabled: false
	default_matchers.enabled: false
	indexers:
	- container:
	matchers:
	- fields.lookup_fields: ['container.id']

    processors:
      - add_cloud_metadata:
      - add_process_metadata:
          match_pids: ['process.pid']
          include_fields: ['container.id']
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          default_indexers.enabled: false
          default_matchers.enabled: false
          indexers:
            - container:
          matchers:
            - fields.lookup_fields: ['container.id']

The drawback of this approach is that it's sensitive to timing. If the process identified by process.pid has terminated by the time the add_process_metadata processor runs, the container.id lookup will fail and no k8s metadata will be added to the event.

This is causing some processes to not be to associated to a container at all, or the association to happen randomly.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[efd6]

Following up comment in 29410, because of the variety of contexts in /proc/N/cgroup and interactions with other parts of the ingest pipeline, I think an approach here would be to capture a copy of the cgroup file for a started process and allow the ingest pipeline to process out what it wants from there in much the same way that add_process_metadata does now.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!