kubernetes: remove need for kube_tag_prefix configuration

[patrick-stephens]

Is your feature request related to a problem? Please describe.

The Kubernetes filter requires kube_tag_prefix to be correctly configured to match the tail input plugin configuration. This is so that it then strips the path from the tag to leave just the filename of the log file so we can extract the relevant information to query the K8S API server for the pod metadata.

There are lots of problems occurring with this simple misconfiguration as it is not obvious it is required or when it goes wrong - separately see #6551 for exposing the error better.

The default assumes kube.* is expanded in the tail plugin to kube.var.log.containers.<filename> so any change in the tail plugin breaks this or if the logs are on a different path.

Describe the solution you'd like

Keep track of the actual filename directly in the metadata for the record and use that, remove the need for any coordination of configuration between plugins.
Potentially we could also then support other inputs too, e.g. they just need to set this metadata to support the query.

Describe alternatives you've considered

Additional context

This problem comes up often enough in various contexts (e.g. #8760) that I wrote a post on it: https://calyptia.com/blog/kubernetes-metadata-enrichment-with-fluent-bit-with-troubleshooting-tips

[edsiper]

@patrick-stephens I think we can extend the Kubernetes filter and promote to run it as a processor, in that way, we could detect and handle that internally instead of asking the user to configure it.

Actually I would love to simplify that plugin config as much as possible

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.