K8S Processor: Include more tags

[pmm-sumo]

Currently, the processor supports following tags:

• namespace
• podName
• deployment
• cluster
• node
• startTime

It would be great to include more tags, such as:

• hostName
• containerName
• daemonSetName
• serviceName
• statefulSetName

Additionally, it would be good to have an ability to include all labels and annotations

[bogdandrutu]

I like having them at least as options for the config, not always on :)

[Oberon00]

hostName should be supported by host.name (https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/resource/semantic_conventions/host.md) and containerName by container.name (https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/resource/semantic_conventions/container.md)

[hansh0801]

I am currently using this processor to map deployment data from istio's sidecar trace data. I would be very happy if it supports statefulset and daemonset other than deployment.

[pmm-sumo]

I am currently using this processor to map deployment data from istio's sidecar trace data. I would be very happy if it supports statefulset and daemonset other than deployment.

What we ended up with (at least for now), was having a custom changeset that provides some of this capabilities (and few others) on top of existing processor. It's not the highest priority, but we would like to move some of those changes back to the main version (the scope of the changes was not accepted previously, so we would need to redo it in a more efficient way). But, if that helps, you could try it out: https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/main/processor/k8sprocessor

[hansh0801]

I am currently using this processor to map deployment data from istio's sidecar trace data. I would be very happy if it supports statefulset and daemonset other than deployment.

What we ended up with (at least for now), was having a custom changeset that provides some of this capabilities (and few others) on top of existing processor. It's not the highest priority, but we would like to move some of those changes back to the main version (the scope of the changes was not accepted previously, so we would need to redo it in a more efficient way). But, if that helps, you could try it out: https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/main/processor/k8sprocessor

Thank you very much ! When will the code (https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/main/processor/k8sprocessor) be upstreamed to otel-contrib main code?

[pmm-sumo]

Thank you very much ! When will the code (https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/main/processor/k8sprocessor) be upstreamed to otel-contrib main code?

Thanks @hansh0801 This is a bit complex, you can refer to the original discussion on that for context.

We started working on a reimplementation using a more lightweight approach, but hit several issues, such lack of agreement how to treat tags when there are several containers running in a Pod (this information cannot be really extracted on that level). Or how to handle host.name vs k8s.pod.hostname (i.e. if it's the same thing or a different one).

The plan was to go back to this after GA

[dmitryax]

@povilasv thanks for starting working on the issue in #12957. Do you want me to assign it to you?

[dmitryax]

I would make a correction to the original request.

• hostName - this seems like beyond k8s attribute processor responsibilities. If user wants that attribute taken from an underlying node, they should install collector as daemonset and use a regular attributes processor with downward k8s API.
• containerName - this is an identifying attribute of a container within a pod. Using client connection association, we can only get a pod identifier not container. Logs coming from the filelog receiver should already have this attribute taken from a file path.
• daemonSetName
• serviceName - I don't think we should fetch this attribute as there is no direct ownership relationships between service and pod. This also increases load on k8s API. We can revisit this later if we really need this attribute.
• statefulSetName

I suggest to add the following attributes as optional that can be clearly associated with a pod:

• k8s.replicaset.name and k8s.replicaset.uid
• k8s.deployment.uid
• k8s.statefulset.name and k8s.statefulset.uid

We also should probably make k8s.deployment.name optional for consistency and make it more robust as part of #12936

[povilasv]

@povilasv thanks for starting working on the issue in #12957. Do you want me to assign it to you?

Yes, please. I want to also add DaemonSet, Job and CronJob attributes. What do you think?

[dmitryax]

I want to also add DaemonSet, Job and CronJob attributes

Makes sense. Let's add Job and CronJob as optional attributes as well 👍

[povilasv]

@dmitryax I think this is done, or should we also implement more reliable way of getting CronJob metadata similiar to #12936?

[dmitryax]

@povilasv I believe functionality-wise this issue can be marked as completed. #12936 can be handled separately. Thanks for your help!