[K8s] cluster metadata not added on all metrics

[emmanuelmathot]

Describe the enhancement:

The cluster metadata such as orchestrator.cluster.name and orchestrator.cluster.ip are not added on all kubernetes metrics. From usage and code review, I can see they are added to

• container
• node
• pod
• state_daemonset
• state_deployment
• state_job
• state_node
• state_pod
• state_replicaset
• state_service
• state_statefulset

but missing to the following metrics where they should be added

• proxy
• system
• volume
• apiserver
• state_daemonset
• state_persistentvolume
• state_persistentvolumeclaim
• state_storageclass
• event

Describe a specific use case for the enhancement or feature:

Being able to account the PVC usage for a given cluster.

[tetianakravchenko]

@emmanuelmathot could you give more details on your k8s setup? are you using managed kubernetes solutions?

[emmanuelmathot]

Yes, we have deployed managed kubernetes as described in https://www.elastic.co/guide/en/fleet/master/running-on-kubernetes-managed-by-fleet.html using fleet policies.
We also added a kubeadm-config config map to add the cluster metadata

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubeadm-config
  namespace: kube-system
data:
  ClusterConfiguration: |
    clusterName: {{ .Values.components.kubeadm_config.clusterName }}
    controlPlaneEndpoint: {{ .Values.components.kubeadm_config.controlPlaneEndpoint }}

We are using elastic cloud with Elasticsearch 7.15.1 and beats image same version

[lrfsh]

Hello,

I am using Elasticsearch 8.0.0 and the beats image same version and I have the same issue.

[RumRogerz]

I am seeing similar results, but with kubernetes metadata.

[tetianakravchenko]

metricsets:

• proxy
• system
• apiserver
• scheduler
• controllermanager
  was added in Add missing cluster metadata to multiple k8s metricsests #32979

metricsets:

• volume
• state_daemonset
• state_persistentvolume
• state_persistentvolumeclaim
  were fixed by Add metadata for missing k8s resources/metricsets #31590

event - was added #31341

state_storageclass - will be fixed by #33032

[emmanuelmathot]

great! Thank you!

[ottramst]

Hi!

Is it correct that for EKS based clusters the orchestrator.cluster.name field is not possible to add?