Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EBS Usage Metrics are not being emitted for some pods #1432

Closed
georgew5656 opened this issue Oct 21, 2022 · 3 comments
Closed

EBS Usage Metrics are not being emitted for some pods #1432

georgew5656 opened this issue Oct 21, 2022 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@georgew5656
Copy link

georgew5656 commented Oct 21, 2022

/kind bug

What happened?
I am trying to look at disk usage metrics emitted in #677 for my pods, and I can only see the metrics for some of my pods.

In the set of pods where the metrics are being published, I have a statefulset with a volumeClaimTemplate specified.

In the set of pods where the metrics are not being published, I am directly creating pods with an ephemeral section, e.g.

- ephemeral:
    volumeClaimTemplate:
     metadata:
      creationTimestamp: null
      labels:
       [app.kubernetes.io/managed-by](http://app.kubernetes.io/managed-by):
       [app.kubernetes.io/name](http://app.kubernetes.io/name):
     spec:
      accessModes:
      - ReadWriteOnce
      resources:
       requests:
        storage: 100Gi
      storageClassName: data-pod-gp3-sc
      volumeMode: Filesystem
   name: var

I don't see any relevant logs related to metric publishing on the controller pods, but I think I might be looking in the wrong place?

As far as differences between the two PVC's created, I can see that the one that is publishing metrics does have a app.kubernetes.io/instance label (created by the statefulset), while the one that isn't doesn't have the label (we don't specify it in the pod template).

The pod pvc (not publishing) also has a ownerReference section:
ownerReferences:

  • apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Pod
    name:
    uid:

What you expected to happen?
I expected both pods to get volume metrics published.

How to reproduce it (as minimally and precisely as possible)?
Create a statefulset with a ebs csi volumeClaimTemplate
Create a pod with a ephemeral ebs csi volumeClaimTemplate

Confirm that the first set of pods does get metrics published while the second one does not

Anything else we need to know?:
N/A

Environment

  • Kubernetes version (use kubectl version): 1.22
  • Driver version: 1.10.0
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 21, 2022
@torredil
Copy link
Member

This is likely related to Kubelet not reporting metrics for generic ephemeral volumes #107218 in K8s 1.22. Generic ephemeral volumes went GA in 1.23.

I can confirm that in K8s 1.25 these metrics are reported by the Kubelet:

  1. Create a pod with a generic ephemeral volume:
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "ebs-sc"
          resources:
            requests:
              storage: 5Gi
  1. Check Kubelet metrics endpoint:
$ kubectl get --raw /api/v1/nodes/i-0c7a5ce66bc55cd03/proxy/metrics | grep 'kubelet_volume_stats_capacity_bytes'

kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="app-persistent-storage"} 5.129273344e+09

@georgew5656
Copy link
Author

thanks! will leave a note when i get around to updating to 1.23

@georgew5656
Copy link
Author

this worked for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants