Add disk usage prometheus metrics to jupyterhub-home-nfs deployment #8
Conversation
The disk usage prometheus metrics are exported by a sidecar container running prometheus node-exporter.
sunu
changed the title
|
@sgibson91 @yuvipanda Any thoughts on including the Prometheus exporter for disk space monitoring as part of If this looks ok, I will tweak the configurations to make sure the reported metrics will be picked up by the relevant |
|
@sunu This looks fine to me, it looks like you've made it an opt-in feature too which would've been my comment! I'm happy if @yuvipanda is happy? |
| args: | ||
| - --collector.disable-defaults | ||
| - --path.rootfs=/export | ||
| - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/) |
There was a problem hiding this comment.
Can you add a comment here explaining why these are excluded?
|
Absolutely love it, this is the way to go! |
|
I also opened #9 |
The dashboard definition in jupyterhub/grafana-dashboards expects the NFS home directory disk to be mounted at /shared-volume and the deployment to have the label `component: shared-volume-metrics`
|
@yuvipanda Looking at adding dirsize-exporter as a sidecar - we need to add the label Should we add a separate dashboard to |
|
@sunu ah, let's then leave dirsize-exporter out of this right now and deal with it separately :) I would like it to have the same component label though. |
There was a problem hiding this comment.
Looks good! But when we deploy this in the 2i2c-org/infrastructure repo, we should disable the prometheus exporter (https://github.com/2i2c-org/infrastructure/blob/main/helm-charts/basehub/templates/home-space-reporter.yaml) that's there for hubs that use this.
@yuvipanda, sounds good! That prometheus exporter is already disabled on the VEDA hubs (https://github.com/2i2c-org/infrastructure/blob/b897a9e57df5825055fee618f403ea0db4e71b72/config/clusters/nasa-veda/common.values.yaml#L4-L5) and all other hubs that use EFS. |
The disk usage prometheus metrics are exported by a sidecar container running prometheus node-exporter.
related to #7 and 2i2c-org/infrastructure#5062
Here's an example of a Grafana dashboard monitoring home directory disk usage through the exported Prometheus metrics: