node_exporter can not access persistent volume mounts

[juris]

Image I'm using:
AMI ID: ami-0c1942e28f6bfa4bd
bottlerocket-aws-k8s-1.18-x86_64-v1.0.4-cef8dbd2

What I expected to happen:
node_exporter should be able to access /local/var/lib/kubelet to get mountpoint stats

What actually happened:
Can not scrape metrics with node_exporter (version 1.0.1) due to:

msg="Error on statfs() system call" rootfs=/host/root/var/lib/kubelet/pods/3d764d0f-7867-41d9-93e7-0306d01f82d4/volume-subpaths/graylog-data-01/graylog/0 err="permission denied"

Looks like due to strict permissions (drwxr-x---) on /local/var/lib/kubelet and /local/var/lib/kubelet/pods node_exporter can not properly scrape such metrics as node_filesystem_avail_bytes

Running node_exporter as root helps though. But runnig containers as root user is not the right thing to do, isn't it?

How to reproduce the problem:
Create workload with persistent volume, launch node_exporter to get available_bytes metric for this volume.

[arnaldo2792]

Hi @juris , thanks for the issue!

As you mentioned, the /var/lib/kubelet directory is protected. It contains sensitive information such as kubelet credentials and certificates. If a workload needs access to that sensitive material, it’s appropriate to raise its access level to reflect that.

If you don’t need to get filesystem metrics about that directory, you could use the node_exporter’s collector.filesystem.ignored-mount-points option and pass /var/lib/kubelet. They already ignore some directories like /var/lib/docker by default.

containers:
  - name: node-exporter
    image: quay.io/prometheus/node-exporter:latest
    args:
      - --path.rootfs=/host-root
      - --log.level=debug
      - --collector.disable-defaults
      - --collector.filesystem
      - --collector.filesystem.ignored-mount-points=var/lib/kubelet

[juris]

In fact, I do not need fs metrics for exactly this this directory. I need to monitor PV usage, and all the persistent volumes are mounted under /var/lib/kubelet. So there are no other options until there is EBS CSI a release with this PR.