node_filesystem_device_error in v0.15.0 yet v0.14.0 works #711

tmegow · 2017-10-23T14:47:57Z

Host operating system: output of `uname -a`

Linux mongodb-backups 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux

node_exporter version: output of `node_exporter --version`

node_exporter, version 0.15.0 (branch: HEAD, revision: 6e2053c557f96efb63aef3691f15335a70baaffd)
  build user:       root@168089f37ad9
  build date:       20171006-11:33:58
  go version:       go1.9.1

node_exporter command line flags

--path.procfs /host/proc
--path.sysfs /host/sys
--collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

Are you running node_exporter in Docker?

yes, with the recommended args

docker run
     --log-driver none
     --name node-exporter
     --net host
     --pid host
     --volume /proc:/host/proc
     --volume /sys:/host/sys
     --volume /:/rootfs
     --volume /etc/node-exporter:/etc/node-exporter
     prom/node-exporter
         --path.procfs /host/proc
         --path.sysfs /host/sys
         --collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

What did you do that produced an error?

Trying to access exported disk metrics for k8s attached/mounted GCE persistent disk shows a node-exporter error metric incremented when node-exporter experienced an error while collecting statistics for this device.

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1

What did you expect to see?

When using v0.14.0, I can see the filesystem metrics for that device/mountpoint (seen below). Yet the new version v0.15.0 does not export the same metrics.

docker run
     --log-driver none
     --name node-exporter
     --net host
     --pid host
     --volume /proc:/host/proc
     --volume /sys:/host/sys
     --volume /:/rootfs
     --volume /etc/node-exporter:/etc/node-exporter prom/node-exporter:v0.14.0
         -collector.procfs /host/proc
         -collector.sysfs /host/sys
         -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_avail{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 9.3990998016e+10
node_filesystem_avail{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 9.3990998016e+10
node_filesystem_files{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.048576e+08
node_filesystem_files{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.048576e+08
node_filesystem_files_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.04857058e+08
node_filesystem_files_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.04857058e+08
node_filesystem_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 9.3990998016e+10
node_filesystem_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 9.3990998016e+10
node_filesystem_readonly{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 0
node_filesystem_readonly{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 0
node_filesystem_size{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.073217536e+11
node_filesystem_size{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.073217536e+11

What did you see instead?

A metric implying that node-exporter encountered an error while collecting statistics for this device.

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1

The text was updated successfully, but these errors were encountered:

tmegow · 2017-10-23T15:20:02Z

One more thing to note, with v0.15.0 this symptom is happening with all GCE PD mounts in our kube clusters (even on our Google Container-Optimized OS instances).

Linux minion-47pw 4.4.21+ #1 SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

# cat /etc/*release
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu-signed-mpkeys
CHROMEOS_RELEASE_BRANCH_NUMBER=84
CHROMEOS_RELEASE_BUILDER_PATH=lakitu-release/R56-9000.84.2
CHROMEOS_RELEASE_BUILD_NUMBER=9000
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=56
CHROMEOS_RELEASE_DESCRIPTION=9000.84.2 (Official Build) stable-channel lakitu
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=2
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=9000.84.2
DEVICETYPE=OTHER
GOOGLE_RELEASE=9000.84.2
HWID_OVERRIDE=LAKITU DEFAULT
BUILD_ID=9000.84.2
NAME="Container-Optimized OS"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=56
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=56
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

discordianfish · 2017-10-25T18:59:53Z

Thanks for the detailed report! Would be most interested in the log output of the node-exporter. Can you provide that too?

tmegow · 2017-10-25T21:46:39Z

Sure, here are the logs for v0.15.0

time="2017-10-25T21:41:47Z" level=info msg="Starting node_exporter (version=0.15.0, branch=HEAD, revision=6e2053c557f96efb63aef3691f15335a70baaffd)" source="node_exporter.go:43"
time="2017-10-25T21:41:47Z" level=info msg="Build context (go=go1.9.1, user=root@168089f37ad9, date=20171006-11:33:58)" source="node_exporter.go:44"
time="2017-10-25T21:41:47Z" level=info msg="Enabled collectors:" source="node_exporter.go:50"
time="2017-10-25T21:41:47Z" level=info msg=" - time" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - conntrack" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - infiniband" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - entropy" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - cpu" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - edac" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - wifi" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - vmstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - netstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - loadavg" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - bcache" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - sockstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - zfs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - timex" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - mdadm" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - meminfo" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - netdev" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - uname" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - filefd" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - filesystem" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - xfs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - arp" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - hwmon" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - ipvs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - stat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - textfile" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - diskstats" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg="Listening on :9100" source="node_exporter.go:76"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/5c1a8db44efd276057e1ec62d10e130aa1f9e1685fca1d9ecde0821f09d117ec/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/proc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev/pts\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev/mqueue\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/pids\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/perf_event\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/freezer\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/cpuset\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/net_cls,net_prio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="ERROR: timex collector failed after 0.000441s: failed to retrieve adjtimex stats: operation not permitted" source="collector.go:123"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/cpu,cpuacct\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/memory\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/blkio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/devices\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/hugetlb\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/shm\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/pts\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/mqueue\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/hugepages\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/proc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/kernel/security\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/pids\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/perf_event\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/freezer\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/cpuset\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/net_cls,net_prio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/cpu,cpuacct\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/memory\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/blkio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/devices\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/hugetlb\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/pstore\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/selinux\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/kernel/debug\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/tmp\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/etc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/home\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/run/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/containers/3f40c3188fa9c87ae7c5f5ac3031954e59f1213a098b1d592a0a0e41cc33b769/shm\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/containers/ac1e9ee6756fe150674431125ebea644650f3fcea0370f3bf830ebfbebdd7c0b/shm\": permission denied" source="filesystem_linux.go:57"

In contrast, v0.14.0 has fewer permission denied errors:

time="2017-10-25T21:45:12Z" level=info msg="Starting node_exporter (version=0.14.0, branch=master, revision=840ba5dcc71a084a3bc63cb6063003c1f94435a6)" source="node_exporter.go:140"
time="2017-10-25T21:45:12Z" level=info msg="Build context (go=go1.7.5, user=root@bb6d0678e7f3, date=20170321-12:12:54)" source="node_exporter.go:141"
time="2017-10-25T21:45:12Z" level=info msg="Enabled collectors:" source="node_exporter.go:160"
time="2017-10-25T21:45:12Z" level=info msg=" - diskstats" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - infiniband" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - loadavg" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - stat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - wifi" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - filesystem" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - netdev" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - textfile" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - uname" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - vmstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - entropy" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - edac" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - filefd" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - sockstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - zfs" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - time" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - conntrack" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - hwmon" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - mdadm" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - meminfo" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - netstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg="Listening on :9100" source="node_exporter.go:186"

discordianfish · 2017-10-26T09:05:52Z

Thanks! This looks like the same as #703, lets discuss over there.

discordianfish · 2017-10-26T09:25:43Z

Okay so the root cause is the same: We're dropping privileges to nobody now, which means you can't statsfs() the bind-mounted mountpoints. If you want to monitor the mongodb-backups volume you should mount it to the node-exporter pod. Then you also need to ignore all of these failing mountpoints, e.g:

      args:
          - --collector.filesystem.ignored-mount-points
          - '^\/rootfs\/(var\/lib|run\/docker)\/.*'

If you don't want to mount the volume to the node-exporter pod, you can of course also force it to run as root like described here: #599 (comment)

discordianfish added the require/feedback label Oct 25, 2017

discordianfish closed this as completed Oct 26, 2017

discordianfish reopened this Oct 26, 2017

discordianfish closed this as completed Oct 26, 2017

sradnev mentioned this issue Nov 10, 2017

Permission denied on statfs() vegasbrianc/prometheus#66

Closed

xiaoping378 mentioned this issue Jan 19, 2018

docker-compose up met ERROR: timex collector failed twang2218/gitlab-ce-zh#45

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

node_filesystem_device_error in v0.15.0 yet v0.14.0 works #711

node_filesystem_device_error in v0.15.0 yet v0.14.0 works #711

tmegow commented Oct 23, 2017 •

edited

Loading

tmegow commented Oct 23, 2017 •

edited

Loading

discordianfish commented Oct 25, 2017

tmegow commented Oct 25, 2017

discordianfish commented Oct 26, 2017

discordianfish commented Oct 26, 2017

node_filesystem_device_error in v0.15.0 yet v0.14.0 works #711

node_filesystem_device_error in v0.15.0 yet v0.14.0 works #711

Comments

tmegow commented Oct 23, 2017 • edited Loading

Host operating system: output of uname -a

node_exporter version: output of node_exporter --version

node_exporter command line flags

Are you running node_exporter in Docker?

What did you do that produced an error?

What did you expect to see?

What did you see instead?

tmegow commented Oct 23, 2017 • edited Loading

discordianfish commented Oct 25, 2017

tmegow commented Oct 25, 2017

discordianfish commented Oct 26, 2017

discordianfish commented Oct 26, 2017

tmegow commented Oct 23, 2017 •

edited

Loading

Host operating system: output of `uname -a`

node_exporter version: output of `node_exporter --version`

tmegow commented Oct 23, 2017 •

edited

Loading