non-existing mountpoint issue in filesystem metrics with 0.15 release binaries

[tanji]

Host operating system: output of uname -a

Linux vbox-4 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

Issue version:

node_exporter, version 0.15.0 (branch: HEAD, revision: 6e2053c557f96efb63aef3691f15335a70baaffd)
  build user:       root@168089f37ad9
  build date:       20171006-11:33:58
  go version:       go1.9.1

Working version:

node_exporter, version 0.15.0 (branch: master, revision: 0eecaa9547af380667a6e83cc3cebc224525e7f5)
  build user:       tanj@thinkpad
  build date:       20171030-12:42:06
  go version:       go1.9.1

node_exporter command line flags

/opt/node_exporter/bin/node_exporter --collector.textfile.directory /opt/node_exporter/textfile_collector --collector.diskstats.ignored-devices '^(ram|dm|rootfs|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$' --collector.filesystem.ignored-fs-types '^(sys|proc|rootfs|rpc_pipefs|tmpfs)fs$' --web.listen-address :9100 --web.telemetry-path '/metrics'

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

When I use the linux-amd64 binary downloaded on Github, non-existent mountpoints (/tmp and /var/tmp) appear in my filesystem metrics:

# TYPE node_filesystem_avail gauge
node_filesystem_avail{device="/dev/md1",fstype="ext3",mountpoint="/boot"} 4.48186368e+08
node_filesystem_avail{device="/dev/md2",fstype="ext4",mountpoint="/"} 2.691158016e+10
node_filesystem_avail{device="/dev/md2",fstype="ext4",mountpoint="/tmp"} 2.691158016e+10
node_filesystem_avail{device="/dev/md2",fstype="ext4",mountpoint="/var/tmp"} 2.691158016e+10
node_filesystem_avail{device="rootfs",fstype="rootfs",mountpoint="/"} 2.691158016e+10
node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.2169715712e+10
node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/lock"} 5.24288e+06

If I compile the binary on my workstation using make and run it on the server, it works fine:

# TYPE node_filesystem_avail gauge
node_filesystem_avail{device="/dev/md1",fstype="ext3",mountpoint="/boot"} 4.48186368e+08
node_filesystem_avail{device="/dev/md2",fstype="ext4",mountpoint="/"} 2.6911551488e+10
node_filesystem_avail{device="rootfs",fstype="rootfs",mountpoint="/"} 2.6911551488e+10
node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.2169719808e+10
node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/lock"} 5.24288e+06
node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 6.75203072e+09

In addition, with my compiled binary, I get the following error:

ERRO[0009] error gathering metrics: [from Gatherer #1] collected metric node_textfile_scrape_error gauge:<value:0 >  was collected before with the same name and label values
  source="<autogenerated>:1"

[SuperQ]

It looks like your build is a few commits past the 0.15.0 release. But looking at the changes between the two commits , I don't see anything that could be causing the problem.

Since this is coming from the filesystem collector, could you attach a copy of /proc/mounts?

[SuperQ]

Also, the textfile gatherer error is reported in #704.

[tanji]

Roger about the #704. I've read the changes as well and didn't find a related change.

As you can see /proc/mounts doesn't have any /tmp or /var/tmp endpoints.

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,relatime 0 0
udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=8239822,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=13187556k,mode=755 0 0
/dev/md2 / ext4 rw,relatime,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
/dev/md1 /boot ext3 rw,relatime,stripe=4,data=ordered 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=6593780k,mode=700 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

[SuperQ]

Are you running the node_exporter under a systemd unit? Can you share this? I'm wondering if there's some fakery going on in the cgroup namespace. Otherwise I have no idea how the node_exporter could produce that data.

[tanji]

Ah, great call! That's it. I used the systemd unit from this repo: https://github.com/idealista/prometheus_server-role/blob/master/templates/prometheus.service.j2
And indeed it has PrivateTmp=yes, which can cause all kinds of issues.
And in the case where I used the compiled binary, I didn't start it with systemd.
I'll go ahead and close this issue for now as that can be easily solved by turning off PrivateTmp in the unit file. Thanks!

[SuperQ]

There's no need for PrivateTmp=yes as the node_exporter writes no temp files. Probably good to send them a PR to remove that.