[disk] Negative free disk space values stored to InfluxDB

[melor]

Bug report

Series of negative free disk space spikes observed on some Linux nodes.

System info:

• telegraf 1.0.0
• influxdb 1.0.0
• Linux n7495 4.7.10-100.fc23.x86_64 clarifying config file #1 SMP Wed Oct 26 23:29:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Expected behavior:

Correct disk free values expected.

Actual behavior:

Sample of raw data from InfluxDB:

   {'free': 8263704576, 'time': '2016-11-15T21:06:00Z'},
   {'free': 8263622656, 'time': '2016-11-15T21:06:30Z'},
   {'free': 8263540736, 'time': '2016-11-15T21:07:00Z'},
   {'free': 8263458816, 'time': '2016-11-15T21:07:30Z'},
   {'free': 8263380992, 'time': '2016-11-15T21:08:00Z'},
   {'free': 8263299072, 'time': '2016-11-15T21:08:30Z'},
   {'free': -8263380992, 'time': '2016-11-15T21:09:00Z'},
   {'free': 8263462911, 'time': '2016-11-15T21:09:30Z'},
   {'free': -8263544831, 'time': '2016-11-15T21:10:00Z'},
   {'free': 8263626750, 'time': '2016-11-15T21:10:30Z'},
   {'free': -8263708670, 'time': '2016-11-15T21:11:00Z'},
   {'free': 8263790589, 'time': '2016-11-15T21:11:30Z'},
   {'free': -8263872509, 'time': '2016-11-15T21:12:00Z'},
   {'free': 8262643712, 'time': '2016-11-15T21:12:30Z'},
   {'free': 8262561792, 'time': '2016-11-15T21:13:00Z'},
   {'free': 8262483968, 'time': '2016-11-15T21:13:30Z'},
   {'free': 8262402048, 'time': '2016-11-15T21:14:00Z'},
   {'free': 8262320128, 'time': '2016-11-15T21:14:30Z'},
   {'free': 8262238208, 'time': '2016-11-15T21:15:00Z'},
   {'free': 8262156288, 'time': '2016-11-15T21:15:30Z'},
   {'free': 8262074368, 'time': '2016-11-15T21:16:00Z'},

Additional info:

Divide the disk free bytes values by 4096 and a pattern emerges. Note that in this series the first negative value of each spike series is -previous_value - 20.

2016569
2016549
2016529
2016510
2016490
2016470
-2016490
2.01651e+06
-2.01653e+06
2.01655e+06
-2.01657e+06
2.01659e+06
-2.01661e+06
2016311
-2016331
2.01635e+06
-2.01637e+06
2.01639e+06
-2.01641e+06
2.01643e+06
-2.01645e+06
2016151
2016130
2016110
2016091
2016071
2016051
2016031
2016011
2015991
2015971
2015951
2015931
2015911
2015892
2015872
2015852
2015832
-2015852
2.01587e+06
-2.01589e+06
2.01591e+06
-2.01593e+06
2.01595e+06
-2.01597e+06
2015672
2015652
2015633
2015613
2015593
2015573
2015553
2015533
2015513
-2015533
2.01555e+06
-2.01557e+06
2.01559e+06
-2.01561e+06
2.01563e+06
2015373
2015353
2015334
2015313
2015293
2015273
2015253

I have not yet verified the actual input going from Telegraf to InfluxDB to confirm that this is not an InfluxDB issue.

I tried reviewing the relevant code, but wasn't able to spot any obvious place where the value could turn negative:

• telegraf/internal/models/makemetric.go

  Line 114 in b00ad65

  fields[k] = int64(val)

• telegraf/plugins/inputs/system/disk.go

  Line 66 in 6648c10

  "free": du.Free,

• telegraf/plugins/inputs/system/ps.go

  Line 86 in 6648c10

  du, err := disk.Usage(mountpoint)

• https://github.com/shirou/gopsutil/blob/master/disk/disk.go#L19
• https://github.com/shirou/gopsutil/blob/master/disk/disk_unix.go#L19

[sparrc]

we are simply using the value returned from https://github.com/shirou/gopsutil, could you open a corresponding issue there as well?

could you also provide more OS information? which distribution are you using? can you test and see if df ever gives you negative values?

[melor]

I got back to investigating this further and it seems the input from telegraf to influxdb is correct (no negative values), yet there are negative values returned from influxdb when querying the raw data. Could be related to influxdata/influxdb#6643

Closing this ticket and continuing on the influxdb side.

[melor]

I upgraded InfluxDB from 1.0 to 1.1 and haven't seen this issue since.