Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: BlockCount: error unpacking block: #8482

Closed
rwrocket opened this issue Jun 13, 2017 · 11 comments
Closed

panic: BlockCount: error unpacking block: #8482

rwrocket opened this issue Jun 13, 2017 · 11 comments
Labels
1.x

Comments

@rwrocket
Copy link

rwrocket commented Jun 13, 2017 •
edited by jwilder
Loading

I am running influxdb (I think latest stable release) on ubuntu 14.04 with grafana and fastnetmon.

Lately influx db seems to be crashing every day or every couple of days but sometimes a reboot of the server fixes it.

If I manually start influxdb it stops after about a minute currently.
graphite database is about 11gig so I suspect it is something to do with the database size and the query grafana is trying to make but I have no idea how to fix it,

See the log.

[run] 2017/06/13 12:26:43 InfluxDB starting, version 1.1.1, branch master, commit e47cf1f2e83a02443d7115c54f838be8ee959644
[run] 2017/06/13 12:26:43 Go version go1.7.4, GOMAXPROCS set to 4
[run] 2017/06/13 12:26:43 Using configuration at: /etc/influxdb/influxdb.conf
[store] 2017/06/13 12:26:43 Using data dir: /var/lib/influxdb/data
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/178/000000009-000000002.tsm (#0) opened in 874.273µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/178/_00047.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/177/000000009-000000002.tsm (#0) opened in 838.934µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/177/_00046.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/176/000000009-000000002.tsm (#0) opened in 911.847µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/176/_00046.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/99/000000115-000000002.tsm (#0) opened in 1.770477ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/99/_00367.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/178 database index loaded in 19.130924ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/178 opened in 21.27582ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/177 database index loaded in 24.731357ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/177 opened in 26.647244ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/176 database index loaded in 25.650049ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/180/000000009-000000002.tsm (#0) opened in 575.143µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/180/_00046.wal, size 0
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/176 opened in 27.769243ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/181/000000009-000000004.tsm (#0) opened in 1.198127ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/181/_00042.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/171/000000637-000000002.tsm (#0) opened in 4.353557ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/171/_01916.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/181 database index loaded in 12.198408ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/181 opened in 14.201666ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/180 database index loaded in 15.433317ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/180 opened in 16.575572ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/182/000000009-000000002.tsm (#0) opened in 651.061µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/182/_00044.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/183/000000009-000000002.tsm (#0) opened in 648.966µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/183/_00044.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/182 database index loaded in 12.914002ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/182 opened in 14.23512ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/184/000000007-000000001.tsm (#2) opened in 537.149µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/184/000000004-000000003.tsm (#0) opened in 771.606µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/184/000000006-000000002.tsm (#1) opened in 562.851µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/_internal/monitor/184/_00029.wal, size 10497027
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/183 database index loaded in 21.563287ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/_internal/monitor/183 opened in 22.798499ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/107/000000105-000000002.tsm (#0) opened in 1.247226ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/107/_00338.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/99 database index loaded in 104.827995ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/99 opened in 107.339209ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/11/000000040-000000003.tsm (#0) opened in 517.594µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/11/_00137.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/11 database index loaded in 14.193775ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/11 opened in 15.601984ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/107 database index loaded in 56.536589ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/107 opened in 58.757263ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/115/000000113-000000002.tsm (#0) opened in 1.366026ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/115/_00361.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/171 database index loaded in 92.469929ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/171 opened in 104.420052ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/123/000000121-000000002.tsm (#0) opened in 1.4784ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/123/_00385.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/147/000000158-000000003.tsm (#0) opened in 1.677378ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/147/_00495.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/115 database index loaded in 23.34836ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/115 opened in 25.380672ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/131/000000117-000000002.tsm (#0) opened in 1.414216ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/131/_00373.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/123 database index loaded in 26.762761ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/123 opened in 29.10852ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/139/000000129-000000002.tsm (#0) opened in 1.375664ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/139/_00408.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/147 database index loaded in 28.942018ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/147 opened in 31.444851ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/163/000000649-000000002.tsm (#0) opened in 6.625843ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/163/_01969.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/131 database index loaded in 25.189236ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/131 opened in 27.367168ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/155/000000356-000000004.tsm (#0) opened in 2.303575ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/155/_01089.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/139 database index loaded in 28.598678ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/139 opened in 30.667517ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/163 database index loaded in 21.230074ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/163 opened in 29.103352ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000499-000000004.tsm (#9) opened in 863.168µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000423-000000003.tsm (#2) opened in 810.089µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000419-000000005.tsm (#1) opened in 1.905759ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/43/000000073-000000002.tsm (#0) opened in 1.438312ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/43/_00240.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000427-000000003.tsm (#3) opened in 860.235µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000431-000000003.tsm (#4) opened in 594.559µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000435-000000003.tsm (#5) opened in 518.781µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000451-000000004.tsm (#6) opened in 905.144µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000467-000000004.tsm (#7) opened in 1.012489ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000483-000000004.tsm (#8) opened in 923.93µs
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm (#0) opened in 7.567861ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/179/_01489.wal, size 503388
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/155 database index loaded in 26.197326ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/155 opened in 29.378596ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/7/000000002-000000002.tsm (#0) opened in 182.146µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/7/_00032.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/7 database index loaded in 1.280471ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/7 opened in 2.27096ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/185/_00001.wal, size 29318
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/43 database index loaded in 26.995891ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/43 opened in 36.683854ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/185/_00003.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/19/000000044-000000004.tsm (#0) opened in 633.81µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/19/_00154.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/185 database index loaded in 13.190015ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/185 opened in 27.718189ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/19 database index loaded in 13.205171ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/19 opened in 14.876682ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/27/000000044-000000004.tsm (#0) opened in 648.895µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/27/_00153.wal, size 0
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/3/000000003-000000002.tsm (#0) opened in 110.838µs
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/3/_00038.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/3 database index loaded in 2.45108ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/3 opened in 3.680566ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/35/000000051-000000002.tsm (#0) opened in 4.406286ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/35/_00174.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/27 database index loaded in 17.704277ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/27 opened in 19.081337ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/83/000000110-000000003.tsm (#0) opened in 1.576388ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/83/_00351.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/35 database index loaded in 9.87444ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/35 opened in 18.130099ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/75/000000110-000000003.tsm (#0) opened in 1.551873ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/75/_00351.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/83 database index loaded in 21.204722ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/83 opened in 25.27172ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/91/000000117-000000002.tsm (#0) opened in 3.337925ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/91/_00374.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/75 database index loaded in 36.560445ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/75 opened in 38.957119ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/59/000000099-000000002.tsm (#0) opened in 1.50047ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/59/_00319.wal, size 0
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/179/_01497.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/91 database index loaded in 26.550443ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/91 opened in 31.842177ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/67/000000115-000000002.tsm (#0) opened in 1.508013ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/67/_00368.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/59 database index loaded in 26.062393ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/59 opened in 28.449777ms
[filestore] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/51/000000113-000000002.tsm (#0) opened in 1.357924ms
[cacheloader] 2017/06/13 12:26:43 reading file /var/lib/influxdb/wal/graphite/autogen/51/_00362.wal, size 0
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/51 database index loaded in 27.2601ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/51 opened in 29.427486ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/67 database index loaded in 43.822542ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/67 opened in 46.104606ms
[shard] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179 database index loaded in 165.120579ms
[store] 2017/06/13 12:26:43 /var/lib/influxdb/data/graphite/autogen/179 opened in 284.061471ms
[tsm1] 2017/06/13 12:26:43 beginning optimize compaction of group 0, 4 TSM files
[tsm1] 2017/06/13 12:26:43 compacting optimize group (0) /var/lib/influxdb/data/graphite/autogen/179/000000451-000000004.tsm (#0)
[tsm1] 2017/06/13 12:26:43 compacting optimize group (0) /var/lib/influxdb/data/graphite/autogen/179/000000467-000000004.tsm (#1)
[tsm1] 2017/06/13 12:26:43 compacting optimize group (0) /var/lib/influxdb/data/graphite/autogen/179/000000483-000000004.tsm (#2)
[tsm1] 2017/06/13 12:26:43 compacting optimize group (0) /var/lib/influxdb/data/graphite/autogen/179/000000499-000000004.tsm (#3)
[tsm1] 2017/06/13 12:26:43 beginning level 3 compaction of group 0, 4 TSM files
[tsm1] 2017/06/13 12:26:43 compacting level 3 group (0) /var/lib/influxdb/data/graphite/autogen/179/000000423-000000003.tsm (#0)
[tsm1] 2017/06/13 12:26:43 compacting level 3 group (0) /var/lib/influxdb/data/graphite/autogen/179/000000427-000000003.tsm (#1)
[tsm1] 2017/06/13 12:26:43 compacting level 3 group (0) /var/lib/influxdb/data/graphite/autogen/179/000000431-000000003.tsm (#2)
[tsm1] 2017/06/13 12:26:43 compacting level 3 group (0) /var/lib/influxdb/data/graphite/autogen/179/000000435-000000003.tsm (#3)
[cacheloader] 2017/06/13 12:26:45 reading file /var/lib/influxdb/wal/_internal/monitor/184/_00030.wal, size 1440268
[cacheloader] 2017/06/13 12:26:45 reading file /var/lib/influxdb/wal/_internal/monitor/184/_00038.wal, size 0
[shard] 2017/06/13 12:26:45 /var/lib/influxdb/data/_internal/monitor/184 database index loaded in 25.87431ms
[store] 2017/06/13 12:26:45 /var/lib/influxdb/data/_internal/monitor/184 opened in 2.017212105s
[subscriber] 2017/06/13 12:26:45 opened service
[monitor] 2017/06/13 12:26:45 Starting monitor system
[monitor] 2017/06/13 12:26:45 'build' registered for diagnostics monitoring
[monitor] 2017/06/13 12:26:45 'runtime' registered for diagnostics monitoring
[monitor] 2017/06/13 12:26:45 'network' registered for diagnostics monitoring
[monitor] 2017/06/13 12:26:45 'system' registered for diagnostics monitoring
[shard-precreation] 2017/06/13 12:26:45 Starting precreation service with check interval of 10m0s, advance period of 30m0s
[snapshot] 2017/06/13 12:26:45 Starting snapshot service
[admin] 2017/06/13 12:26:45 Starting admin service
[admin] 2017/06/13 12:26:45 DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release
[monitor] 2017/06/13 12:26:45 Storing statistics in database '_internal' retention policy 'monitor', at interval 10s
[admin] 2017/06/13 12:26:45 Listening on HTTP: [::]:8083
[continuous_querier] 2017/06/13 12:26:45 Starting continuous query service
[httpd] 2017/06/13 12:26:45 Starting HTTP service
[httpd] 2017/06/13 12:26:45 Authentication enabled: false
[httpd] 2017/06/13 12:26:45 Listening on HTTP: [::]:8086
[retention] 2017/06/13 12:26:45 Starting retention policy enforcement service with check interval of 30m0s
[graphite] 2017/06/13 12:26:45 Starting graphite service, batch size 5000, batch timeout 1s
[monitor] 2017/06/13 12:26:45 'graphite:tcp::2003' registered for diagnostics monitoring
[graphite] 2017/06/13 12:26:45 Listening on TCP: [::]:2003
[run] 2017/06/13 12:26:45 Listening for signals
2017/06/13 12:26:45 Sending usage statistics to usage.influxdata.com
panic: BlockCount: error unpacking block: unpackBlock: not enough data for timestamp

goroutine 221 [running]:
panic(0x99e160, 0xc421908040)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/influxdb/tsdb/engine/tsm1.BlockCount(0x7f1ceca9e527, 0x6aa, 0x8fc455, 0x1)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/encoding.go:203 +0x1c4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).combine(0xc4213c7b00, 0x10100c420024c00, 0x0, 0x200, 0xc4200d4c00)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1125 +0x6eb
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).merge(0xc4213c7b00)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1061 +0x100
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).Next(0xc4213c7b00, 0xc40042096b)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:937 +0xf38
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write(0xc420ca5540, 0xc42196d220, 0x47, 0xde3180, 0xc4213c7b00, 0x0, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:743 +0x29b
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles(0xc420ca5540, 0x1b3, 0x4, 0xde3180, 0xc4213c7b00, 0xde3180, 0xc4213c7b00, 0x0, 0x0, 0xc420452bf8)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:697 +0x284
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact(0xc420ca5540, 0xb73900, 0xc421822740, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:623 +0x52a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFull(0xc420ca5540, 0xc421822740, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:641 +0x18e
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup.func1(0xc4208ae000, 0xc421822740, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1057 +0x15d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup(0xc4208ae000, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1059 +0x40d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply.func1(0xc421bba180, 0xc4208ae000, 0x0)
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1032 +0x5d
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply
	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1033 +0xcc
@rwrocket
Copy link
Author

influxdb.zip

Influxdb conf file

@jwilder
Copy link
Contributor

jwilder commented Jun 13, 2017

Can you describe your storage setup? Are you using directly attached SSDs or is there RAID/NFS/SAN or anything else in the mix? If appears that one of your TSM files seems to have lost some of it's contents. Since TSM files are immutable after being written, this usually hints at a file system/hardware issue.

From the logs, it looks like it might be one of these 4 files:

/var/lib/influxdb/data/graphite/autogen/179/000000451-000000004.tsm
/var/lib/influxdb/data/graphite/autogen/179/000000467-000000004.tsm
/var/lib/influxdb/data/graphite/autogen/179/000000483-000000004.tsm
/var/lib/influxdb/data/graphite/autogen/179/000000499-000000004.tsm

Could your run influx_inspect verify -dir /var/lib/influxdb as well as
influx_inspect dumptsm -all /path/to/file for each of those 4 TSM files?

@rwrocket
Copy link
Author

Storage is a 3TB raid 5 array consisting of physical disks (not SSD but sata or older scsi I think struggling to remember)

It's possible there could be an issue with the raid I guess.

Here is the output of the command

/var/lib/influxdb/data$ sudo influx_inspect verify -dir /var/lib/influxdb
/var/lib/influxdb/data/_internal/monitor/176/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/177/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/178/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/180/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/181/000000009-000000004.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/182/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/183/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000004-000000003.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000006-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000007-000000001.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/107/000000105-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/11/000000040-000000003.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/115/000000113-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/123/000000121-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/131/000000117-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/139/000000129-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/147/000000158-000000003.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/155/000000356-000000004.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/163/000000649-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/171/000000637-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 201876207 but expected 654780389 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=bps#!#value, block 274581
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1236391794 but expected 2097455171 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=bps#!#value, block 274584
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 26638336 but expected 514071834 for key hosts,a pp=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274585
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3624074936 but expected 4197688196 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274586
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 32694529 but expected 3506966906 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274587
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 224583681 but expected 522428476 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274588
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2013274847 but expected 3821747874 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274589
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 153021448 but expected 3527127979 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274590
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 4072801545 but expected 415347498 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274591
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1107604792 but expected 1144644387 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274592
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2013265952 but expected 3897908089 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274593
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 67907679 but expected 3464833328 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274594
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 662365441 but expected 3535328451 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274595
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 852888566 but expected 2838865974 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274596
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1392717421 but expected 2556253546 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274597
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2322530582 but expected 1766954545 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274598
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3909713644 but expected 3896770092 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274599
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3959354630 but expected 458012330 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274600
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 11039051 but expected 2702577359 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274601
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2524446861 but expected 3215814139 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274602
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 301216330 but expected 3035443565 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274603
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3325654 but expected 920604844 for key hosts,ap p=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274604
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 923110135 but expected 3647417724 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274605
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1762275907 but expected 951266348 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274606
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 88694954 but expected 2697918662 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274607
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1300916481 but expected 3658340899 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274608
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 17660350 but expected 3060531528 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274609
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1652162773 but expected 1769098313 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274610
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 77717517 but expected 1950116332 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274611
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1208230544 but expected 1348551277 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274612
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2321613418 but expected 2400617811 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274613
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2598764808 but expected 243473601 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274614
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3584267989 but expected 3728941898 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274615
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3808507520 but expected 1955460445 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274616
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 27898366 but expected 2846658498 for key hosts, app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274617
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 3191428970 but expected 152471385 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274618
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 11594016 but expected 288338574 for key hosts,a pp=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274619
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1584070916 but expected 2306414884 for key host s,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274620
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2377463958 but expected 992588388 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274621
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 2013277719 but expected 787938410 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!#value, block 274622
/var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm: got 1778451629 but expected 885207105 for key hosts ,app=fastnetmon,cidr=103_27_175_3,direction=incoming,function=average,resource=flows#!~#value, block 274623
unexpected fault address 0x7f6ebc495002
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f6ebc495002 pc=0x5605d4]

goroutine 1 [running]:
runtime.throw(0x65132f, 0x5)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc42003b938 sp=0xc42003b918
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:21 +0xf3 fp=0xc42003b990 sp=0xc42003b938
hash/crc32.updateSlicingBy8(0x0, 0xc42014e000, 0x7f6ebc494eab, 0x39a, 0xeba0690, 0x0)
/usr/local/go/src/hash/crc32/crc32.go:148 +0x64 fp=0xc42003b9d0 sp=0xc42003b990
hash/crc32.updateIEEE(0x7f6e00000000, 0x7f6ebc494eab, 0x39a, 0xeba0690, 0x0)
/usr/local/go/src/hash/crc32/crc32_amd64.go:56 +0x156 fp=0xc42003ba18 sp=0xc42003b9d0
hash/crc32.ChecksumIEEE(0x7f6ebc494eab, 0x39a, 0xeba0690, 0x14c53576e8e08800)
/usr/local/go/src/hash/crc32/crc32.go:189 +0x47 fp=0xc42003ba50 sp=0xc42003ba18
github.com/influxdata/influxdb/cmd/influx_inspect/verify.(*Command).Run(0xc4200166e0, 0xc42000a120, 0x2, 0x2, 0x1, 0xc42 000a120)
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/verify/verify.go:90 +0x877 fp=0xc42003bd78 sp=0xc 42003ba50
main.(*Main).Run(0xc42003bf00, 0xc42000a120, 0x2, 0x2, 0x40520b, 0xc42006a058)
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:76 +0xa31 fp=0xc42003bea0 sp=0xc42003bd78
main.main()
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:21 +0x18d fp=0xc42003bf48 sp=0xc42003bea0
runtime.main()
/usr/local/go/src/runtime/proc.go:183 +0x1f4 fp=0xc42003bfa0 sp=0xc42003bf48
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42003bfa8 sp=0xc42003bfa0

@jwilder
Copy link
Contributor

jwilder commented Jun 14, 2017

Looks like /var/lib/influxdb/data/graphite/autogen/179/000000274-000000006.tsm is a problem file. You can move that file out of the directory and run the verify again until it completes and says everything is OK. I'm guessing you have other files that are corrupted though.

Unfortunately, I've seen several cases of RAIDed drives corrupting data files. Since the index is intact, (at the end of the file), but some of data blocks are failing checksums, this is very likely a hardware/filesystem issue where the block data was corrupted after being written. It looks like like some of the data in those files may be recoverable, but you will probably need to restore from backup or move the bad files out of the shard to get back up and running again.

@rwrocket
Copy link
Author

Ok I moved all of 179 out of the directory and ran verify and now I am getting this:

influx_inspect verify -dir /var/lib/influxdb
/var/lib/influxdb/data/_internal/monitor/176/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/177/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/178/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/180/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/181/000000009-000000004.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/182/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/183/000000009-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000004-000000003.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000006-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/184/000000007-000000001.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/107/000000105-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/11/000000040-000000003.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/115/000000113-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/123/000000121-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/131/000000117-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/139/000000129-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/147/000000158-000000003.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/155/000000356-000000004.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/163/000000649-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/171/000000637-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/19/000000044-000000004.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/27/000000044-000000004.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/3/000000003-000000002.tsm: healthy
/var/lib/influxdb/data/graphite/autogen/35/000000051-000000002.tsm: healthy
unexpected fault address 0x7f28e739d003
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f28e739d003 pc=0x5605d4]

goroutine 1 [running]:
runtime.throw(0x65132f, 0x5)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc42027a948 sp=0xc42027a928
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:21 +0xf3 fp=0xc42027a9a0 sp=0xc42027a948
hash/crc32.updateSlicingBy8(0x0, 0xc42012c000, 0x7f28e739ccbc, 0xa41, 0x3ba4ab7, 0x0)
/usr/local/go/src/hash/crc32/crc32.go:148 +0x64 fp=0xc42027a9e0 sp=0xc42027a9a0
hash/crc32.updateIEEE(0x7f2800000000, 0x7f28e739ccbc, 0xa41, 0x3ba4ab7, 0x0)
/usr/local/go/src/hash/crc32/crc32_amd64.go:56 +0x156 fp=0xc42027aa28 sp=0xc42027a9e0
hash/crc32.ChecksumIEEE(0x7f28e739ccbc, 0xa41, 0x3ba4ab7, 0x14a1d985dec05a00)
/usr/local/go/src/hash/crc32/crc32.go:189 +0x47 fp=0xc42027aa60 sp=0xc42027aa28
github.com/influxdata/influxdb/cmd/influx_inspect/verify.(*Command).Run(0xc420014760, 0xc42000a120, 0x2, 0x2, 0x1, 0xc42000a120)
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/verify/verify.go:90 +0x877 fp=0xc42027ad88 sp=0xc42027aa60
main.(*Main).Run(0xc42027af10, 0xc42000a120, 0x2, 0x2, 0x40520b, 0xc4200160b8)
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:76 +0xa31 fp=0xc42027aeb0 sp=0xc42027ad88
main.main()
/root/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:21 +0x18d fp=0xc42027af58 sp=0xc42027aeb0
runtime.main()
/usr/local/go/src/runtime/proc.go:183 +0x1f4 fp=0xc42027afb0 sp=0xc42027af58
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42027afb8 sp=0xc42027afb0

@rwrocket
Copy link
Author

Ok I just moved all the data files out of the data directory and started my service again and so far so good.
(historical data is not critical at this time and if so I have it sitting in another directory)
I will look into my RAID health.

Thanks for the help!

@twm
Copy link

twm commented Jan 22, 2018

I am experiencing a similar failure following a disk filling. The result is an influxd restart loop as it attempts to compact, hits the corrupt file, and fails.

Jan 21 18:03:12 aorist influxd[10643]: [I] 2018-01-22T02:03:12Z beginning level 3 compaction, 1 TSM files engine=tsm1
Jan 21 18:03:12 aorist influxd[10643]: [I] 2018-01-22T02:03:12Z compacting level 3 /var/lib/influxdb/data/collectd/autogen/211/000000027-000000003.tsm (#0) engine=tsm1
Jan 21 18:03:21 aorist influxd[10643]: panic: BlockCount: error unpacking block: unpackBlock: not enough data for timestamp
Jan 21 18:03:21 aorist influxd[10643]: goroutine 140 [running]:
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.BlockCount(0x7f1f77710b2f, 0x2a9a, 0xbf3b1d1, 0x688)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/encoding.go:244 +0x1ba
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).combineFloat(0xc42b9a82c0, 0xc42d2ec800, 0x1e8, 0xc4321197c0, 0x4451e4)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:108 +0x6c0
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).mergeFloat(0xc42b9a82c0)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:31 +0x14e
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).merge(0xc42b9a82c0)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1376 +0x3c
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmKeyIterator).Next(0xc42b9a82c0, 0x7f1f093edc93)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1367 +0x12f5
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write(0xc42098b260, 0xc421558000, 0x47, 0x10e3060, 0xc42b9a82c0, 0x0, 0x0)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1011 +0x13b
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles(0xc42098b260, 0x1b, 0x4, 0x10e3060, 0xc42b9a82c0, 0x1, 0x10e3060, 0xc42b9a82c0, 0x0, 0x0)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:942 +0x244
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact(0xc42098b260, 0xce9b01, 0xc428f20050, 0x1, 0x1, 0x1, 0x10e8160, 0xc420c5c4e0, 0xc427192000, 0x4c7f24)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:851 +0x5c9
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFast(0xc42098b260, 0xc428f20050, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:901 +0x18e
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup(0xc4207b4d00)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1550 +0x396
Jan 21 18:03:21 aorist influxd[10643]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply.func1(0xc42116e070, 0xc4207b4d00)
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1528 +0x51
Jan 21 18:03:21 aorist influxd[10643]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply
Jan 21 18:03:21 aorist influxd[10643]:         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1526 +0x9b
Jan 21 18:03:21 aorist systemd[1]: influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

I have InfluxDB version 1.4.2, branch 1.4, commit 6d2685d

It would be nice if influxdb did not crash in this scenario, as this prevents it from accepting writes.

@timhallinflux
Copy link
Contributor

@twm -- are you still experiencing this issue?

@twm
Copy link

twm commented Mar 31, 2018

I moved the corrupted file aside and InfluxDB was able to start successfully. That's about as much as I can expect from a single-host deployment, I think. It would have been nice if the failure message indicated that I could do this — I was not familiar with the on-disk layout at all, so I didn't realize that removing a file would simply drop the data within it until I found this issue.

@dgnorton dgnorton added the 1.x label Jan 7, 2019
@russorat
Copy link
Contributor

closing since you were able to work around this issue.

@dgnorton
Copy link
Contributor

Fixed in v1.8.0 by #17570

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1.x
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@twm @dgnorton @jwilder @russorat @timhallinflux @rwrocket