error compacting TSM files: unknown block type

[guohaospace]

system

#cat /etc/redhat-release
CentOS release 6.4 (Final)
#uname -a
Linux xxx 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
#df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
ext4 45G 9.2G 34G 22% /

tail -100 /var/log/influxdb/influxd.log

[I] 2018-01-19T07:01:34Z beginning level 1 compaction, 3 TSM files engine=tsm1
[I] 2018-01-19T07:01:34Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000003-000000001.tsm (#0) engine=tsm1
[I] 2018-01-19T07:01:34Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000004-000000001.tsm (#1) engine=tsm1
[I] 2018-01-19T07:01:34Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000005-000000001.tsm (#2) engine=tsm1
[I] 2018-01-19T07:01:34Z error compacting TSM files: unknown block type: 45 engine=tsm1
[I] 2018-01-19T07:01:36Z beginning level 1 compaction, 3 TSM files engine=tsm1
[I] 2018-01-19T07:01:36Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000003-000000001.tsm (#0) engine=tsm1
[I] 2018-01-19T07:01:36Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000004-000000001.tsm (#1) engine=tsm1
[I] 2018-01-19T07:01:36Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000005-000000001.tsm (#2) engine=tsm1
[I] 2018-01-19T07:01:36Z error compacting TSM files: unknown block type: 45 engine=tsm1
[I] 2018-01-19T07:01:37Z beginning level 1 compaction, 3 TSM files engine=tsm1
[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000003-000000001.tsm (#0) engine=tsm1
[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000004-000000001.tsm (#1) engine=tsm1
[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000005-000000001.tsm (#2) engine=tsm1
[I] 2018-01-19T07:01:37Z error compacting TSM files: unknown block type: 45 engine=tsm1

help

Is this error affecting?，what should I do？

[jwilder]

How are your disks mounted? Are they directly attached SSDs, mounted over a NAS or is there a RAID involved? It looks like one of the files in /var/lib/influxdb/data/_internal/monitor/23 has a bad block. Since TSM files are immutable once written, this can occur because of hardware or filesystem issues.

It's not clear which specific file is problematic, but it is likely one of these 3:

[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000003-000000001.tsm (#0) engine=tsm1
[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000004-000000001.tsm (#1) engine=tsm1
[I] 2018-01-19T07:01:37Z compacting level 1 /var/lib/influxdb/data/_internal/monitor/23/000000005-000000001.tsm (#2) engine=tsm1

You can move those files out the directory and the error should go away.

[guohaospace]

The disk is broken ，thank you。

[tablatronix]

FYI influxdb will not recover from this at all it will just keep at it over and over killing your cpu, you have to remove the files afaik. fun