[10.3] runtime error: slice bounds out of range => data corrupted

[max3163]

Hi,

This afternoon, ur influxdb database ceased operating with this error :

panic: runtime error: slice bounds out of range

goroutine 2979019 [running]:
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc21bc71220, 0x7fd4bd78d6ab, 0xd1ca5, 0xd1cad, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:651 +0x451
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc21d2fa9b0, 0x0, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:887 +0x4dd

Impossible to restart the application, when the corrupted data loaded, influxdb crash again.
The only solution, delete in data and wall directory the corresponding file....

[filestore]2016/03/18 14:05:20 /tech/influxdb/var/lib/influxdb/data/qualif/one_week/125/000000018-000000001.tsm (#1) opened in 1.03395ms
[filestore]2016/03/18 14:05:20 /tech/influxdb/var/lib/influxdb/data/qualif/one_week/125/000000017-000000004.tsm (#0) opened in 2.722464ms
[cacheloader] 2016/03/18 14:05:20 reading file /tech/influxdb/var/lib/influxdb/wal/qualif/one_week/125/_00020.wal, size 0
[tsm1wal] 2016/03/18 14:05:20 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/03/18 14:05:20 tsm1 WAL writing to /tech/influxdb/var/lib/influxdb/wal/qualif/one_week/127
panic: runtime error: slice bounds out of range

goroutine 332 [running]:
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc2091c4320, 0x7f08701366ab, 0xd1ca5, 0xd1cad, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:651 +0x451
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc20b72c280, 0x0, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:887 +0x4dd
github.com/influxdb/influxdb/tsdb/engine/tsm1.NewTSMReaderWithOptions(0x0, 0x0, 0xc20a3db2d8, 0x5, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:154 +0x2f4
github.com/influxdb/influxdb/tsdb/engine/tsm1.func·009(0x5, 0xc20a3db2d8)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:259 +0xa9
created by github.com/influxdb/influxdb/tsdb/engine/tsm1.(*FileStore).Open
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:269 +0x757

[jsternberg]

@jwilder

[joelegasse]

@max3163 Could you describe what kind of data you were writing in to the database? Did the server crash while running, or was after a clean shutdown and restart?

Is this reproducible? What library/script are you using to populate the database? How many measurements/series were you creating? How many fields and of what types were you creating for each series?

Would you be able to upload/share an archive of your influxdb directory?

This appears to be the same error in #6022, but we haven't gotten any further detail from that issue. Any information you can provide to help us diagnose or reproduce this would be very helpful and appreciated.

[max3163]

Hi,

Assuming the crash come from the rp : one_week ( like I could see in the log ) I wrote only log data ( so string data ) in this database.

The server crash while running. And after a restart, the server crash too, when the corrupted data loaded.

I can't reproduce it.
I use a java programm who parse log and use java http api to populate the database.
Only one measurement with only one field but many tags ( 4 or 5 )

I think it's possible that a special caractere was sent and crash / corrupt the database but I can't know which one.

I don't have saved the corrupted data file so I can't upload/share the archive to you...

Hope I've helped you.

[joelegasse]

Thank you for providing what information you could. I'll continue to track this via #6022