Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.10.0_nightly_308cab0-1] panic: runtime error: index out of range #5202

Closed
ivanscattergood opened this issue Dec 22, 2015 · 14 comments
Closed

[0.10.0_nightly_308cab0-1] panic: runtime error: index out of range #5202

ivanscattergood opened this issue Dec 22, 2015 · 14 comments
Assignees
@jwilder
Labels
area/tsm panic

Comments

@ivanscattergood
Copy link 

[cacheloader] 2015/12/22 18:17:41 reading file /var/lib/influxdb/wal/open_access_db/detail/14/_02984.wal, size 0
[tsm1wal] 2015/12/22 18:26:19 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2015/12/22 18:26:19 tsm1 WAL writing to /var/lib/influxdb/wal/open_access_db/detail/16
panic: runtime error: index out of range

goroutine 23 [running]:
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc9b03dd040, 0x7f9cf7950b0c, 0x8403003e, 0x84030046, 0x0, 0x0)
        /tmp/tmp.MmCxSXUQt9/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:614 +0xd03
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xccfd040460, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.MmCxSXUQt9/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:842 +0x57c
github.com/influxdb/influxdb/tsdb/engine/tsm1.NewTSMReaderWithOptions(0x0, 0x0, 0xc9b09a32f0, 0xece0b87ae, 0x0, 0x0)
        /tmp/tmp.MmCxSXUQt9/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:139 +0x2d8
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*FileStore).Open.func1(0xcad77e6af0, 0xcabf598a20, 0x3, 0xc9b09a32f0)
        /tmp/tmp.MmCxSXUQt9/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:251 +0x8f
created by github.com/influxdb/influxdb/tsdb/engine/tsm1.(*FileStore).Open
        /tmp/tmp.MmCxSXUQt9/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:261 +0x6bb
@beckettsean
Copy link
Contributor

@ivanscattergood can you give some context as to what was happening? Was this during startup? Did this happen during normal load? Does it recover on restart?

@ivanscattergood
Copy link
Author

Hi,

Sorry, I realise now in isolation that error message is useless!

I have been running the 0.9.6.1 binary for just about a week and saving collecting 0.5 million datapoints (over 1 million series) every 10 minutes.

I wanted to upgrade to the nightly after advice from @pauldix (in ticket #4977). When I upgraded to the new binary, this was the error I got on startup.

I have rolled back to 0.9.6.1 and it works fine.

Will I need to start with a Fresh database to use the nightly?

@otoolep
Copy link
Contributor

otoolep commented Dec 23, 2015

@jwilder -- any thoughts?

@jwilder
Copy link
Contributor

jwilder commented Dec 23, 2015

If you have existing tsm1 files created by the tsm1 engine in 0.9.6.1, you will need to start with a fresh DB to use the nightlies. The tsm1 engine in 0.9.6.1 uses a different file format than the current nightly version which is likely causing the panic you are seeing.

@otoolep otoolep closed this as completed Dec 23, 2015
@otoolep
Copy link
Contributor

otoolep commented Dec 23, 2015

Please wipe your tsm database and try again.

@pauldix
Copy link
Member

pauldix commented Dec 23, 2015

I think there's some confusion here. There shouldn't be any breaking changes between the 0.9.6.1 TSM format and master. Would you be able to send us the TSM file that it's crashing on?

@pauldix pauldix reopened this Dec 23, 2015
@jwilder
Copy link
Contributor

jwilder commented Dec 23, 2015

@ivanscattergood Also, could you attach the output of an ls on your shards for that DB? I'd like to see the number and sizes of the TSM files in each shard.

@jwilder jwilder self-assigned this Dec 23, 2015
@ivanscattergood
Copy link
Author

The raw data is getting saved in a shard called detail:

[root@visual-edge detail]# pwd
/ext_data/influxdb/data/open_access_db/detail

[root@visual-edge detail]# ls -ltrah
total 12G
drwx------. 4 influxdb influxdb 4.0K Dec 11 00:00 ..
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 14 00:03 7
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 15 00:05 9
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 16 00:05 12
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 17 00:12 14
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 18 16:59 16
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 19 20:28 18
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 20 00:07 20
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 21 00:15 22
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 21 23:25 27
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 22 00:17 24
drwx------. 14 influxdb influxdb 4.0K Dec 23 00:00 .
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 23 01:02 29
-rw-r--r--. 1 influxdb influxdb 2.0G Dec 23 18:32 2
-rw-r--r--. 1 influxdb influxdb 10G Dec 23 19:31 4
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 23 22:38 31

And the offending directory 16:
total 9.3G
-rw-r--r--. 1 influxdb influxdb 2.1G Dec 16 09:45 000000143-000000004.tsm
-rw-r--r--. 1 influxdb influxdb 2.2G Dec 16 18:29 000000302-000000004.tsm
-rw-r--r--. 1 influxdb influxdb 2.2G Dec 16 22:35 000000377-000000004.tsm
-rw-r--r--. 1 influxdb influxdb 2.9G Dec 18 16:59 000000400-000000003.tsm
drwxr-xr-x. 2 influxdb influxdb 4.0K Dec 18 16:59 .
drwx------. 14 influxdb influxdb 4.0K Dec 23 00:00 ..

Which TSM file would you like to see?

@ivanscattergood
Copy link
Author

I cannot load the TSM files via this interface as the largest upload size is 10mb, is there another way I can send you these files?

jwilder added a commit that referenced this issue Jan 4, 2016
@jwilder
Fix panic: runtime error: slice bounds out of range
7794b9c 
The block count was an uint16 when incrementing the index location
which was an int32.  This caused the value the uint16 value to overflow
before the index location was incremented causing the wrong location
to be read on the next iteration of the loop.  This triggers the slice
out of range errors.

Added a test that recreates the panic seen in #5257 and possibly #5202 which
is older code.

Fixes #5257
@jwilder jwilder mentioned this issue Jan 4, 2016
Merged
@jwilder
Copy link
Contributor

jwilder commented Jan 4, 2016

@ivanscattergood If you are still seeing this issue, #5264 may have fixed it. That fix should be in tonights nightly build.

@jwilder
Copy link
Contributor

jwilder commented Jan 19, 2016

Fixed via #5264

@zstyblik
Copy link

@jwilder close then?

@ivanscattergood
Copy link
Author

I'm Happy to close this issue

@jwilder
Copy link
Contributor

jwilder commented Jan 21, 2016

Thanks! Forgot to close.

@jwilder jwilder mentioned this issue Mar 31, 2016
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jwilder jwilder

Labels
area/tsm panic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@pauldix @jwilder @otoolep @zstyblik @beckettsean @ivanscattergood