Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stability: crash on nil pointer in markDead #6001

Closed
mberhault opened this issue Apr 12, 2016 · 0 comments
Closed

stability: crash on nil pointer in markDead #6001

mberhault opened this issue Apr 12, 2016 · 0 comments

Comments

@mberhault
Copy link
Contributor

Build sha: c382dd7

Beta cluster restarted fresh earlier today.
A few issues already encountered, including node deaths (see #5998, #6000)

ec2-54-209-69-52.compute-1.amazonaws.com was restarted after dying of #5998. About 5 minutes after restart, it died with:

W160412 13:17:12.531435 server/node.go:665  [node=2] unable to log node_restart event for node 2: failed to send RPC: too many errors encountered (2 of 2 total): rpc error: code = 4 desc = context deadline exceeded
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x896fc6]

goroutine 12 [running]:
panic(0x163ce80, 0xc82000c0a0)
        /usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/cockroachdb/cockroach/storage.(*storeDetail).markDead(0xc822b92200, 0x14449f073e5e3d6d, 0x0)
        /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:57 +0x1b6
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
        /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:203 +0x4f8
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82036acb0, 0xc8203d5880)
        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62

I'm intentionally including the warning as it indicates problems writing to the event log and may or may not be related.
Full log for this node since restart:
node1.log.txt
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 12, 2016
@nvanbenschoten
storage: Fix nil pointer in markDead on log
03bd11d 
Fixes cockroachdb#6001.

The descriptor could have never been set on the `storeDetail` if it was
markedAlive and enqueue'd in getStoreDetailLocked and never markedAlive again.
I'm not sure if it makes sense to log anything, because chances are the
store was never alive to begin with.
@nvanbenschoten nvanbenschoten mentioned this issue Apr 12, 2016
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 12, 2016
@nvanbenschoten
storage: Fix nil pointer in markDead on log
3d32a23 
Fixes cockroachdb#6001.

The descriptor could have never been set on the `storeDetail` if it was
markedAlive and enqueued in getStoreDetailLocked and never markedAlive again.
I'm not sure if it makes sense to log anything, because chances are the
store was never alive to begin with.
@mberhault mberhault mentioned this issue Apr 13, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mberhault