csi: panic after re-claim of volume

[tgross]

Probably related to cleanup failure #7629, but also a problem with the way we're keeping track of alloc claims that makes it easy to hit a NPE by mistake. #7708 will reduce the risk of this case, but I want to track this issue separately so we make sure we fix these edge cases after that's merged.

Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbed026]
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: goroutine 52 [running]:
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/nomad/structs.(*CSIVolume).ClaimWrite(0xc000d227e0, 0xc000bc7880, 0x0, 0xffffffffffffffff)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/nomad/structs/csi.go:399 +0x166
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/nomad/structs.(*CSIVolume).Claim(0xc000d227e0, 0x1, 0xc000bc7880, 0xc000d227e0, 0x0)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/nomad/structs/csi.go:362 +0xef
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/nomad/state.(*StateStore).CSIVolumeClaim(0xc000722f00, 0x9c, 0x305b240, 0x7, 0xc000d26d65, 0x7, 0xc000bc7880, 0x1, 0x0, 0x0)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/nomad/state/state_store.go:2049 +0x2df
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/nomad.(*nomadFSM).applyCSIVolumeClaim(0xc00038ee70, 0xc0007c0b81, 0x7c, 0x7c, 0x9c, 0x0, 0x0)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/nomad/fsm.go:1181 +0x455
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/nomad.(*nomadFSM).Apply(0xc00038ee70, 0xc000960460, 0x577acc0, 0xbf9d5c9fa0d084a4)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/nomad/fsm.go:272 +0x1771
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc0004f69e0)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:90 +0x2c1
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc00082ba00, 0x20, 0x40)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:113 +0x75
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc0006ad500)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:219 +0x42f
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc0006ad500, 0xc00080c100)
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:146 +0x55
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc
Apr 13 23:51:26 ip-172-31-16-16 nomad[10325]:         github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:144 +0x66

[tgross]

The patch for this is shipping in 0.11.1

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.