CSI: panic in CLI when rendering verbose node status with failed alloc

[tgross]

While testing #8505 and encountering an unrelated error with attaching a CSI volume, I got a panic while running nomad node status -verbose :node_id:

$ nomad node status -verbose 495
ID              = 4956f359-b478-0177-7a06-a6548a0ce2ac
Name            = ip-172-31-81-78
Class           = <none>
DC              = dc1
Drain           = false
Eligibility     = eligible
Status          = ready
CSI Controllers = aws-ebs0
CSI Drivers     = aws-ebs0,aws-efs0
Uptime          = 9m8s

Host Volumes
Name  ReadOnly  Source

CSI Volumes
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5dc648d]

goroutine 1 [running]:
github.com/hashicorp/nomad/command.(*NodeStatusCommand).outputNodeCSIVolumeInfo(0xc00052f110, 0xc00002a1c0, 0xc0005670e0, 0xc0000d4940, 0x4, 0x4)
        github.com/hashicorp/nomad/command/node_status.go:566 +0x53d
github.com/hashicorp/nomad/command.(*NodeStatusCommand).formatNode(0xc00052f110, 0xc00002a1c0, 0xc0005670e0, 0x0)
        github.com/hashicorp/nomad/command/node_status.go:436 +0x1a93
github.com/hashicorp/nomad/command.(*NodeStatusCommand).Run(0xc00052f110, 0xc0001b8030, 0x2, 0x2, 0xc0000908a0)
        github.com/hashicorp/nomad/command/node_status.go:279 +0x87f
github.com/mitchellh/cli.(*CLI).Run(0xc0005be640, 0xc0005be640, 0xc0005b2fc0, 0x44)
        github.com/mitchellh/[email protected]/cli.go:255 +0x1da
main.RunCustom(0xc0001b8010, 0x4, 0x4, 0xc00018e058)
        github.com/hashicorp/nomad/main.go:138 +0x482
main.Run(...)
        github.com/hashicorp/nomad/main.go:83
main.main()
        github.com/hashicorp/nomad/main.go:79 +0x64

The panic is in command/node_status.go#L566. The (Node) CSIVolumes method calls /v1/volumes, and judging from the output of the HTTP API below my guess is that the volume ID we get below isn't in the list of api.VolumeRequest we have (probably because the alloc is placed but isn't running).

$ curl -s "$NOMAD_ADDR/v1/volumes?type=csi&node_id=4956f359-b478-0177-7a06-a6548a0ce2ac" | jq .
[
  {
    "ID": "ebs-vol0",
    "Namespace": "default",
    "Name": "ebs-vol0",
    "ExternalID": "vol-0af79d5317ff8ba09",
    "Topologies": [],
    "AccessMode": "single-node-writer",
    "AttachmentMode": "file-system",
    "CurrentReaders": 0,
    "CurrentWriters": 1,
    "Schedulable": true,
    "PluginID": "aws-ebs0",
    "Provider": "ebs.csi.aws.com",
    "ControllersHealthy": 1,
    "ControllersExpected": 1,
    "NodesHealthy": 2,
    "NodesExpected": 2,
    "CreateIndex": 37,
    "ModifyIndex": 65
  },
  {
    "ID": "efs-vol0",
    "Namespace": "default",
    "Name": "efs-vol0",
    "ExternalID": "fs-9b7c7318",
    "Topologies": [],
    "AccessMode": "single-node-writer",
    "AttachmentMode": "file-system",
    "CurrentReaders": 0,
    "CurrentWriters": 0,
    "Schedulable": true,
    "PluginID": "aws-efs0",
    "Provider": "efs.csi.aws.com",
    "ControllersHealthy": 0,
    "ControllersExpected": 0,
    "NodesHealthy": 2,
    "NodesExpected": 2,
    "CreateIndex": 84,
    "ModifyIndex": 137
  }
]

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.