csi: support Secrets parameter in CSI RPCs

[tgross]

Fixes #7668

CSI plugins can require credentials for some publishing and unpublishing workflow RPCs. Secrets are configured at the time of volume registration, stored in the volume struct, and then passed around as an opaque map by Nomad to the plugins.

---

I've left some comments in this PR as I was working through it to help us implement #7771 and #7670 later, but we're running out of time in the 0.11.2 window to complete all three and make sure it's tested completely.

[tgross]

I don't have a Ceph setup to validate this end-to-end yet, but fortunately some of the other plugins log that they got the secrets (sometimes just insecurely printing them!) if you send them, even if the plugin doesn't use it.

Hostpath plugin node publish operations:

I0511 17:58:59.728882       1 server.go:117] GRPC call: /csi.v1.Node/NodeStageVolume
I0511 17:58:59.728899       1 server.go:118] GRPC request: {"secrets":"***stripped***","staging_target_path":"/csi/staging/test-volume0/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_id":"17264bf3-93b1-11ea-8b80-0242ac110002"}
I0511 17:58:59.730818       1 server.go:123] GRPC response: {}
I0511 17:58:59.734539       1 server.go:117] GRPC call: /csi.v1.Node/NodePublishVolume
I0511 17:58:59.734551       1 server.go:118] GRPC request: {"secrets":"***stripped***","staging_target_path":"/csi/staging/test-volume0/rw-file-system-single-node-writer","target_path":"/csi/per-alloc/8c97cc5a-3e03-2016-b0fc-bdc5f4001c6b/test-volume0/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_id":"17264bf3-93b1-11ea-8b80-0242ac110002"}

EBS register a volume:

I0511 18:08:00.118455       1 controller.go:324] ValidateVolumeCapabilities: called with args {VolumeId:vol-024d9fe7b511b9be6 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[secretsauce:xyzzy] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS controller publish volume:

I0511 18:09:21.052749       1 controller.go:229] ControllerPublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 NodeId:i-087bb3ed4f5954a3a VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS node publish (note that node unpublish has no secrets options):

I0511 18:09:24.699656       1 node.go:94] NodeStageVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/csi/staging/ebs-vol0/rw-file-system-single-node-writer VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:09:24.699794       1 node.go:158] NodeStageVolume: find device path /dev/xvdba -> /dev/xvdba
I0511 18:09:24.699995       1 node.go:193] NodeStageVolume: formatting /dev/xvdba and mounting at /csi/staging/ebs-vol0/rw-file-system-single-node-writer with fstype ext4
I0511 18:09:24.700008       1 mount_linux.go:274] Checking for issues with fsck on disk: /dev/xvdba
I0511 18:09:24.733185       1 mount_linux.go:287] `fsck` error fsck from util-linux 2.30.2
fsck.ext2: Bad magic number in super-block while trying to open /dev/xvdba
/dev/xvdba:
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I0511 18:09:24.733212       1 mount_linux.go:293] Attempting to mount disk: ext4 /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:24.733224       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer])
E0511 18:09:24.741314       1 mount_linux.go:140] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
Output: mount: /csi/staging/ebs-vol0/rw-file-system-single-node-writer: wrong fs type, bad option, bad superblock on /dev/xvdba, missing codepage or helper program, or other error.

I0511 18:09:24.741350       1 mount_linux.go:346] Attempting to determine if disk "/dev/xvdba" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdba])
I0511 18:09:24.751793       1 mount_linux.go:349] Output: "", err: exit status 2
I0511 18:09:24.751814       1 mount_linux.go:322] Disk "/dev/xvdba" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/xvdba]
I0511 18:09:26.094995       1 mount_linux.go:326] Disk successfully formatted (mkfs): ext4 - /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:26.095021       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:26.103375       1 node.go:143] NodeStageVolume: volume="vol-024d9fe7b511b9be6" operation finished
I0511 18:09:26.103434       1 node.go:145] donedone
I0511 18:09:26.104369       1 node.go:279] NodePublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/csi/staging/ebs-vol0/rw-file-system-single-node-writer TargetPath:/csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:09:26.104411       1 node.go:440] NodePublishVolume: creating dir /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:26.104424       1 node.go:450] NodePublishVolume: mounting /csi/staging/ebs-vol0/rw-file-system-single-node-writer at /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer with option [bind] as fstype ext4
I0511 18:09:26.104434       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o bind /csi/staging/ebs-vol0/rw-file-system-single-node-writer /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:26.106889       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o bind,remount /csi/staging/ebs-vol0/rw-file-system-single-node-writer /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:42.485772       1 node.go:349] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:10:12.486556       1 node.go:349] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS controller unpublish:

I0511 18:14:07.456645       1 controller.go:275] ControllerUnpublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 NodeId:i-087bb3ed4f5954a3a Secrets:map[secretsauce:xyzzy] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:14:12.723475       1 controller.go:298] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:14:20.126080       1 controller.go:292] ControllerUnpublishVolume: volume vol-024d9fe7b511b9be6 detached from node i-087bb3ed4f5954a3a
I0511 18:14:42.724676       1 controller.go:298] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

[cgbaker]

should MountOptions be removed from the CSI Volume list stub structs in api and structs?

[tgross]

I missed it in the stub but we're redacting it in the full response only in the HTTP API: https://github.com/hashicorp/nomad/pull/7923/files#diff-91535e68265a2446b33edbefba3453d0. Will fix to include the stub response too.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.