Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csi: allow more than 1 writer claim for multi-writer mode #9040

Merged
merged 3 commits into from
Oct 7, 2020
Merged

csi: allow more than 1 writer claim for multi-writer mode #9040

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
87f62b5
csi: allow more than 1 writer claim for multi-writer mode
tgross Oct 7, 2020
b3342ad
extend CSI claim endpoint test to exercise multi-reader
tgross Oct 7, 2020
8e8e6c7
make sure WriteFreeClaims is exercised for multiwriter
tgross Oct 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ BUG FIXES:

* core: Fixed a bug where blocking queries would not include the query's maximum wait time when calculating whether it was safe to retry. [[GH-8921](https://github.com/hashicorp/nomad/issues/8921)]
* consul: Fixed a bug to correctly validate task when using script-checks in group-level services [[GH-8952](https://github.com/hashicorp/nomad/issues/8952)]
* csi: Fixed a bug where multi-writer volumes were allowed only 1 write claim. [[GH-9040](https://github.com/hashicorp/nomad/issues/9040)]

## 0.12.5 (September 17, 2020)

Expand Down
19 changes: 19 additions & 0 deletions nomad/csi_endpoint_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,25 @@ func TestCSIVolumeEndpoint_Claim(t *testing.T) {
require.Equal(t, id0, volGetResp.Volume.ID)
require.Len(t, volGetResp.Volume.ReadAllocs, 1)
require.Len(t, volGetResp.Volume.WriteAllocs, 1)

// Make a second reader claim
alloc3 := mock.Alloc()
alloc3.JobID = uuid.Generate()
summary = mock.JobSummary(alloc3.JobID)
index++
require.NoError(t, state.UpsertJobSummary(index, summary))
index++
require.NoError(t, state.UpsertAllocs(index, []*structs.Allocation{alloc3}))
claimReq.AllocationID = alloc3.ID
err = msgpackrpc.CallWithCodec(codec, "CSIVolume.Claim", claimReq, claimResp)
require.NoError(t, err)

// Verify the new claim was set
err = msgpackrpc.CallWithCodec(codec, "CSIVolume.Get", volGetReq, volGetResp)
require.NoError(t, err)
require.Equal(t, id0, volGetResp.Volume.ID)
require.Len(t, volGetResp.Volume.ReadAllocs, 2)
require.Len(t, volGetResp.Volume.WriteAllocs, 1)
}

// TestCSIVolumeEndpoint_ClaimWithController exercises the VolumeClaim RPC
Expand Down
7 changes: 6 additions & 1 deletion nomad/structs/csi.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,8 +384,13 @@ func (v *CSIVolume) WriteSchedulable() bool {
// WriteFreeClaims determines if there are any free write claims available
func (v *CSIVolume) WriteFreeClaims() bool {
switch v.AccessMode {
case CSIVolumeAccessModeSingleNodeWriter, CSIVolumeAccessModeMultiNodeSingleWriter, CSIVolumeAccessModeMultiNodeMultiWriter:
case CSIVolumeAccessModeSingleNodeWriter, CSIVolumeAccessModeMultiNodeSingleWriter:
return len(v.WriteAllocs) == 0
case CSIVolumeAccessModeMultiNodeMultiWriter:
// the CSI spec doesn't allow for setting a max number of writers.
// we track node resource exhaustion through v.ResourceExhausted
// which is checked in WriteSchedulable
return true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a bit unfortunate - but it makes sense. Does it make sense to have an higher level/integration/e2e test to test the semantics?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason it wasn't originally E2E tested was because very few CSI plugins actually allow this mode (typically on-prem solutions), so we can't feasibly do an E2E test for it. But we could probably add this dimension to one of the existing integration-style tests in the nomad package. Let me take a look at that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dug into this a bit and that node resource exhaustion check isn't being made during the claim but during scheduling (in the feasibility checker), which in retrospect makes sense as we only validate arguments in an RPC and you can add resources to a cluster after the job submission has been made to make it feasible.

I've swapped one of the volumes in the existing feasibility checker tests to multiwriter and verified this code path is getting hit. I've also extended one of the existing RPC tests a bit to make sure the logic for ReadFreeClaims is being checked better while I'm in here. Our CSI E2E tests could use some work in general, but adding exhaustion checking would be a good idea for future work there.

default:
return false
}
Expand Down
5 changes: 5 additions & 0 deletions nomad/structs/csi_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ func TestCSIVolumeClaim(t *testing.T) {
vol.ClaimRelease(claim)
require.True(t, vol.ReadSchedulable())
require.True(t, vol.WriteFreeClaims())

vol.AccessMode = CSIVolumeAccessModeMultiNodeMultiWriter
require.NoError(t, vol.ClaimWrite(claim, alloc))
require.NoError(t, vol.ClaimWrite(claim, alloc))
require.True(t, vol.WriteFreeClaims())
}

func TestCSIPluginJobs(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion scheduler/feasible_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ func TestCSIVolumeChecker(t *testing.T) {
vol := structs.NewCSIVolume(vid, index)
vol.PluginID = "foo"
vol.Namespace = structs.DefaultNamespace
vol.AccessMode = structs.CSIVolumeAccessModeMultiNodeSingleWriter
vol.AccessMode = structs.CSIVolumeAccessModeMultiNodeMultiWriter
vol.AttachmentMode = structs.CSIVolumeAttachmentModeFilesystem
err := state.CSIVolumeRegister(index, []*structs.CSIVolume{vol})
require.NoError(t, err)
Expand Down