Use lock per VolumeId during NodeUnstageVolume operation to avoid conflicts during volume unmount

[chethanv28]

What this PR does / why we need it:
PR is adding usage of locking mechanism per VolumeId during NodeUnstageVolume operation.
There have few observations where the 1st NodeUnstageVolume call takes more time and is till going on. Meanwhile, k8s will issue a 2nd NodeUnstageVolume call assuming the 1st NodeUnstageVolume has timed out. The 2nd call succeeds as the target Mountpoint is not found. Therefore, a DetachVolume will be invoked while the 1st NodeUnstageVolume is still in-progress and in-turn corrupts the volume. To avoid the above issue, we can keep a lock per VolumeID during the NodeUnstageVolume operation.

A similar locking mechanism is applied to NodePublish, NodeUnPublish, NodeStage operations as well

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done:
Running e2e pipeline

Release note:

Use lock per VolumeId during NodeUnstageVolume operation

[svcbot-qecnsdp]

Started vanilla Block pipeline... Build Number: 2555

[svcbot-qecnsdp]

Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
------------------------------

Ran 1 of 816 Specs in 400.726 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 815 Skipped
PASS

Ginkgo ran 1 suite in 7m45.155169578s
Test Suite Passed
--
Ran 14 of 816 Specs in 12384.724 seconds
FAIL! -- 7 Passed | 7 Failed | 0 Pending | 802 Skipped
--- FAIL: TestE2E (12384.82s)
FAIL

Ginkgo ran 1 suite in 3h26m40.870854721s

Test Suite Failed

[svcbot-qecnsdp]

Started Vanilla block pre-checkin pipeline... Build Number: 2682

[deepakkinni]

/approve

[lipingxue]

@chethanv28 Change looks good to me. Are you able to repro the issue locally and test it with your fix to make sure the issue we observed in SR is fixed?

[svcbot-qecnsdp]

Started Vanilla block pre-checkin pipeline... Build Number: 2700

[svcbot-qecnsdp]

Build ID: 2700
Block vanilla build status: SUCCESS 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
------------------------------

Ran 1 of 816 Specs in 1889.692 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 815 Skipped
PASS

Ginkgo ran 1 suite in 32m27.542882256s
Test Suite Passed

[divyenpatel]

Prefix message with method name: NodeUnstageVolume

[divyenpatel]

some more minor comments. Change looks good to me.

[divyenpatel]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chethanv28, deepakkinni, divyenpatel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chethanv28,deepakkinni,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[divyenpatel]

/ok-to-test

[divyenpatel]

/lgtm

[tgelter]

Hello all, my team submitted the case with VMware support which led to this PR (thank you!). For our tracking purposes related to the bug this addresses, would you mind letting me know what release version should include this change?

[tgelter]

Hello all, my team submitted the case with VMware support which led to this PR (thank you!). For our tracking purposes related to the bug this addresses, would you mind letting me know what release version should include this change?

I see it was included in v3.3.0, thanks!