volumewatcher: stop watcher goroutines when there's no work

[tgross]

Fixes #7837

The watcher goroutines will be automatically started if a volume has updates, but when idle we shouldn't keep a goroutine running and taking up memory.

In addition to the unit test updates, I've tested this out on a real-world cluster and everything seems to work out ok.

[langmartin]

I think this looks good. The idea is that since we're just collecting the claim, we'll:

• block until notify is called (or shutdown)
• read the one claim (or shutdown)
• exit
  so there's no way to notify twice, but we don't need to do that, because we're just using the channel to facilitate the one delay. I've got all that right?

[tgross]

so there's no way to notify twice, but we don't need to do that, because we're just using the channel to facilitate the one delay. I've got all that right?

Yes, with the addition that checkpointing will re-trigger the notify, ensuring that if we error-out we retry from the point of the checkpoint.

[tgross]

Hm, TestVolumeWatch_StartStop is failing here but was passing for me locally. There may still be something racy there I need to fix. It just can take a little while to quiesce, bumping up the require.Eventually timeout a bit makes it pass reliably.

[tgross]

Remaining test failure is a flake being fixed in #7913

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.