Fix cache warming rapid loop

[stefanprodan]

Fix #926

Tested with stefanprodan/flux:cpufix-090b064-wip3 on GKE:

[rade]

The correct way to fix is is to

1. keep the fall-through in the first select
2. ditch the fall-through in the 2nd select
3. expand the 2nd select with branches for stop and w.Priority

[rade]

FWIW, the spin loop was introduced in #835.

[rade]

Why was the original code not consuming an entire core all the time? Is that because every askForNewImagesInterval (1min) it did some real work, which involves i/o and hence released the CPU?

[stefanprodan]

Yes every time the refresh timer hits it does some work for a couple of seconds. The rest of the time just sits in the default loop.

[rade]

LGTM, though @Sambooo or @squaremo should take a look since they are more familiar with this code than me.

[squaremo]

Looks good in sum @stefanprodan, thank you. Thanks @rade for helping knock it into shape.

Please rebase into a single commit -- or two, if you want to do the factoring of priorityWarm separately -- and write a short commit message explaining what the problem was and how this fixes it.

[stefanprodan]

@squaremo I've squashed the commits into one.
@rade thanks for guiding me through this.

[stefanprodan]

@squaremo let me know if it's ok for me to merge this PR.

[squaremo]

I've squashed the commits into one.

Thank you. I am a bit baffled by the commit message though -- did you really add those things? (I don't think you did.) Something like this would describe what's being fixed:

To implement the loop as described in the comment above it, when the backlog is empty we need to block until the refresh tick fires (or a priority image comes in). By having a default case, it's possible to spin.

(the commit summary can be a bit shorter too -- just "Avoid spin in cache refresh loop" would be fine IMO).

[squaremo]

LGTM, thank you for your patience Stefan.