Image check improvements

[DrJosh9000]

What

1. Make the default image pull policy configurable. Stop setting it to PullIfNotPresent unilaterally - set it to the default if not set already.
2. Move the creation of image check init containers to after applying PodSpecPatches, in order to catch images changed by the patches.
3. Check each image ref is well-formed by parsing it.
4. Derive the image pull policy to use for an image check from the policy set on the container. If none is set there, use a new configurable default. If that is not set either, default to PullAlways unless the image has a digest, in which case use PullIfNotPresent (Use PullIfNotPresent for images that have a digest? #415)
5. Since a container with pull policy Never can fail for a new reason (ErrImageNeverPull), check for this new reason.
6. A minor consistency cleanup in jobWatcher - stalledJobs is keyed by job UUID.

Why

1. If someone goes to the trouble of setting imagePullPolicy on a container, we should respect that.
2. PodSpecPatch can introduce images that slip through the pre-flight checks. (Nobody should be using PodSpecPatch to mutate the image checks currently - it would be hard to do reliably when there is more than 1 distinct image, because iteration order is essentially random.)
3. We can avoid wasting k8s resources by parsing image refs ourselves prior to creating a k8s job, and the parsed ref is needed to decide the pull policy based on whether there is a digest or not.
4. Fixes Use PullIfNotPresent for images that have a digest? #415 - to control what the pull policy for an image is in an image-check init container, one merely needs to set it on the corresponding containers in their podSpec{,Patch}, or configure a default when deploying the stack.
5. While testing I noticed that the informer seems to inform podWatcher of a pending pod with waiting container in this status only once, which makes it hard to apply a grace period in the same way as ImagePullBackOff. Thus the image pull checks now work in a separate goroutine looping over the current list of pending and running pods, rather than only on informer events. However this means ImagePullBackOff is failed slightly faster!

[wolfeidau]

Just a wee bit of feedback.

[wolfeidau]

Handy library, have starred that one ⭐

[DrJosh9000]

I tried finding one among the k8s libraries, and discovered they were using this under the hood.