Detect image ID changes when determining whether or not to update

[jpellizzari]

When you push up a new image with the same tag(s), Flux does not allow you to deploy that image.

This situation arose when trying to re-build our internal ui-server image with new changes to Scope. Our specific workflow looks like this:

1. Merge a PR to master in the Scope repo. This starts a CI run for Scope
2. When the Scope CI run finishes, it triggers a build of service-ui, which will pull in the latest Scope source and re-build the bundle
3. service-ui does not have new commits, but it's dependency has changed, so a new ui-server image is built
4. The ui-server image is pushed but still has both the same :latest and the :master-{commit sha} tags as the previous build
5. Flux diffs the new image tag with the current running one and sees the same tags, so it thinks nothing needs to be deployed, even though the image itself has changed.

I think this will be a problem that a lot of users will run into, particularly at the beginning of the adoption of containers and/or kubernetes. While it might be a best-practice to incorporate commit hashes with container image tags (and treat tags as immutable), I don't believe we should enforce that opinion and potentially alienate users who haven't adopted that process yet. In the above scenario, I had a very simple use case of "I want my latest image deployed to staging so I can test it".

One simple solution would be to allow users to release a service, even when no changes are detected. I assume this would fetch the new container image and restart the pod, which would give the desired result.

[squaremo]

One simple solution would be to allow users to release a service, even when no changes are detected. I assume this would fetch the new container image and restart the pod, which would give the desired result.

There is an option for releasing new config:

fluxctl release --service=... --no-update

but it doesn't do what you'd want, because Kubernetes won't in general restart pods if nothing has (textually) changed.

[squaremo]

We can in principle compare layer IDs to detect if the image to which the tag refers is different; but again, in the absence of a textual change, Kubernetes won't treat it as a change. 😞

[errordeveloper]

Actually, I think it'd be nice if we looked up the ID and inserted that in the manifest. Please note that tag can be preserved too, so the diff will remain readable, yet it would be strongly bound to the content.

Here is a example of a diff:

diff --git a/load-test-deployment.yaml b/load-test-deployment.yaml
index 47e3821..5c0f7a9 100644
--- a/load-test-deployment.yaml
+++ b/load-test-deployment.yaml
@@ -16,7 +16,7 @@ spec:
         name: load-test
     spec:
       containers:
-      - image: errordeveloper/prom-nodejs-demo-load-test:slow@sha256:649fa202d914ac90a9e25cbb8cf7225181bfa1b2c00fa4fcab1b50a2617515ca
+      - image: errordeveloper/prom-nodejs-demo-load-test:slow@sha256:422d0105e815435a55cac3e6fc0e3ecb2b1f8c76847142eb5496cf191753848a
         name: load-test
         ports:
         - name: dummy

However, my test so far surfaced one issue, the tag (in my case slow) is completely ignored by docker, namely:

docker run errordeveloper/prom-nodejs-demo-load-test:anything-goes@sha256:649fa202d914ac90a9e25cbb8cf7225181bfa1b2c00fa4fcab1b50a2617515ca

still runs the same image, as specified by the ID, so the tag is effectively a comment.

Despite this, I think this notation is still useful, as user really doesn't have a capability to lookup the ID, otherwise we can insert a comment at the end of the line specifying the tag name.

I think Flux is rather well positioned to help user leverage immutable image IDs, and right now one can say that git is not source of truth as they can push whatever image under the same tag in the registry. Using image IDs sounds like a better defence on that front.

And by the way, Kubernetes already started using IDs internally (see output of docker ps on a node, or kubectl describe pod). So it'd be a nice story for Flux to adopt.

[errordeveloper]

Kubernetes already started using IDs internally

Actually, I've just re-checked this and it adds imageID field, perhaps that's what we should be considering.

[errordeveloper]

Also #909 is related to the general topic of using SHA256 digests.

[samb1729]

Because kubernetes won't change anything when reapplying a manifest as @squaremo said, the behaviour requested here is essentially

Make Flux tell every node in the cluster to re-pull a tag, then kill all relevant pods.

In general it's not recommended to push a different image to the same tag (I would argue the decision to allow that was a mistake), but if that's really what a user wants they can use imagePullPolicy: Always in their manifests.

[keikumata]

@squaremo any chance we can consider allowing this? We have a use case where we would like to keep pushing image updates to the same image tag, but flux won't automatically detect this as a change since the tag is the same.