Decide what to do about :tag@digest changes in Docker

[mtrmac]

https://github.com/moby/moby/releases/tag/v20.10.20 has changed :tag@digest semantics to a more secure one.

• Do we want that at all? It may break users.
• Now, or only in Podman 5?
• (And if we do, how do we implement that?)

[vrothberg]

I am in favor of doing it. It seems like a reasonable bug fix to me and since we aim for Docker compat I don't think we need to wait for major bump. It also impacts Buildah btw.

[vrothberg]

@rhatdan @Luap99 WDYT?

[Luap99]

I cannot really comment about that part of the code but as a user it sounds like a bug to me if only the digest is used and the name is ignored. I also don't think we need a major version bump to change it.

[lcarva]

One use case for having the :tag@digest syntax is to provide metadata to assist in updates. For example, when using services like dependabot or renovate bot, this syntax allows users to use a pinned reference to achieve reproducible builds, while allowing those services to provide correct reference updates when a newer image is available. This syntax has wide adoption in the container ecosystem, including projects within this GitHub organization.

Podman supports it:

$ podman pull registry.redhat.io/ubi8:8.7@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6
Trying to pull registry.redhat.io/ubi8@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6...

Buildah supports it

$ cat Dockerfile 
FROM registry.redhat.io/ubi8:8.7@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6

$ buildah bud .
STEP 1/1: FROM registry.redhat.io/ubi8:8.7@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6
COMMIT
--> 16fb1f57c5a
16fb1f57c5af458cd4ee28c5d90e44a2ee4f264dd617f8e13111617928b87aaa

OpenShift 4 also supports such references. It didn't initially, and it made things a bit awkward in some projects, e.g. use this yaml to deploy this app, but if on OpenShift then use this other yaml.

Skopeo doesn't support it:

$ skopeo inspect --no-tags docker://registry.redhat.io/ubi8:8.7@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6
FATA[0000] Error parsing image name "docker://registry.redhat.io/ubi8:8.7@sha256:83c0e63f5efb64cba26be647e93bf036b8d88b774f0726936c1b956424b1abf6": Docker references with both a tag and digest are currently not supported

In the skopeo case, users are left in a position where they can't just feed the image reference they have into a skopeo command. They have to detect usage of the syntax :tag@digest in the image reference, and if used, remove the tag value from it.

It's unfortunate that usage/handling of the :tag@digest syntax is being framed as a bug. It serves an important use case. It may cause confusion to users new to the container ecosystem, but nothing that can't be addressed by a little documentation.

[mtrmac]

One use case for having the :tag@digest syntax is to provide metadata to assist in updates. For example, when using services like dependabot or renovate bot, this syntax allows users to use a pinned reference to achieve reproducible builds, while allowing those services to provide correct reference updates when a newer image is available.

To be very clear, following the Docker semantics change linked to above (instead of ignoring the tag, enforcing that the tag exists and points at the specified digest), would turn this “assisting in updates” into “outage on updates”. So that’s the point of concern.

And from that point of view, Skopeo is in the, in my opinion, fortunate position, of not facing this dilemma because it didn’t allow that usage. And now Skopeo has the option, in principle, of allowing the :tag@digest in the considered enforce-both semantics, without breaking any existing users.

[Luap99]

Maybe I have misunderstood this, but it validates the image name right now, not tag!

Docker now checks if the digest matches the repository
name used to pull the image

This is the current podman behaviour:

$ podman images --digests 
REPOSITORY                         TAG         DIGEST                                                                   IMAGE ID      CREATED       SIZE
registry.fedoraproject.org/fedora  latest      sha256:7d36dae8e7a197a561818356374034d8efab31f84083d852632cb8efad1f46e6  7ae63e9321d0  4 weeks ago   171 MB
docker.io/library/alpine           latest      sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad  9c6f07244728  3 months ago  5.83 MB

Now use the alpine image reference with the digest of the fedora image, this will use the fedora one

$ podman run docker.io/library/alpine@sha256:7d36dae8e7a197a561818356374034d8efab31f84083d852632cb8efad1f46e6 cat /etc/os-release 
NAME="Fedora Linux"
...

Of course that only works when the image with the digest is in local storage.
IMO this behaviour makes no sense and changing it is a bug fix.

[mtrmac]

Oh… so that’s all my mistake. I’m sorry.

I thought that they are enforcing the tag value on the registry to match the expected digest; but they are enforcing the repo part, to ensure that the digest is present. The tag continues to be completely ignored.

[lcarva]

Of course that only works when the image with the digest is in local storage.
IMO this behaviour makes no sense and changing it is a bug fix.

Agree. I believe that's the exact bug fix that docker did as well:

Docker now checks if the digest matches the repository name used to pull the image, and otherwise will produce an error.

[mtrmac]

Now use the alpine image reference with the digest of the fedora image, this will use the fedora one

c/image is supposed to be enforcing the repo match: https://github.com/containers/image/blob/333c50e3eac84711807ef743be643bc0585e36be/storage/storage_reference.go#L136 .

It seems that c/common doesn’t, in

common/libimage/runtime.go

Line 441 in 24d7820

if isDigested {

— but IIRC most of lookupImageInDigestsAndRepoTags is already a “last desperate heuristic attempt” legacy path (compare containers/podman#11964 and IIRC earlier discussions), so I don’t feel at all strongly about what that should do.

[rhatdan]

I think we should support it.

[mtrmac]

@rhatdan support what? The original topic is restricting what inputs match.

Do you mean to enforce the same restrictions, or to support the looser matches which currently find images?

[rhatdan]

I think we should match Docker. Which I think is the looser matching.

[vrothberg]

The crazy digest matching popped up again in #1503. Fortunately, @Luap99 has the memory of an elephant and pointed me to this conversation.

As shown in #1248 (comment), foobar@digest may totally ignore foobar and only look for digest and hence may pick a non-foobar image.

What I propose is to

1. Drop this logic from libimage and see what Podman and Buildah CI have to say
2. Based on these results decide how to proceed