Include imageId / image hash in /images response

[foot]

To allow UI to accurately determine if the most recent tag is the same as whatever tag you're currently running.

For:
https://github.com/weaveworks/service-ui/issues/1172
https://github.com/weaveworks/service-ui/issues/1418

[squaremo]

Implementation notes ..

Here's the digests and image IDs of some images:

$ docker images --digests  nats
REPOSITORY          TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
nats                1.0.4-linux         sha256:7478d6a6f01b03be7827fa1a8a0f2c5b3a353aa1816bf51410d8c5e85d47f59c   4adf3ecbf9f3        2 weeks ago         6.53 MB
nats                1.0.4-linux         sha256:78cf8dbf2f1e1df7889e05e9c4ea13529aa57dbdcabe358fef7b0b6f723cc033   4adf3ecbf9f3        2 weeks ago         6.53 MB
nats                latest              sha256:7478d6a6f01b03be7827fa1a8a0f2c5b3a353aa1816bf51410d8c5e85d47f59c   4adf3ecbf9f3        2 weeks ago         6.53 MB
nats                latest              sha256:78cf8dbf2f1e1df7889e05e9c4ea13529aa57dbdcabe358fef7b0b6f723cc033   4adf3ecbf9f3        2 weeks ago         6.53 MB
nats                0.9.4               sha256:b22e176b878c315daac41d7f8b8b23d9ce6d738e0938a73389406513af9713f1   1a94f6e42778        15 months ago       7.54 MB
nats                0.8.0               sha256:2dfb204c4d8ca4391dbe25028099535745b3a73d0cf443ca20a7e2504ba93b26   0f7905e57de4        18 months ago       7.48 MB

NB:

• more than one tag can point at an image ID
• an {image ID, tag} pair can be associated with more than one digest

The manifests associated with nats:1.0.4-linux are in the gist at https://gist.github.com/squaremo/aca1537a2e4f3081b4a8669464cf9fb9.

Observations:

• the image ID for nats:1.0.4-linux in the docker images output corresponds with .config.digest in the schema v2, and nothing in the schema1 digest.

That's an image with a schema v2 manifest -- not all images have one. Here's one that doesn't:

quay.io/weaveworks/flux   1.0.0-beta                              sha256:43ebd38f6d8f4ba5659ac6cf34706d133b1e92924d2df36135d3c781d7f472f9   be493c7cf97b

The schema v1 manifest is at https://gist.github.com/squaremo/aca1537a2e4f3081b4a8669464cf9fb9#file-flux-1-0-0-beta-json.

Observation: the v1 schema doesn't mention the image ID anywhere! (And neither do the blobs to which it refers -- they are just bits of filesystem.)

[errordeveloper]

Image IDs didn't exist in v1. In the olden days, Docker relied on randomly assigned UUIDs, it now uses SHA256 which actually are generated based on layer data.

More in this video and slides.

Personally, the way they expose IDs to the user is still quite confusing. Also, note that @sha1:-notation in docker pull is actually also a little buggy #498 (comment).

FYI, Kuberentes can pull by image ID, if you set imageID here some code that seems like it could answer some of the questions about manifests:

https://github.com/kubernetes/kubernetes/blob/e4f01051b55958619dd006ccbb6e7ff54890511d/pkg/kubelet/kuberuntime/kuberuntime_image.go#L83

[errordeveloper]

> docker image ls --digests quay.io/weaveworks/flux
REPOSITORY                TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
quay.io/weaveworks/flux   1.0.0-beta          sha256:43ebd38f6d8f4ba5659ac6cf34706d133b1e92924d2df36135d3c781d7f472f9   be493c7cf97b        4 months ago        152MB

So the be493c7cf97b in the last column (confusing called IMAGE ID) is the old random UUID, which is there to maintain backwards compatibility (not sure why they still maintain it this way, they could just use first 12 chars of the secure hash, bur anyway...

[squaremo]

quay.io only serves schema version 1, but if you pull an image from quay.io, Docker will happily display an image ID for it. So what I would like to know is where does Docker get that image ID?

In the case of images in DockerHub, you can get both a schema v1 and a schema v2, and only the schema v2 has the ID.

Ultimately what we're trying to solve here is: how do I know if two tags actually point at the same image? The digest is no good, since it will change for different tags. And, if the image is in quay.io, we don't appear to be able to get an image ID either, which might have helped.

[errordeveloper]

quay.io only serves schema version 1

🤦‍♂️

[errordeveloper]

I bumped into this:

https://github.com/joyent/node-docker-registry-client/blob/f23b994e23f3eb0104ad393c41cb6225ba54d3fd/lib/registry-client-v2.js#L830

Looks like you have to opt-in to v2 using a heard.

[squaremo]

Looks like you have to opt-in to v2 using a heard.

A header, yes I know.

[justincormack]

You should not use v1 schema. It was replaced with v2 because random IDs were an enormous security hole, ie you can effectively replace any content in another image by duplicating the id...

[squaremo]

You should not use v1 schema

Surely good advice. In this situation, it's not up to us, but up to the users of flux, and/or the maintainers of quay.io, which doesn't return v2 schema upon request.

@justincormack Is there any answer to the question above -- "how do I know if two tags point at the same image?" -- with resort only to v1 schemas (and the blobs they point at)?

[justincormack]

See distribution/distribution#1662 (comment) for an explanation...

[squaremo]

OK thanks @justincormack. Perhaps my reading comprehension just isn't up to it, but I'm not able to get a definitive answer from that -- unless it's "when using older versions of docker or the registry the answer may require pulling an image to get the identifier used within docker" (presumably Docker itself does some trickery to obtain a stable ID, but this is considered dark magic and not spoken of).

I think I'll have to look at Alternative Means, like using the topmost image from the history (of a v1 schema). It won't be the ID Docker uses itself, but (I think?) it will be enough to see when tags are pointing at the same image.