Simplify kicbase cache handling by removing digest #11411
Comments
|
Images with only digests could be handled too. The problem is when trying to have both RepoTags and RepoDigest. $ docker save busybox@sha256:be4684e4004560b2cd1f12148b7120b0ea69c385bcc9b12a637537a2c60f97fb > busybox.tar
$ docker load < busybox.tar
Loaded image ID: sha256:c55b0f125dc65ee6a9a78307d9a2dfc446e96af7477ca29ddd4945fd398cc698manifest.json [{"Config":"c55b0f125dc65ee6a9a78307d9a2dfc446e96af7477ca29ddd4945fd398cc698.json","RepoTags":null,"Layers":["a355ae7461fdd43484ed16e7e48620ff19b187adc03bcd4b5cfd5ba3ce2ee670/layer
.tar"]}]However, such an image will be listed as Which makes it rather useless for the kicbase, at least with the current cache implementation. Again, the fields are populated from registry: $ docker pull busybox:latest
latest: Pulling from library/busybox
Digest: sha256:be4684e4004560b2cd1f12148b7120b0ea69c385bcc9b12a637537a2c60f97fb
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest |
|
Hi @afbjorklund, this sounds like an issue I can work on. Can I take it? Thanks! |
|
@dinever : it's part of an ongoing effort to fix the kicbase and podman, see We currently have a working solution, with a patched go-containerregistry, so this is mostly simplication... If we remove the digest requirement, we can drop the patch and drop the current extra "daemon" code. And always use the "kic" files in the cache, and load the kicbase image from tarball instead of remote ? |
afbjorklund
added
co/docker-driver
|
@afbjorklund how about this, we first try to get the image by digest and if we fail, we try to get it without digest ... ? would that be any better ? are you saying in the current path we are not using the Digest SHA at all and giving the use no feeling of confidence of integrity at all ? I kind of do want to provide the SHA anywhere that we could, it would be nice for local developement too, if u accidently tag your image kicbase:v0.0.20 minikube wont use it |
|
if removing the digest from kicsha makes it easier for everyone, I will not object but I would love to see a proposal that at least we tried to verify the integrity and reduce the surface attack for our users |
That's not really how it works, first we pull the image by digest and use our patched library to save it. After loading it from the cache, we ignore the digest and pull the tag again and hope that they match. There is also no error reporting in the library, so the fetch and the load can fail silently and it goes on. So we would have to pull it by tag, and then we could check the digest to see if it matches before saving. This is what the manifest looks like: crane pulldocker savepodman saveWhen trying to save a regular image. docker.io/library/busybox:latest@sha256:be4684e4004560b2cd1f12148b7120b0ea69c385bcc9b12a637537a2c60f97fb So they each fail, in their own special way. Without the patch, we can only save tagged images (not digests). If you use both, then docker happily uses the digests and tags it with whatever you want. There is no check. $ docker pull gcr.io/k8s-minikube/kicbase:1.2.3@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e
gcr.io/k8s-minikube/kicbase@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e: Pulling from k8s-minikube/kicbase
Digest: sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e
Status: Image is up to date for gcr.io/k8s-minikube/kicbase@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e
gcr.io/k8s-minikube/kicbase:1.2.3@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962eWithout the patch, I suppose we go back to editing the manifest.json afterwards or something ? To get rid of the legacy domain "index.docker.io" and to replace the practical joke "i-was-a-digest" https://github.com/google/go-containerregistry/blob/main/pkg/v1/tarball/README.md |
|
To clarify, we would keep the old patch that saves the tag of the manifest also as the tag of the image.
But we would drop the patch that pulls the tag from the registry after loading the image from the tarball.
So when pulling from the registry, we would use the gcr.io/k8s-minikube/kicbase:v0.0.22@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e But when looking in the daemon/podman, we use the gcr.io/k8s-minikube/kicbase:v0.0.22 This way we don't have to always contact the registry to get the digest, but it would work offline as well... The downside would be that it would still match the old The way to check this would be to contact the registry and move the tag, but then it doesn't work offline. dockerpodmanApparently podman keeps both the multi-arch manifest and the image manifest under "RepoDigests". docker manifest inspect gcr.io/k8s-minikube/kicbase@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 7821,
"digest": "sha256:9bea9232c23bda2ed17835918bbaf18ae4d22075b9bc5b25085415f33947e268",
"platform": {
"architecture": "arm64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 7821,
"digest": "sha256:996fd2ec79bd471af1ca21889c8f61b021b46ff303f89b66f6b153026a9bd4e5",
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}And then podman stores the digest as Digest instead, whereas docker only uses RepoDigests. So with our workaround we would tag both images (amd64 + arm64) with the same original tag. The digest is then only used when creating the cache file, but not when loading or checking.
If we wanted to support having multiple images with the same tag, we could always load from file. It takes longer than just checking the tag, but is still faster than having to pull the image of course. So if you wanted to load an image with a different digest, it would move the tag accordingly.
|
spowelljr
added
kind/improvement
|
We added a feature to failover to trying to pull without the sha, which solves some of these issues. |
Currently we load the kicbase image using both tag and digest:
gcr.io/k8s-minikube/kicbase:v0.0.22@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962eHowever, we can only load the tag from the cache and not the digest:
gcr.io/k8s-minikube/kicbase:v0.0.22This makes the code not find the image next time, since it has no digest.
The current workaround is to pull the manifest down from the registry again:
The layers were already downloaded, so it pulls down the manifest and fills in the digest:
And so the code can find the image in the list. But it doesn't work offline or with podman.
It would be much simpler if we would only use the digest for the first download (to the cache).
After that, we would only use the (name and) tag to check if the image has already been loaded.
The only downside is that you cannot have two images with the same tag, but different digests.
However that doesn't really work anyway, when you pull the second one the tag will also move.
And we wouldn't have to patch go-containerregistry, if we did this...
We could still keep the digest in the filename, if we are paranoid.
kicbase_v0.0.22@sha256_7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e.tarBut then we would have to provide two parameters (or strip the digest)
The text was updated successfully, but these errors were encountered: