-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Registry push fails with "unexpected commit digest" #3856
Comments
Is it a multi-platform build or a build where the result could contain multiple identical layers? |
Single platform-build. Not sure if the images could contain identical layers. The build graph pushes multiple images, which will share layers. |
I looked into this a bit more. I realized that the specific error message I didn't notice anything looking through the relevant code yet, but if you have any buildkitd logs from around the time these errors occurred @aaronlehmann, that could be super helpful. Specifically, any error logs and any debug logs for the http requests made to registries would be the most helpful, but as much as you can share would be ideal. |
Also would be good to track down what these digests are: "sha256:e58168ce66f4e17b1c89ff4584004deb04ad5b4e91a7f6cbacef8739b4a6de9a, expected sha256:5fc8839f3cb47293c8e4505ab59e4da0e2133c77b2036cd90b21742d1ebd6b99" . I would hope one of them is a blob in the state directory but not sure which one is the correct one. If we find the layer then what image(or --cache-from) did it come from and how was that image used in the build. It is possible the same digest is somewhere in the progress logs as well, for example when it is doing cache import. |
This builder instance doesn't exist anymore, so all I have to go on is the logs, which are quite busy. Unfortunately, reproducing the issue is not straightforward, since it only showed up in production use, not in our automated tests.
|
Seen after an upgrade to buildkit v0.11 (08941b1):
The error is repeated for a few different refs being pushed simulatenously, with the same SHAs referenced.
It's followed by a stack trace including
github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
(the rest of the stack trace doesn't seem very interesting, and just seems to show theSolve
hander).I suspect this may be related to the use of MergeOp in the build graph, as we haven't seen this error on similar build systems where MergeOp isn't used.
cc @sipsma @coryb
The text was updated successfully, but these errors were encountered: