buildx 0.10.0 seems to have changed something which leads to a different kind of manifests (at least for GCP) #187

Lawouach · 2023-01-20T08:16:02Z

Behaviour

Steps to reproduce this issue

Create a build with buildx v0.9.1 and push it to GCP container registry (or even their new artefact registry)
Use said image from Cloud Run => it works
Do the same with v0.10.0 and it now fails to be deployed by Cloud Run

Expected behaviour

Either version should work.

Actual behaviour

What I could notice is that, with v0.9.1 we get a single image with a single manifest. With the newer version, it looks like we have three images and a different manifest,. Two of the images are actually empty.

with v0.9.1

{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:02dc4c100ef625d2dc5b40499b076a86648d58b05968214a228868966c04cd47",
      "size": 4445
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:3f701d9643d37eca8cb445ba978c767bc1e07bf958e94504db0298999bfe58c1",
         "size": 27478988
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:5359441c5fbe3749694e1aa16b863c2f51af7b2860919abfc226754102cdabfd",
         "size": 15815598
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:75c4a9d4d7ef0593255452f2ec0a848a59c17d60b2fe862e0d57e748e8cec377",
         "size": 1677
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:b67de70bba634431f53729f10b75ac2385b21c9d970c0a48734ff9a2798ee171",
         "size": 26863764
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
         "size": 32
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:aa1b41bbbf18f17dd7bc8b55ba7048e96d3c855d70c3d928f2d62f5ea4959171",
         "size": 229
      }
   ]
}

with v0.10.0

{
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:640eb44cbf9bc53054086d68131f4345138085b0b37d061a812c07c922dbf76c",
         "size": 1432,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:752970a0a4550a28f85a0a49530c38e787394f2f10884712816814e3932fc71e",
         "size": 566,
         "annotations": {
            "vnd.docker.reference.digest": "sha256:640eb44cbf9bc53054086d68131f4345138085b0b37d061a812c07c922dbf76c",
            "vnd.docker.reference.type": "attestation-manifest"
         },
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

My guess is that there is no actual bug with buildx or your action, but GCP cannot handle the different kind of manifest. But I wondered if your action should/could expose a flag that allows using the newer version but keep the old manifest?

Lawouach · 2023-01-20T08:16:40Z

For now, the workaround is to pin to v0.9.1

tonistiigi · 2023-01-20T08:38:14Z

it now fails to be deployed by Cloud Run

What fails and what error?

it looks like we have three images

?

This is new provenance attestation feature. You can disable it by setting --provenance false, but the image you are showing is a valid OCI image, and there should be no issues with pushing or pulling it.

Lawouach · 2023-01-20T08:43:15Z

Thank you for the update. I must admit I have not dug very deeply into new formats so I appreciate the guidance indeed.

and there should be no issues with pushing or pulling it.

That's exactly the problem however. Cloud Run complains the image doesn't exist (not the tag, the image itself). When I can happily pull it with docker directly indeed.

azrshana · 2023-01-20T09:27:57Z

We have just hit this issue as well. With v0.9.1, buildx build application/vnd.docker.distribution.manifest.v2+json images, and starting with v10.0, it started building application/vnd.oci.image.index.v1+json by default. This causes Cloud Run to say that the image is not found and to fail. We have also reverted to using version v0.9.1 to resolve the issue.

Could we understand where this default behavior for buildx is documented and how we may pin the image type to application/vnd.docker.distribution.manifest.v2+json` while still using v.10.0+.

Workaround:

 - name: Setup buildx
        uses: docker/setup-buildx-action@v1
        with:
          install: true
          version: v0.9.1

sylr · 2023-01-20T09:46:27Z

Have the same problem with ghcr.io, docker manifest inspect did not work on new images generated with latest buildx. Also they appeared weirdly on github.

v0.9.1:

v0.10.0:

jedevc · 2023-01-20T10:36:07Z

👋 Thanks for the bug reports! Very much appreciated, thanks for taking the time!

@Lawouach @azrshana I've opened a separate tracking issue for the GCR failure - docker/buildx#1533. There's not a lot we can really do about this one - buildkit is producing spec-compliant images in this case, cloud run is just not really doing anything with them 😢

As a temporary workaround, instead of pinning to a lower buildx version (once pinned, things often stay pinned 👀), I can recommend setting --provenance=false on the buildx cli, or provenance: false if using the docker/build-push-action. This will disable the attestation that's causing that change to the multi-platform image.

@sylr docker manifest unfortunately doesn't support OCI images - we're tracking this docker/buildx#1509. We hadn't realized how much buildx and docker manifest were being used together - as a workaround until we can get support for docker manifest to support OCI images, you might be interested in using buildx imagetools - there's some more detail in the linked issue.

cc @crazy-max do you know about the GitHub manifest display issue? I remember we discussed previously.

I'm gonna close this issue, we can track each component separately 🎉 If there's anything I missed, please shout ❤️

Lawouach · 2023-01-20T10:58:10Z

Thanks @jedevc for the swift care for this one and I appreciate the advice. I'll try.

sylr · 2023-01-20T11:09:08Z

For the record I used docker manifest inspect in my deployment pipelines to be sure I'm not about to deploy an helm release with versions of a docker images that haven't been pushed to the registry yet.

see docker/setup-buildx-action/issues/187

ref docker/setup-buildx-action#187

pantelis-karamolegkos · 2023-01-24T11:26:49Z

Facing a similar error in GCR even with

      - name: set up docker buildx
        id: setup-buildx
        uses: docker/[email protected]
        with:
          version: v0.9.1
          driver-opts: network=host

and

      - name: build and push to local registry
        uses: docker/[email protected]
        with:
          context: ${{ inputs.context }}
          file: ${{ inputs.context }}/${{ inputs.dockerfile }}
          no-cache: ${{ inputs.no_cache }}
          build-args: ${{ inputs.build_args }}
          push: true
          tags: ${{ env.LOCAL_IMAGE }}
          builder: ${{ steps.setup-buildx.outputs.name }}
          provenance: false
          sbom: false
          attests: ""

Could the issue be related to BuildKit? I see that it using 0.11 no matter the buildx version I am using.

pantelis-karamolegkos · 2023-01-24T14:08:41Z

Is there a way to set the BuildKit version as an action input?
No matter the action version I pin, BuildKit is always 0.11

Refs https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/33, docker/setup-buildx-action#187

This commit updates some of the docker action dependencies to fix issues related to the latest versions of buildx. - Updates docker/setup-buildx-action to v2 - Updates docker/build-push-action to v4 - Adds the `provenance: false` property to build-push-action to fix issue with multi-arch builds. See docker/setup-buildx-action#187

majidakbaridh · 2023-12-11T11:15:51Z

Is there any way to use newer version of setup-buildx-action but changing mediaType to "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json" ?

Geczy · 2023-12-28T05:11:46Z

this is still an issue in 2024

majidakbaridh · 2023-12-28T11:10:48Z

Is there any way to use newer version of setup-buildx-action but changing mediaType to "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json" ?

I found the solution, as mentioned in other comments, you need to set the provenance: false, however the point is that you need to change it in build-push-action action not this setup-buildx-action one.

Geczy · 2023-12-28T13:12:34Z

i almost had it working but the docker buildx shit was ruining the manifest, see left. correct is on the right side

i tried everything to make it do a better manifest but idk how

like

  - name: Build and push
    uses: docker/bake-action@v4
    with:
      push: true
      sbom: false
      provenance: false
      set: |
        *.output=type=docker,oci-mediatypes=false
        *.platform=linux/arm64

after docker buildx push

➜ ~ docker pull ghcr.io/dotabod/steam:v2.2
v2.2: Pulling from dotabod/steam
manifest unknown

vs docker compose push

➜ ~ docker pull ghcr.io/dotabod/steam:v2.2
v2.2: Pulling from dotabod/steam
1207c741d8c9: Pulling fs layer
854bbbc098b2: Pulling fs layer
d30ccb81ed57: Pulling fs layer
74a11634af6c: Waiting
e68101b738c0: Waiting

I even tried 0.9.1
dotabod/backend@e2b88ea

This is the main trials
https://github.com/dotabod/backend/blob/abb6c688de712ac8d94c30da13ee67c1590c79de/.github/workflows/builder.yml

You can see x bake here also

https://github.com/dotabod/backend/blob/abb6c688de712ac8d94c30da13ee67c1590c79de/docker-compose.yml

How do I fix this manifest while still using the action buildx since it gives me caching abilities

majidakbaridh · 2023-12-28T13:56:26Z

build-push-action

i almost had it working but the docker buildx shit was ruining the manifest, see left. correct is on the right side

i tried everything to make it do a better manifest but idk how

like
  - name: Build and push
    uses: docker/bake-action@v4
    with:
      push: true
      sbom: false
      provenance: false
      set: |
        *.output=type=docker,oci-mediatypes=false
        *.platform=linux/arm64
after docker buildx push

➜ ~ docker pull ghcr.io/dotabod/steam:v2.2 v2.2: Pulling from dotabod/steam manifest unknown

vs docker compose push

➜ ~ docker pull ghcr.io/dotabod/steam:v2.2 v2.2: Pulling from dotabod/steam 1207c741d8c9: Pulling fs layer 854bbbc098b2: Pulling fs layer d30ccb81ed57: Pulling fs layer 74a11634af6c: Waiting e68101b738c0: Waiting

I even tried 0.9.1 dotabod/backend@e2b88ea

This is the main trials https://github.com/dotabod/backend/blob/abb6c688de712ac8d94c30da13ee67c1590c79de/.github/workflows/builder.yml

You can see x bake here also

https://github.com/dotabod/backend/blob/abb6c688de712ac8d94c30da13ee67c1590c79de/docker-compose.yml

How do I fix this manifest while still using the action buildx since it gives me caching abilities

The explained method is working with build-push-action action. However you are using bake-action action.

sudo-bmitch · 2023-12-28T14:20:57Z

@Geczy it looks like you are storing the cache to the same tag as the output image. Try using a different tag for the cache output.

Geczy · 2023-12-28T14:22:01Z

Build push action doesn't work for me because I'm using a docker compose file. Do I need to use build push action and call it once for every service I have and also convert my compose yml into the dockerfile just to get this working?

Geczy · 2023-12-28T14:23:25Z

@Geczy it looks like you are storing the cache to the same tag as the output image. Try using a different tag for the cache output.

Thanks I'll give that a shot

Geczy · 2023-12-28T14:55:06Z

@sudo-bmitch caching looks to be working now, and the manifest looks good too, but it doesn't seem to update my package with the new build even though i made a file change that caused it to rebuild

https://github.com/dotabod/backend/actions/runs/7348686262/job/20007250704

any ideas what else i can try?

sudo-bmitch · 2023-12-28T15:04:47Z

@Geczy The output type=docker will push the result to the local docker engine. If you want it pushed to the registry, use type=registry or type=image,push=true. https://docs.docker.com/engine/reference/commandline/buildx_build/#output

Geczy · 2023-12-28T15:15:53Z

wow all my issues i spent a day on are solved now! tysm

stamtos · 2024-11-17T21:11:08Z

Hi everyone,

I wanted to share my experience with this issue, particularly in the context of multi-platform builds and AWS ECR.

I encountered several problems:

My builds were resulting in multi-platform manifests being pushed to ECR instead of single-platform image layers. This caused compatibility issues, and I also noticed “weird” images with zero size in the registry.
When I inspected the manifest using docker manifest inspect, I saw a structure like this:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1997,
         "digest": "sha256:29bb64ac7479b2b98ad64c8b67a7deb8ca09964503c8ad3885fc0036fb87d29e",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:afd16d556c6441b2e5bc6d9198f2d96da715d816092123cf9542dbbfcbbef24e",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

When I tried to create a manifest manually using docker manifest create, I got an error like:

my-image is a manifest list

This made it impossible to manage the images as intended.

Following a suggestion in this thread, I added the --provenance=false flag to my docker buildx command. This resolved the issue. Here’s the updated command I used:

docker buildx build --provenance=false --platform linux/amd64 -t my-image:tag --push .

This adjustment ensured that the image was built and pushed as single-platform layers instead of a multi-platform manifest. After applying this change:
• AWS ECR handled the images correctly.
• The zero-size image issue disappeared.
• I was able to work with the image layers without encountering manifest-related errors.

Hope this helps others!

jedevc closed this as completed Jan 20, 2023

simone-pedrazzi mentioned this issue Jan 20, 2023

fix: disabling provenance as a workaround for buildx issues mirta-com/workflows#8

Merged

jsermer mentioned this issue Jan 21, 2023

Unable to update docker (with watchtower) pi-hole/docker-pi-hole#1289

Closed

stephenjamieson added a commit to stephenjamieson/changedetection.io that referenced this issue Jan 22, 2023

Docker build fix for buildx issue

5f5a72d

see docker/setup-buildx-action/issues/187

stephenjamieson mentioned this issue Jan 22, 2023

Docker build fix for buildx issue dgtlmoon/changedetection.io#1332

Closed

revam added a commit to ShokoAnime/ShokoServer that referenced this issue Jan 23, 2023

misc: workaround for incompatible buildx output

fa24790

ref docker/setup-buildx-action#187

chayim mentioned this issue Jan 25, 2023

Multi-platform docker build fix redis-stack/redis-stack#320

Merged

wrench12435678 mentioned this issue Jan 25, 2023

Watchtower gets a 404 when trying to update xavier-hernandez/goaccess-for-nginxproxymanager#130

Closed

jmontleon mentioned this issue Jan 26, 2023

Disable provenance, which results in extra artifacts in quay konveyor/builder#11

Merged

thewilkybarkid added a commit to PREreview/prereview.org that referenced this issue Feb 2, 2023

Update config

437c98e

Refs https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/33, docker/setup-buildx-action#187

zachhannum mentioned this issue Feb 14, 2023

Update docker build action dependencies rstudio/checkrs-tew#17

Merged

zachhannum mentioned this issue Feb 14, 2023

Update docker build action dependencies rstudio/checkrs-tew#18

Merged

Envek mentioned this issue Feb 21, 2023

It doesn't seem that 3.1.3 is available? evilmartians/fullstaq-ruby-docker#22

Closed

crazy-max mentioned this issue Jul 7, 2023

Manifest not displayed in GHCR UI when using attestations docker/build-push-action#900

Open

dbast mentioned this issue Aug 1, 2023

continuumio/miniconda3:latest manifest type Regression v2 to v1 anaconda/docker-images#366

Open

Geczy mentioned this issue Dec 28, 2023

Can’t set oci media type to off docker/bake-action#185

Closed

3 tasks

yorugac mentioned this issue Apr 3, 2024

Investigate if buildx version can be removed grafana/k6-operator#393

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildx 0.10.0 seems to have changed something which leads to a different kind of manifests (at least for GCP) #187

buildx 0.10.0 seems to have changed something which leads to a different kind of manifests (at least for GCP) #187

Lawouach commented Jan 20, 2023

Lawouach commented Jan 20, 2023

tonistiigi commented Jan 20, 2023

Lawouach commented Jan 20, 2023

azrshana commented Jan 20, 2023 •

edited

Loading

sylr commented Jan 20, 2023 •

edited

Loading

jedevc commented Jan 20, 2023

Lawouach commented Jan 20, 2023

sylr commented Jan 20, 2023

pantelis-karamolegkos commented Jan 24, 2023 •

edited

Loading

pantelis-karamolegkos commented Jan 24, 2023 •

edited

Loading

majidakbaridh commented Dec 11, 2023

Geczy commented Dec 28, 2023

majidakbaridh commented Dec 28, 2023

Geczy commented Dec 28, 2023

majidakbaridh commented Dec 28, 2023

sudo-bmitch commented Dec 28, 2023

Geczy commented Dec 28, 2023

Geczy commented Dec 28, 2023

Geczy commented Dec 28, 2023

sudo-bmitch commented Dec 28, 2023

Geczy commented Dec 28, 2023

stamtos commented Nov 17, 2024

buildx 0.10.0 seems to have changed something which leads to a different kind of manifests (at least for GCP) #187

buildx 0.10.0 seems to have changed something which leads to a different kind of manifests (at least for GCP) #187

Comments

Lawouach commented Jan 20, 2023

Behaviour

Steps to reproduce this issue

Expected behaviour

Actual behaviour

Lawouach commented Jan 20, 2023

tonistiigi commented Jan 20, 2023

Lawouach commented Jan 20, 2023

azrshana commented Jan 20, 2023 • edited Loading

sylr commented Jan 20, 2023 • edited Loading

jedevc commented Jan 20, 2023

Lawouach commented Jan 20, 2023

sylr commented Jan 20, 2023

pantelis-karamolegkos commented Jan 24, 2023 • edited Loading

pantelis-karamolegkos commented Jan 24, 2023 • edited Loading

majidakbaridh commented Dec 11, 2023

Geczy commented Dec 28, 2023

majidakbaridh commented Dec 28, 2023

Geczy commented Dec 28, 2023

majidakbaridh commented Dec 28, 2023

sudo-bmitch commented Dec 28, 2023

Geczy commented Dec 28, 2023

Geczy commented Dec 28, 2023

Geczy commented Dec 28, 2023

sudo-bmitch commented Dec 28, 2023

Geczy commented Dec 28, 2023

stamtos commented Nov 17, 2024

azrshana commented Jan 20, 2023 •

edited

Loading

sylr commented Jan 20, 2023 •

edited

Loading

pantelis-karamolegkos commented Jan 24, 2023 •

edited

Loading

pantelis-karamolegkos commented Jan 24, 2023 •

edited

Loading