400 Bad Request when uploading cache manifest

[grzleadams]

Version

  "versions": [                                                                                          
    {                          
      "component": "core",     
      "version": "3.49.1",   
      "package": "pulpcore",                       
      "module": "pulpcore.app",                                                                          
      "domain_compatible": true
    },                        
    {                           
      "component": "container",        
      "version": "2.19.2",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": false
    },

Describe the bug
We use docker/build-push-action@v3 in our workflows and when we try to use image caching, we get 400 Bad Request:

#20 writing cache manifest sha256:aab0bc0ce8d3b398d3a61724a079214b0f72cbd0b8e2ee76188fa2b1233a13ab
#20 preparing build cache for export 151.3s done
#20 writing cache manifest sha256:aab0bc0ce8d3b398d3a61724a079214b0f72cbd0b8e2ee76188fa2b1233a13ab 0.2s done
#20 ERROR: error writing manifest blob: failed commit on ref "sha256:aab0bc0ce8d3b398d3a61724a079214b0f72cbd0b8e2ee76188fa2b1233a13ab": unexpected status from PUT request to https://pulp.<domain>/v2/<image>/manifests/latest: 400 Bad Request

To Reproduce
Steps to reproduce the behavior:

1. Use the docker/build-push-action@v3 action and try to cache image layers in the Pulp registry:

    - name: Build and push
      id: build-and-push
      uses: docker/build-push-action@v3
      with:
        context: '.'
        cache-from: type=registry,ref=${{ inputs.docker-cache-registry }}/${{ inputs.image-name }}-cache:latest
        cache-to: type=registry,ref=${{ inputs.docker-cache-registry }}/${{ inputs.image-name }}-cache:latest,mode=max
        push: true
        <snip>
        tags: ${{ inputs.docker-cache-registry }}/${{ inputs.image-name }}:latest

2. Workflow will fail to upload cache manifest.

Expected behavior
I would expect image caching to work as expected.

Additional context
N/A

[grzleadams]

For reference, docs on registry cache are here.

[ipanova]

do you have any logs on that 400 error?

[ipanova]

does it push only manifest without blobs?

[grzleadams]

Unfortunately, the logs in my initial post are all I really have to go on, but it should be fairly easy to reproduce (just do a simple build and then set cache-to to type=registry, where Pulp is the registry).

[grzleadams]

does it push only manifest without blobs?

I believe blobs are written (as needed), but the manifest can reference blobs that had been pushed to the cache previously, saving build time.

[grzleadams]

I wonder if the issue is related to docker/buildx#271... I'll add the image-manifest=true to my cache-to to see if it helps.

[grzleadams]

Adding image-manifest=true,oci-mediatypes=true to cache-to did in fact resolve the issue. Unless there are documentation updates that you think would be appropriate, we can close this.