question: copy from docker hub to local registry failing with MANIFEST_BLOB_UNKNOWN

[rgl]

I'm trying to copy a multi-platform image from https://hub.docker.com/repository/docker/ruilopes/example-docker-buildx-go to my local registry (a distribution registry:2.8.1 instance) with:

crane copy \
  --verbose \
  --allow-nondistributable-artifacts \
  docker.io/ruilopes/example-docker-buildx-go:v1.6.0 \
  localhost:5000/example-docker-buildx-go:v1.6.0

But its failing with:

2022/07/26 18:48:27 --> PUT http://localhost:5000/v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a
2022/07/26 18:48:27 PUT /v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a HTTP/1.1
Host: localhost:5000
User-Agent: crane/0.11.0 go-containerregistry/0.11.0
Content-Length: 1128
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1740,
      "digest": "sha256:7aad7680a864c9f637031aa86fa6d0ad3cc2d9b78d50aca9ced64ff708b1723c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 103153235,
         "digest": "sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26",
         "urls": [
            "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26"
         ]
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2429631,
         "digest": "sha256:e9309d65b2001b4a4c7352642f3a7e0e9b1717f695a73e5c731075d9ec7d2d6f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1083,
         "digest": "sha256:1e4e395adf8e39f6a1da5f2d8b6aafd12051e3a9c4304b27b000201e37bb9c30"
      }
   ]
}
2022/07/26 18:48:27 <-- 500 http://localhost:5000/v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a (3.497426ms)
2022/07/26 18:48:27 HTTP/1.1 500 Internal Server Error
Content-Length: 268
Content-Type: application/json; charset=utf-8
Date: Tue, 26 Jul 2022 17:48:27 GMT
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff

{"errors":[{"code":"UNKNOWN","message":"unknown error"},{"code":"UNKNOWN","message":"unknown error","detail":{}},{"code":"MANIFEST_BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26"}]}

Error: failed to copy index: PUT http://localhost:5000/v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a: multiple errors returned: UNKNOWN: unknown error; UNKNOWN: unknown error; map[]; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26

Please note that this is a multi-platform image, which contains linux and windows platform images.

Also note that I'm trying to copy all images and all their layers (including the nondistributable/foreign-layers) to the local registry.

Can you please help me troubleshoot what is going on?

The full log is attached as crane.log.

[imjasonh]

What version of crane are you using, in case it matters?

Copying non-distributable layers was added fairly recently in #1348 -- there might still be bugs lingering.

In your --verbose output, do you see requests that indicate that the non-distributable blob sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26 was actually being pushed to the destination repo before the manifest push failed?

[rgl]

Ops sorry, I'm using crane 0.11.0 and I've attached the full log to this issue.

The destination registry is empty. My goal is to have everything in there (including whatever is at https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26) for doing an air-gap install.

[imjasonh]

So that log seems to include a successful HEAD request for the blob in the destination registry:

2022/07/26 18:48:27 <-- 200 http://localhost:5000/v2/example-docker-buildx-go/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26 (4.064412ms)
2022/07/26 18:48:27 HTTP/1.1 200 OK
Content-Length: 103153235
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Content-Type: application/octet-stream
Date: Tue, 26 Jul 2022 17:48:27 GMT
Docker-Content-Digest: sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26"
X-Content-Type-Options: nosniff

2022/07/26 18:48:27 existing blob: sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26

This signal from the destination registry tells crane cp that it doesn't need to push that blob, but then the registry fails because it wasn't pushed. 🤔

Does this happen with regular non-non-distributable layers? It might help us tell whether this is a bug in the registry's support for non-distributable layers, or if there's some weird state happening in your registry that's causing problems for "regular" pushes too.

[rgl]

It works fine with regular layers. The problem seems to be the pushed manifest. I think it should have rewritten the first layer to be a regular layer, that is, use a application/vnd.docker.image.rootfs.diff.tar.gzip layer instead of application/vnd.docker.image.rootfs.foreign.diff.tar.gzip:

2022/07/26 18:48:27 PUT /v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a HTTP/1.1
Host: localhost:5000
User-Agent: crane/0.11.0 go-containerregistry/0.11.0
Content-Length: 1128
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1740,
      "digest": "sha256:7aad7680a864c9f637031aa86fa6d0ad3cc2d9b78d50aca9ced64ff708b1723c"
   },
   "layers": [
     {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 103153235,
         "digest": "sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26",
         "urls": [
            "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26"
         ]
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2429631,
         "digest": "sha256:e9309d65b2001b4a4c7352642f3a7e0e9b1717f695a73e5c731075d9ec7d2d6f"
      },

[imjasonh]

Hmm, it sounds like the theory is that distribution registry responds to a foreign.diff.tar.gzip layer with a MANIFEST_BLOB_UNKNOWN even if that blob has already been pushed? Is this something we could confirm with them, and see if they could relax if that's the case?

I'm not really comfortable changing the manifest contents (and therefore its digest) when we copy an image that includes non-distributable layers. This seems like something that will just cause more confusion and issues for folks later.

[rgl]

If crane does not rewrite the manifest, how would the client get that foreign reference without internet access?

Maybe I've got this wrong, but I though that --allow-nondistributable-artifacts intent was to copy all the manifests/layers from one registry to another, without leaving references to external registries.

Maybe I didn't made this clear, but my goal is to create an offline copy of all the manifests/layers. And them, import them in an registry without internet access. Maybe crane copy is not what I really want to use? And there is something else more approriate?

[imjasonh]

If crane does not rewrite the manifest, how would the client get that foreign reference without internet access?

Maybe I've got this wrong, but I though that --allow-nondistributable-artifacts intent was to copy all the manifests/layers from one registry to another, without leaving references to external registries.

Maybe I didn't made this clear, but my goal is to create an offline copy of all the manifests/layers. And them, import them in an registry without internet access. Maybe crane copy is not what I really want to use? And there is something else more approriate?

--allow-nondistributable-artifacts tells crane cp to ignore the foreign layer mediatype and fetch the content anyway, and pass that along to the destination registry. It doesn't rewrite the manifest to indicate the layer is no longer foreign, because that would change the manifest's digest, which we generally avoid outside of crane mutate.

ccing @jwhb who added the flag and @StevenLocke who added the behavior in #930 in case they have any more input/context to add.

[rgl]

@imjasonh, thanks for the explanation! its now clearer for me :-)

I also want to add that with the @mtrmac help at containers/skopeo#545 (comment), I was able to understand why the manifest upload was failing. By default a docker distribution registry is not configured to accept external references. After changing the default registry configuration, crane copy was able to upload the manifest. This is what I appended to the /etc/docker/registry/config.yml file:

validation:
  manifests:
    urls:
      allow:
        - .+

[imjasonh]

Thanks for digging into it, and for sharing the config that makes distribution accept this. We should probably document that somewhere, but I'm not sure where. In the meantime this issue is better than nowhere! 😅

[MrXhh]

@imjasonh, thanks for the explanation! its now clearer for me :-)

I also want to add that with the @mtrmac help at containers/skopeo#545 (comment), I was able to understand why the manifest upload was failing. By default a docker distribution registry is not configured to accept external references. After changing the default registry configuration, crane copy was able to upload the manifest. This is what I appended to the /etc/docker/registry/config.yml file:

validation:
  manifests:
    urls:
      allow:
        - .+

validation:
  disabled: true

config