How to export cache in buildkit

[ms-choudhary]

I can't seem to figure out how to export cache from buildkit to registry/local. There seems no documentation around on how to do that.

I tried following unsuccessfully:

bash-4.4# buildctl build --export-cache registry --export-cache-opt name=docker.io/mschoudhary/hello-world --export-cache-opt push=true --frontend=dockerfile.v0 --local context=. --local dockerfile=. 
[+] Building 6.2s (10/10) FINISHED
 => [internal] load .dockerignore                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                                                                                         0.0s
 => => transferring dockerfile: 32B                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/golang:alpine                                                                                                             2.8s
 => [internal] helper image for file operations                                                                                                                              0.0s
 => => resolve docker.io/tonistiigi/copy:v0.1.7@sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089                                                      0.0s
 => [build 1/3] FROM docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                 0.0s
 => => resolve docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                       0.0s
 => [internal] load build context                                                                                                                                            0.0s
 => => transferring context: 28B                                                                                                                                             0.0s
 => CACHED [build 2/3] COPY main.go .                                                                                                                                        0.0s
 => CACHED [build 3/3] RUN go build -o /bin/hello .                                                                                                                          0.0s
 => CACHED [release 1/1] COPY --from=build /bin/hello .                                                                                                                      0.0s
 => ERROR exporting cache                                                                                                                                                    3.4s
 => => preparing build cache for export                                                                                                                                      0.0s
 => => writing layer sha256:3aaf202967879d295dd06275ef7322bc20087424938d4fbcf196cf2d693c1d4e                                                                                 3.3s
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing layer blob: server message: insufficient_scope: authorization failed

There is no indication of what's wrong here. (I'm able to successfully push image to registry) Would be grateful if someone could help with this.

Buildkit client version:

bash-4.4# buildctl --version
buildctl github.com/moby/buildkit 0.0.0 fb5324c609465f9b0713cbce5f8a36eb119be144

Buildkit daemon version:

bash-4.4# buildkitd --version
buildkitd github.com/moby/buildkit 0.0.0 fb5324c609465f9b0713cbce5f8a36eb119be144

[tonistiigi]

buildctl build --export-cache docker.io/mschoudhary/hello-world --export-cache-opt mode=min|max

Note that this syntax may change soon #615

[ms-choudhary]

@tonistiigi Thanks for prompt reply.

However, I've two problems:

• When I use AWS ECR registry, export-cache fails (However, its working perfectly fine for docker hub):

 => ERROR exporting cache                                                                                                                                                  106.6s
 => => preparing build cache for export                                                                                                                                      0.1s
 => => writing layer sha256:1e24609eb02c3418f52959533d62690ce5e5e5b5be2ef82a027ff3dc88bdeadc                                                                                84.8s
 => => writing layer sha256:2e793f0ebe8a0d4496ef2fca6d5d45493df6dc40503657b357e153fa91548171                                                                                 0.2s
 => => writing layer sha256:37a0800ca4c6854b1d735c4f4c4527702e24847325a190dae95b9b7cfb6f35ae                                                                                 0.2s
 => => writing layer sha256:387d73a1a3a0156951154c2b802c9c65cbe51fa032e9e3687f1af41567e68c4f                                                                                 0.2s
 => => writing layer sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde                                                                                 1.4s
 => => writing layer sha256:77995fba19184aad4b702852da2d04d7b92693f2923499f5d388fd8d2ce2dd81                                                                                 0.2s
 => => writing layer sha256:b85d63701330edd42642a6aa842dde00c2a41078a38db9d9eb8a9fa4d67f7563                                                                                 0.8s
 => => writing layer sha256:bb56b1a33aae9f2198007c10de128ed7ee4c3b4a0c0232106d6daf82d64a28cd                                                                                17.4s
 => => writing layer sha256:ca7a931ff4ef28db137d3a6ad858079260b4300645f31a022873184acbd8a88c                                                                                 0.2s
 => => writing config sha256:da380c99b3c020b04115f7d0743ee31d0702d728edda9db71105173f15b5f9ac                                                                                1.0s
 => => writing manifest sha256:b6d8f0c6d2e2d0b7453adaaf026e83413fbe07666cb7e6f22aee33e51975f585                                                                              0.3s
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:b6d8f0c6d2e2d0b7453adaaf026e83413fbe07666cb7e6f22aee33e51975f585": unexpected status: 400 Bad Request

• Buildkit does not seem to pull from cache, even if import-cache option is used.

bash-4.4# buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --import-cache mschoudhary/hello-world:cache-master --exporter=image --exporter-opt name=mschoudhary/hello-world:master --exporter-opt push=true --export-cache mschoudhary/hello-world:cache-master --export-cache-opt mode=max
[+] Building 199.6s (13/13) FINISHED
 => importing cache manifest from docker.io/mschoudhary/hello-world:cache-master                                                                                             5.8s
 => [internal] load build definition from Dockerfile                                                                                                                         0.0s
 => => transferring dockerfile: 278B                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/golang:alpine                                                                                                             2.9s
 => [build 1/4] FROM docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                 6.5s
 => => resolve docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                       0.0s
 => => sha256:37a0800ca4c6854b1d735c4f4c4527702e24847325a190dae95b9b7cfb6f35ae 110.17MB / 110.17MB                                                                           0.0s
 => => sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b 2.04kB / 2.04kB                                                                               0.0s
 => => sha256:1d9eb297527b4a67366f25219175217c11aff5901c44dc3349d3f3405585f951 1.36kB / 1.36kB                                                                               0.0s
 => => sha256:387d73a1a3a0156951154c2b802c9c65cbe51fa032e9e3687f1af41567e68c4f 126B / 126B                                                                                   0.0s
 => => sha256:2e793f0ebe8a0d4496ef2fca6d5d45493df6dc40503657b357e153fa91548171 308.97kB / 308.97kB                                                                           0.0s
 => => sha256:95ec94706ff6cd099d2cba25cf80961b94dce516a029a2713449314717164465 3.80kB / 3.80kB                                                                               0.0s
 => => sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde 2.21MB / 2.21MB                                                                               0.0s
 => => sha256:77995fba19184aad4b702852da2d04d7b92693f2923499f5d388fd8d2ce2dd81 153B / 153B                                                                                   0.0s
 => => unpacking docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                     6.4s
 => [internal] helper image for file operations                                                                                                                              4.8s
 => => resolve docker.io/tonistiigi/copy:v0.1.7@sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089                                                      1.1s
 => => sha256:08bf20359e34f0e859ed9251ede8e5600e3c0319a0d0875008058b958aff30e9 300.75kB / 300.75kB                                                                           3.1s
 => => sha256:572dd9fa83b20b238273b0eb0921b18e2d8642be58bb367657f1bd71c2ccf0f3 172.86kB / 172.86kB                                                                           3.1s
 => => sha256:b16fcda8ffa72fc280319967379a6b061ae8684f335d5df662160754c43ce433 1.38kB / 1.38kB                                                                               0.0s
 => => sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089 2.04kB / 2.04kB                                                                               0.0s
 => => sha256:f2f4298def7b9af8b5bef79488d0eb14beade2db743a08504315865b3010dd52 1.36kB / 1.36kB                                                                               0.0s
 => => sha256:0b9e0c15d44a08746fabce2a371538e8981e36e897fd028ff5f43a5822d5122b 859.51kB / 859.51kB                                                                           4.4s
 => => sha256:05fb0b6da3085c804381e8c0f9cd54e59a65f0a7802ac60ce276ba20629f5ed2 70.73kB / 70.73kB                                                                             3.3s
 => => sha256:5c6c674792ce138a9058c97a5ef6df32417a80bf42664774d54ce1ada74502f9 98.70kB / 98.70kB                                                                             4.1s
 => => unpacking docker.io/tonistiigi/copy:v0.1.7@sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089                                                    0.3s
 => [internal] load build context                                                                                                                                            0.0s
 => => transferring context: 106B                                                                                                                                            0.0s
 => [build 2/4] RUN apk add --update curl wget vim bash bind-tools python3                                                                                                  21.4s
 => [build 3/4] COPY main.go .                                                                                                                                               0.5s
 => [build 4/4] RUN go build -o /bin/hello .                                                                                                                                 0.9s
 => [release 1/1] COPY --from=build /bin/hello .                                                                                                                             0.4s

Same when run with docker cli (after exporting DOCKER_BUILDKIT env) is pulling from cache:

➜  hello-world git:(master) ✗ docker build -t mschoudhary/hello-world:master --cache-from mschoudhary/hello-world:cache-master .
[+] Building 39.3s (11/11) FINISHED
 => local://context (.dockerignore)                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                              0.0s
 => local://dockerfile (Dockerfile)                                                                                                                                          0.0s
 => => transferring dockerfile: 37B                                                                                                                                          0.0s
 => importing cache manifest from docker.io/mschoudhary/hello-world:cache-master                                                                                             4.7s
 => docker-image://docker.io/library/golang:alpine@sha256:ccb871525d23b7ae5be6a159841fb3b4b5dc5c789ce36489bd9ef83c9ca0b93b                                                   0.0s
 => CACHED docker-image://docker.io/tonistiigi/copy:v0.1.3@sha256:e57a3b4d6240f55bac26b655d2cfb751f8b9412d6f7bb1f787e946391fb4b21b                                           0.0s
 => local://context                                                                                                                                                          0.0s
 => => transferring context: 28B                                                                                                                                             0.0s
 => CACHED /bin/sh -c apk add --update curl wget vim bash bind-tools python3                                                                                                25.0s
 => => pulling sha256:61a64bb611c0f7cfddce601634c1ff38cde2ab50dfaf438520d48b7bc2eddb16                                                                                      23.0s
 => copy /src-0/main.go go/src/code.scripbox.io/devops/hello-world/                                                                                                          0.6s
 => /bin/sh -c go build -o /bin/hello .                                                                                                                                      1.2s
 => CACHED copy /src-0/hello ./                                                                                                                                              0.0s
 => exporting to image                                                                                                                                                       0.0s
 => => exporting layers                                                                                                                                                      0.0s
 => => writing image sha256:c964880ed22a6122c1536e19e0dbefc6d5fe34028d6f965e10bc724989168ed9                                                                                 0.0s
 => => naming to docker.io/mschoudhary/hello-world:master                                                                                                                    0.0s

This is the dockerfile:

FROM golang:alpine as build
WORKDIR /go/src/code.scripbox.io/devops/hello-world
RUN apk add --update curl wget vim bash bind-tools python3
COPY main.go .
RUN go build -o /bin/hello .

FROM scratch as release
COPY --from=build /bin/hello .

Is there any option that I'm missing with buildkit?

As we don't have any way of exporting cache via docker cli at present, I wanted to use buildctl. Caching per build is super important in my use case.

Great job BTW. Awesome tool. :-)

[tonistiigi]

Buildkit does not seem to pull from cache,

Are you sure nothing in your build changed that would have caused the miss? If not please open issue with a reproducer.

Looking at the output, one difference I see is the difference of the copy image (although the first cache mismatch is before that). You may want to use a fixed dockerfile implementation with #syntax= directive if you move between different implementations to avoid the copy helper image changing (this limitation will go away with #396).

When I use AWS ECR registry, export-cache fails (However, its working perfectly fine for docker hub):

Maybe you can report this to ECR. There isn't anything special that registry should need for this. It is pushing a regular manifest list. Added to the opensource registry codebase in Docker 1.10

[ms-choudhary]

Ok. So I was debugging why AWS ECR is giving 400 bad request.

When we push a normal image manifest, this is the request:

time="2018-10-25T02:39:15Z" level=debug msg="do request" digest=sha256:ed15c30778039aeadc534c1c18cdd56be82c7b6f6087aeed2648e5d0880e673b mediatype="application/vnd.docker.distribu
tion.manifest.v2+json" request.headers=map[Content-Type:[application/vnd.docker.distribution.manifest.v2+json]] request.method=PUT size=526 url="https://826XXXXXXXX.dkr.ecr.ap-s
outheast-1.amazonaws.com/v2/hello-world/manifests/master"

Content type : application/vnd.docker.distribution.manifest.v2+json
This passes with 201 Created.

This is the request generated when we push cache image manifest:

time="2018-10-25T02:40:21Z" level=debug msg="do request" request.headers=map[Content-Type:[application/vnd.docker.distribution.manifest.list.v2+json]] request.method=PUT url="https://826XXXXXXXXXX.dkr.ecr.ap-southeast-1.amazonaws.com/v2/hello-world/manifests/cache-master"

Content type : application/vnd.docker.distribution.manifest.list.v2+json
This fails with 400 Bad request

I tried to send same request manually, this is the error I'm getting:

{
    "errors": [
        {
            "code": "UNSUPPORTED",
            "message": "Unsupported manifest type: \"application/vnd.docker.distribution.manifest.list.v2+json\""
        }
    ]
}

Is this something that is expected to be supported by the registries? If yes, will open a ticket on AWS ECR team.

[tonistiigi]

Is this something that is expected to be supported by the registries?

Yes, see
https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#media-types

[samuelkarp]

Amazon ECR does not currently support manifest lists, but we are planning to add support (see here).

[ms-choudhary]

Sure. Closing this issue.

[dreyks]

thx @samuelkarp
any particular timeframe available?

[eedwards-sk]

@samuelkarp Your linked thread is dead with no replies. Any updates?

[samuelkarp]

@dreyks @eedwards-sk You can follow aws/containers-roadmap#505 for updates.