Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache exporting / importing not working as expected - or just not working #723

Closed
brad-jones opened this issue Nov 16, 2018 · 13 comments
Closed

Comments

@brad-jones
Copy link

Because --cache-from does not yet work in docker I figured I would have a go at using buildkit directly.

I am trying to setup CI/CD infrastructure that will use buildkit in a distributed fashion, across many hosts.
I note there are a few issues tagged with distributed, these are somewhat over my head at the moment, perhaps there is something in these that I have missed, my apologies if so.

Here is an example case: https://gist.github.com/brad-jones/536064fcc180b62be6febfd2fa843024

I feel like I am having similar frustrations to #699 but in my case I am using the Docker Registry and not ECR.

Actually in production we are using nexus but that has other issues (buildkit won't pull images from it for a start) and does not fully support the v2 docker registry spec, hence why my example case uses registry

@tonistiigi
Copy link
Member

The first part of this is fixed with #730 (or just use 0.3.1). I need to validate if the rest of it now works as well or if there is another issue.

@tonistiigi
Copy link
Member

Sorry for a delayed response.

You can find the fork of your gist (with modifications) in https://gist.github.com/tonistiigi/1ba21a5bae3b9fd895b28589e57175e9 . All outputs seem correct there to me now.

Two issues with the old one:

  • 0.3.2 buildctl has an issue where it signals the daemon that it doesn't need cache for frontends. This is fixed in buildctl: fix frontends ignore-cache setting #730 . v0.3.1 also doesn't have an issue. I'll release v0.3.3 after that PR is merged.
  • --import-cache only worked with LLB builds. On frontend builds, cache import can be controlled by the frontend itself. The dockerfile frontend accepts remote cache with --frontend-opt cache-from=ref. I admit this is very unintuitive and I even missed it myself on first look so I made a PR buildctl: allow import-cache on frontend builds #731 that will automatically set the frontend-opt if --frontend is used from buildctl.

After these changes everything seemed ok to me.

Depending on the version of buildkit you use, you might notice a mismatch of the copy helper image when using docker build --cache-from=ref .. I'd suggest you using # syntax=docker/dockerfile to use the same external dockerfile implementation in both in case you need to use both docker and buildkit that have different versions. After that change all commands from your example script were also cached in docker 18.09 https://gist.github.com/tonistiigi/1ba21a5bae3b9fd895b28589e57175e9#file-_docker-out, even when built with older buildkit.

@brad-jones
Copy link
Author

Thanks @tonistiigi looks great, will take a look shortly.

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

@tonistiigi
Copy link
Member

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

Correct

@brad-jones
Copy link
Author

Just tested out v0.3.3 working perfectly. Thanks alot @tonistiigi

@pikeas
Copy link

pikeas commented Dec 23, 2018

Thanks @tonistiigi looks great, will take a look shortly.

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

@tonistiigi Is there an open issue tracking this? It's surprising that DOCKER_BUILDKIT=1 cannot be used with --cache-from.

@AkihiroSuda
Copy link
Member

@pikeas

tracked in #752

@virgile-hogman
Copy link

@tonistiigi @AkihiroSuda this is so confusing. If we can't use --cache-from with DOCKER_BUILDKIT=1 why is the option even allowed in the CLI?!

@virgile-hogman
Copy link

Or if this depends on what the image has been built with, there should be an error when trying to use an incompatible image. Otherwise how would the user make a difference between a format error or a simple cache miss?

@AkihiroSuda
Copy link
Member

19.03 supports --import-cache

@eedwards-sk
Copy link

eedwards-sk commented Dec 10, 2019

@AkihiroSuda

→ docker --version
Docker version 19.03.5, build 633a0ea
→ DOCKER_BUILDKIT=1 docker build --import-cache
unknown flag: --import-cache
See 'docker build --help'.

Is there a different argument I should be using?

@AkihiroSuda
Copy link
Member

Sorry I meant --cache-from

@zengqingfu1442
Copy link

Can you give a example code of how to use buildctl --export-cache? I just found the doc https://github.com/moby/buildkit#--export-cache-options but don't know how to use BuildKit with --cache-from together.
@tonistiigi @brad-jones Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants