mode=max for multi-stage build BuildKit

[antoinelyset]

Hey Garden,

I might be wrong but currently, when building the Docker image, we only push the cache of the final image. If you use a multistage build it isn't optimal.

Command example:

Execing command in gh-run-3038213701-attempt-1-7b8477d5c/Pod/garden-buildkit-6bd5f49f85-8dhzg/buildkitd: buildctl build --frontend=dockerfile.v0 --local context=/garden-build/14124024-b248-443f-8365-69f2afa06b6c/api --local dockerfile=/garden-build/14124024-b248-443f-8365-69f2afa06b6c/api --opt filename=./api/Dockerfile --output type=image,"name=eu.gcr.io/slite-development/api:v-5f2ce1810f,eu.gcr.io/slite-development/api:_buildcache",push=true --export-cache type=inline --import-cache type=registry,ref=eu.gcr.io/slite-development/api:_buildcache --opt build-arg:GARDEN_MODULE_VERSION=v-5f2ce1810f --opt target=production

I think Garden should use mode=max option as described here

https://github.com/moby/buildkit#export-cache

What do you think?

[stefreak]

@antoinelyset absolutely! Thank you for reporting this. I've been thinking about this a lot recently. This is definitely something we want to implement.

One issue is that some registries do not support the manifest format required for mode=max (more specifically AWS ECS, see also aws/containers-roadmap#876).

Would you kindly give us feedback about the approach described in #3110 ? Would that approach work for you? Would you prefer that we auto-detect compatibility and just use mode=max if possible?

Thanks a lot for your time spent on this.

[antoinelyset]

Hey! Thanks for the quick answer. The auto-detect implementation would be nice for us (we wouldn't need to reimplement --cache-to and --cache-from) but having other options is also nice.

I'm glad you are working on this because it's a big slowdown for us :D

[antoinelyset]

Something that might also go in favour of the override is that we sometimes don't want to push the cache of a build. For example, in the context of a CI build on a feature branch we don't want to override the cache of our main branch.

[stefreak]

@antoinelyset that is actually a great thought, thanks for the idea.

actually, buildkit supports multiple --import-cache flags, and we could use a different cache tag depending on the branch, while trying both (so that the first build in a feature branch benefits from the main branch cache). I'll think about this a little bit.

[stefreak]

@antoinelyset how do you feel about something like the following example config, would having a cache config option like this make it customizable enough for you? (Thanks @TimBeyer for pairing on this, was fun :))

  clusterBuildkit:
      # feature-branch / master use case
      # this cache config would ensure good cache hit rate both on master builds and feature branch builds
      # only exporting to feature branch cache tag makes sure not to pollute the main branch cache
      cache:
        - type: registry
          mode: max
          tag: _buildcache-${camelCase(git.branch)}
        - type: registry
          mode: max
          tag: _buildcache-main
          export: false

resulting buildkit args would be:

--output type=image,registry/image:v-xxxxx
--import-cache type=registry,ref=registry/image:_buildcache-featureBranch,mode=max
--import-cache type=registry,ref=registry/image:_buildcache-main,mode=max
--export-cache type=registry,ref=registry/image:_buildcache-featureBranch,mode=max

[antoinelyset]

Seems like a fine idea!

[stefreak]

@antoinelyset Thanks for your feedback on this.