[FEATURE]: Override default garden buildkit flags (e.g. --cache-from)

[stefreak]

Feature Request

We need a way to override buildctl build flags. The default flags can be found here:

garden/core/src/plugins/kubernetes/container/build/buildkit.ts

Line 126 in 1bcdf7a

const command = [

You can already specify additional flags with extraFlags (which is not documented in the in-cluster building guide btw)

What's missing is the possibility to override default flags.

Background / Motivation

I was optimizing the Garden Cloud CD build performance. We have optimal Dockerfile layout, but caching is not optimal for multi-stage builds without mode=max. We also decided to store the final images on Amazon ECR, while we want the cache to be stored on ghrc.io for optimal performance and lower cost when using the cache in GitHub actions.

What should the user be able to do?

User should be able to overriding the existing --cache-from and --cache-to options.

It should be possible to use a different cache registry than the push registry. This is because some popular registries like Amazon ECR do not support the cache manifests needed for mode=max. So in our case we are using Amazon ECR as the push registry, and ghcr.io as the cache registry.

Why do they want to do this? What problem does it solve?

As of today, to get optimal build performance you need to use buildkit directly and spend loads of time to figure it all out.

And then you have to maintain duplicate code: Garden YAML and Some-CI-YAML both specify how to build your docker containers.

Also, you not only want optimal build performance in the CI: The developers should also get optimal cache hits on the inner loop experience and thus save loads of time.

Suggested Implementation(s)

Example module config:

kind: Module
type: container
extraFlags: ["--cache-to", "type=registry,mode=max,ref=gchr.io/something/foo:buildcache", "--cache-from type=registry,ref=gchr.io/something/foo:buildcache"]

Open question

• If cache registry differs from push registry, how can we pass the login secrets for the cache registry?

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

We are not using garden for CI builds, but we can still use garden workflows etc. It is just painful not being able to have the speed mode=max offers.

[vvagaytsev]

Thanks, @stefreak! We'll take a look at this after #3114.