[FEATURE]: Support multi-platform builds with local-docker mode (enable BuildKit)

[ivanbabenko]

Feature Request

Background / Motivation

Multi-architecture/multi-platform support for container images is already and will be a big topic for many companies that start switching to use ARM worker nodes or ARM developer machines (and inevitably mixing with existing legacy x86).
Garden is a great tool to automate local image build and its deploy to a remote cluster, but there will be more often cases when architectures of developer's machine and cluster infrastructure differ.

What should the user be able to do?

Garden user should have possibility to easily and in one command:

1. Build an image for a platform other than a platform of the machine where Garden is executed (by specifying --platform=arch on extraFlags of container module) – currently not possible out of the box without creating docker alias to use buildx as a default docker builder.
2. Build a multi-platform images (by specifying --platform=arch1,arch2 on extraFlags of container module) together with additional BuildKit flags required to push result images to remote – currently fails due to Garden expecting to tag and push images that being already pushed by BuildKit to remote without storing locally.

Why do they want to do this? What problem does it solve?

This problem can be solved only be allowing multi-platform builds on all of the tools involved in SDLC (from local to CI and infrastructure), and since Garden is the tool that brings local development closer - I believe this functionality is crucial for it.

The problem can be solved currently only by using in-cluster build mode (which already uses BuildKit), however, for those who want to use local builds, BuildKit is not supported yet natively. I've been able to do some workarounds to force Garden to use BuildKit for local builds, but since it already takes care of pushing multi-platform images to remote and local images are not present (support by Docker is planned though), default steps on Garden to tag local image and push it after build are redundant and fail the overall garden build (though images are pushed by BuildKit successfully).

Suggested Implementation(s)

Idea is to add a new localBuildKit: true|false property on kubernetes provider config to rely on local buildx builder when it is set (comes in Docker Engine by default and supported by Docker CLI included in Garden already), and update localBuild build handler to produce multi-platform images based on --platform=arch1,arch2 flag and other required flags (e.g. --push or --load) as container module extraFlags. New local flow for BuildKit should also use different tagging and skip docker tag and docker push commands used in local-docker mode by default (BuildKit expects full final tag and pushes to remote without storing images locally).

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[vvagaytsev]

@ivanbabenko thanks for reporting this! We're in the middle of the big internal refactoring now -- it is #3114.

We'll take a look after 0.13 release is ready.

[stefreak]

As a work around, it is already possible to add extraFlags

for example:

kind: module
[..]
extraFlags: [ "--platform", "linux/amd64" ]

Most important thing missing is to include the architecture in the calculation of the garden hash / tag. Otherwise weird things will happen if your container modules depend on each other and cached tags are present with a different platform.

[stefreak]

To clarify: With high priority we need to solve the issue of the version hash being the same, even if the image is being built using another architecture.

Support for multi-arch images is important too, but not as important as solving the version hash issue.

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[stefreak]

solved in #6208; Version hash issue if sharing a registry with ARM and AMD64 users and no platforms specified still exists though. Needs a separate issue.