Consider a very shallow 'buildx' "emulation mode"

[markusthoemmes]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Steps to reproduce the issue:

Run any docker buildx build script/command and see it fail. In my case, I'm trying to run kind build node-image using the podman provider (docker not installed at all)

Describe the results you received:

The command fails immediately.

Describe the results you expected:

I kind of expected podman to at least rudimentary support buildx build in that it is able to output builds for a specific platform the same way that buildx is. Buildah doesn't quite support the entire set of options on buildx just yet (see containers/buildah#3268), but for very simple scenarios, we should be able to fairly easily bridge the gap.

In fact, I was able to get the Kubernetes build (what runs as part of kind build node-image) to function just fine by adding the following bits to the /usr/bin/docker bridge script:

if [ "$1" = buildx ]
then
    if [ "$2" != build ]
    then
        echo "ERROR: 'buildx $2' is currently not supported. Only 'buildx build' is." >&2
        exit 1
    else
        echo "Best effort 'buildx build' emulation. Not all flags are supported." >&2
    fi
fi

# Drop buildx subcommand and flags we support "by default" to basically call 'podman build'
# directly.
for arg do
    shift
    [ "$arg" = "buildx" ] && continue
    [ "$arg" = "--load" ] && continue
    set -- "$@" "$arg"
done

They basically drop buildx so that build is called directly and also drop the "redundant" --load parameter (which podman doesn't know but it's its default).

I'm wondering if others would find this useful as well, hence leaving it here as a feature request. Happy to send a PR if y'all think this'd be useful in general.

[mheon]

@rhatdan @nalind PTAL, I know we were looking at this.

[rhatdan]

First let's look at the easiest and make build and buildx synonyms.

[rhatdan]

What does the --load option do?

We are adding some features of buildx into buildah such as the RUN --mount options.
containers/buildah#3409

[markusthoemmes]

Regarding --load: https://docs.docker.com/engine/reference/commandline/buildx_build/#load

Shorthand for --output=type=docker. Will automatically load the single-platform build result to docker images.

So basically what podman does by default, we just need to "swallow" it and not return an "unknown flag" error.

[rhatdan]

It would seem that we need to add a command then buildx and have it screan out certain commands and call what podman build calls.

[markusthoemmes]

Yep, that's why I've thrown it into the docker script, since it felt like it'd pollute podman itself unnecessarily.

[rhatdan]

I actually just updated the PR with some clever (In my opinion) changes to get you what you want, without us having to formally state we support buildx.

[afbjorklund]

kind uses docker buildx (BuildKit) so that it can cross-build for multi-arch (with qemu), which docker build doesn't support.

e.g.
--platform=linux/amd64,linux/arm64

kubernetes-sigs/kind@6d5ec59#diff-879e8c18c989850cae30c8983a392d6472e6df2f893ac8c576dd0762a572c5d8

I'm trying to run kind build node-image using the podman provider (docker not installed at all)

I think using buildah will only work for the "quick" build (for the local arch only), but not for the base image (just yet, anyway) ?

no image found in manifest list for architecture amd64,linux, variant "arm64", OS linux

---

EDIT: Added an issue for it (i.e. multi-arch with --platform), in case it becomes more common now: #11156

It doesn't actually work with Docker either (only with BuildKit), but could use a better errror in Podman

docker buildx build (--platform) error: multiple platforms feature is currently not supported for docker driver

docker buildx build (--load) error: docker exporter does not currently support exporting manifest lists

So it's fine to not support multi-arch (i.e. until Buildah does), it just needs to say so with an error message.

[afbjorklund]

(some semi-related side information)

Both kubernetes and minikube have switched from Docker (build) to BuildKit (buildx), to be able to build multi-arch more easy.

Before it would use multiple separate builds from docker build, and then stitch them together with docker manifest...
Now everything is done with the --platform flag, which involves tricky stuff like cross-building with a Windows worker etc

Then there are some more trivial flags like --load and --progress, which can probably be ignored and use the defaults.

---

It is possible to use BuildKit without Docker, we do this (buildctl) for minikube when using the containerd container runtime.

When using the cri-o container runtime, we just call podman build since there has been no need for using buildah yet.
It also makes it somewhat easier to talk to the daemon remotely, even if that is not needed anymore with our "image" wrapper.

Normally we would just rely on CRI and crictl, but unfortunately this API is still missing the load and build features.

[markusthoemmes]

@afbjorklund I agree this only works for the "quick" local build of the image. I thought it'd be neat to support that though, as it unlocks being productive working on Kubernetes (through kind at least) without installing docker at all.

[afbjorklund]

I suppose you could ask kind to run buildx build for the base and build for the (quick) node, until the syntax is supported.

There are also a bunch of pre-requirements for it, that are hidden in the setup script (experimental CLI, creating a builder, etc)

[Luap99]

The PR from @rhatdan is merged so I think this can be closed.