Create podmanBuilder for Node Image Builds

[bagnaram]

What would you like to be added:
KIND_EXPERIMENTAL_PROVIDER=podman should utilize a podmanBuilder instead of the default dockerBuilder when performing a kind build node-image
Why is this needed:
in order o maintain parity across container runtime providers.

[BenTheElder]

That's not possible. The "docker" here is referring to the upstream Kubernetes build. Kubernetes's build uses buildx now and does not support podman (and never has, and other Kubernetes maintainers have voiced that we should not maintain support for multiple ways for building there).

[BenTheElder]

Also: --type was deprecated in v0.11, we're removing that concept entirely. We just support the one standard Kubernetes upstream build. We removed bazel from Kubernetes and from KIND.

kubernetes/enhancements#2420
https://github.com/kubernetes-sigs/kind/releases/tag/v0.11.0#breaking-changes

kubernetes/kubernetes#79048 (comment)

[bagnaram]

I ended up rolling my own kind node build successfully on podman and figured I would share what was done in case others wanted to see.

1. I symlinked ~/bin/docker to podman because the command is hardcoded in the kubernetes build scripts.
2. Created and applied the following patch to my local cloned kubernetes tag v1.21.1 kube-root:

From 46029e96e750e9239bdf8d7fcbe6a5dd09b3fbdd Mon Sep 17 00:00:00 2001
From: bagnaram <[email protected]>
Date: Thu, 27 May 2021 10:19:51 -0500
Subject: [PATCH] podman patches

---
 build/lib/release.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/build/lib/release.sh b/build/lib/release.sh
index d8fb1f3d..8226d826 100644
--- a/build/lib/release.sh
+++ b/build/lib/release.sh
@@ -379,10 +379,10 @@ function kube::release::create_docker_images_for_server() {
         ln "${binary_file_path}" "${docker_build_path}/${binary_name}"

         local build_log="${docker_build_path}/build.log"
-        if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" buildx build \
+        if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" build \
           -f "${docker_file_path}" \
           --platform linux/"${arch}" \
-          --load ${docker_build_opts:+"${docker_build_opts}"} \
+          ${docker_build_opts:+"${docker_build_opts}"} \
           -t "${docker_image_tag}" \
           --build-arg BASEIMAGE="${base_image}" \
           --build-arg SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE}" \
--
2.31.1

3. Execute kind node build while specifying local node image KIND_EXPERIMENTAL_PROVIDER=podman kind build node-image --base-image containers-storage:localhost/kindest/base:v20210526-96d954b3-dirty --kube-root ~bagnaram/go/kubernetes
   This produces an image in my local container storage:

$ kind (main) ✘ podman images
REPOSITORY                              TAG                           IMAGE ID      CREATED            SIZE
localhost/kindest/node                  latest                        ad9cc3d2a1a2  13 minutes ago     1.18 GB

[BenTheElder]

You should actually be able to set DOCKER env in the Kubernetes build scripts for other reasons (e.g. you want sudo docker).

Kubernetes will not accept the patch above though. We build with buildx specifically to ensure correctness of the metadata. There's a long history behind that.

The provider env in kind is for the nodes, it is intended to support running clusters only. Kubernetes does not support development of the project without docker. The kind project has simar requirements for our base image etc.

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/quick-start/#building-images
->
https://github.com/kubernetes/community/blob/master/contributors/devel/development.md#building-kubernetes-with-docker
(note the latter only says "with docker" because "with bazel" was an option for a portion of the project for some time. It is not now)

[bagnaram]

Thank you for the back story.
This is what I thought was interesting redhat-actions/buildah-build#27 and that the multiarch support is already implemented but extra logic would be required to break buildkit out as stand alone.

Although docker is the only official supported option, podman, or other container runtime based builds can be "at your own risk", which is why I shared the patch

[BenTheElder]

Appreciate it! I definitely think others will find this useful 😅

I also want to clarify that with respect to upstream Kubernetes while I happen to also be one of the build maintainers I would not single handedly make such a decision and that comes from others. But there's a long history there so I can be pretty confident based on that ...

For the kind project it's just very unusual to be building anything other than the node image anyhow where docker is currently necessary.

In the future we will try to select whatever available tool when we support building from existing releases as I'm certain the other aspects of the node image are portable*.

* excluding the multi-arch assembling in the push-node.sh wrapper we use for official images