new driver: kubernetes

[AkihiroSuda]

Tested with kind and GKE
TODOs:

• Fix No Auth Provider found for name "gcp" error on GKE
• Support consistent hashing
• Use watch API instead of polling (maybe follow-up PR)
• Use exec API instead of executing kubectl binary
• Support rootless
• Support specifying kubeconfig via the current docker context
• Change driver name from kube to kubernetes for docker CLI consistency

Note: "nodes" shown in docker buildx ls are unrelated to Kubernetes "nodes".
Probably buildx should come up with an alternative term.

Usage:

  $ kind create cluster
  $ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

  $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use
  $ docker buildx build -t foo --load .

--load loads the image into the local Docker.

Driver options

• image=IMAGE - Sets the container image to be used for running buildkit.
• namespace=NS - Sets the Kubernetes namespace. Defaults to the current namespace.
• replicas=N - Sets the number of Pod replicas. Defaults to 1.
• rootless=(true|false) - Run the container as a non-root user without securityContext.privileged. Using Ubuntu host kernel is recommended. Defaults to false.
• loadbalance=(sticky|random) - Load-balancing strategy. If set to "sticky", the pod is chosen using the hash of the context path. Defaults to "sticky"

Signed-off-by: Akihiro Suda [email protected]

Close #22

[tonistiigi]

Note: "nodes" shown in docker buildx ls are unrelated to Kubernetes "nodes".
Probably buildx should come up with an alternative term.

Would it make sense that if you do --replicas 3 then 3 nodes would automatically be generated under the builder instance?

[AkihiroSuda]

That doesn't seem to work well with Deployment lifecycle

[tonistiigi]

@AkihiroSuda Why? They would all be running or inactive based on if the deployment is running or not. Can't we even have it so that the consistent hashing (optionally) works if you just add two container nodes manually?

[AkihiroSuda]

You need to synchronize buildx state with Kube Deployment when pods are rescaled (by either user or HPA) or recreated.

Also, it seems hard to bind pod lifecycle to buildx node without manually managing pods (i.e. reinventing Deployment) or using StatefulSet which is slow.

[AkihiroSuda]

Can we rename buildx "Nodes" to "Instances"?

[AkihiroSuda]

Or just add a document to clarify buildx "Node" is unrelated to Kubernetes "Node".
Same applies to Swarm-mode "Node" if we are going to add support for docker service create driver as well

[AkihiroSuda]

@tonistiigi Is this different from moby/buildkit:v0.6.2?

[tonistiigi]

It is atm, but it is a different tag in case we need to break compatibility in the future. Eg. moby/buildkit:v0.7.0 might be equal to moby/buildkit:buildx-stable-1 or moby/buildkit:buildx-stable-2 depending if it is compatible.

[AkihiroSuda]

Ready for review

[tonistiigi]

Can we rename buildx "Nodes" to "Instances"?

Instance is what we call one named builder already. Instance can have many nodes.

[AkihiroSuda]

Subinstance?

[AkihiroSuda]

How can we move this forward?

[AkihiroSuda]

added new (hacky) commit: kubernetes: show Kubernetes Pods as buildx "Nodes" in docker buildx inspect

$ k get pods
NAME                     READY   STATUS    RESTARTS   AGE
kube0-75897b8444-dnfhg   1/1     Running   0          2m
kube0-75897b8444-dsfht   1/1     Running   0          2m
kube0-75897b8444-ghthr   1/1     Running   0          2m

$ docker buildx inspect
Name:   kube
Driver: kubernetes

Nodes:
Name:      kube0-75897b8444-dnfhg
Endpoint:  
Status:    running
Platforms: linux/amd64, linux/386

Name:      kube0-75897b8444-dsfht
Endpoint:  
Status:    running
Platforms: linux/amd64, linux/386

Name:      kube0-75897b8444-ghthr
Endpoint:  
Status:    running
Platforms: linux/amd64, linux/386

[tiborvass]

I'm merging, we can improve in followups.