feasibility-research(doc): Remote build with buildkit or kaniko

[gaocegege]

Description

We should investigate how to build envd images remotely with buildkit or kaniko. And we should document it.

• Add doc for the experience

CI/CD systems to support:

• GitHub Action
• GitLab CI

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[gaocegege]

/assign

[gaocegege]

It is hard to support envd with kaniko since it does not support dockerfile/1.2.0 or 1.3.0 now. RUN --mount=cache is hard to run.

GoogleContainerTools/kaniko#1712

[gaocegege]

Currently, we support context in #512 . In the next PR we should support kube-pod builder in the context.

[gaocegege]

We need to decide if we should use kubernetes client-go or controller-runtime client to create the buildkitd pod on Kubernetes.

I think the client in controller-runtime wraps client-go a lot and it should be easier to use.

WDYT @hezhizhen @knight42

[knight42]

I think the client in controller-runtime wraps client-go a lot and it should be easier to use.

@gaocegege Yeah the client provided by controller-runtime is a general one and is more flexible(if we need to access some custom CRs, we only need to register theire scheme).

The only limitation came to my mind is that if we need some functionalities more than objects' CRUD, such as executing a command in container, streaming container logs, port-forwarding etc, then we could only use client-go.

[gaocegege]

Thanks for the explanation! forget that the controller-runtime client does not support sub-resources. I think we should use client-go directly.

[arcosx]

Hi, I've been working on something similar recently.I also think it is possible to use client-go directly.
As I see https://github.com/docker/buildx/blob/57156ee95cccc172e3757520283b04d09f1646e0/driver/kubernetes/driver.go#L192 and https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/blob/3cdb0965332177c4f254ea81b96c9ce7b8c46d3d/pkg/driver/kubernetes/driver.go#L229 have related work in them that is similar to your needs.
They both do it by passing in the client-go and reading the local kubeconfig.It actually create a tcp channel between the local and Pod.

[gaocegege]

Yep. Currently, we support communicating with the buildkit on the Kubernetes with the help of context:

envd context create --name test --builder-name <pod-name> --use --builder kube-pod

Next step is to support envd up with the help of client-go.

[aseaday]

It this issued related to #556
I want to solve that issue but there seems to be a lack of specific context.