Add option to provide Kubernetes cluster for testing

[ktff]

With #1970, running Kubernetes tests with make will start Kubernetes cluster using kind which is known to being slow at times which can trigger timeouts in out tests.

One of the solutions for this is to enable usage of user provided cluster.

This would solve the issue of timeouts, and enable easier testing on different clusters.

Main issue with this is how to robustly provide builded image, of vector binary to be executed, to the provided Kubernetes cluster.

One way of doing it is by requesting the user to provide a registry to which the image can be pushed.

[MOZGIII]

One way of doing it is by requesting the user to provide a registry to which the image can be pushed.

I like this actually. Currently, we spin-up the localhost registry and use that by default. If we provide a way to specify custom docker image/tag - user would be able to pass their existing registries. If we also provide an extension point to integrate with something like ngrok (and/or frp or ssh tunnels) to expose the docker-registry we run - we'll cover even more cases, since not everyone can use an external registry.

[ktff]

So, there are 4 different components/steps to testing Kubernetes:

1. Providing a repository
2. Building and pushing Vector image
3. Providing a cluster
4. Running the tests

We could expose the script on three different granularitis:

1. Largest granularity, as it's now, which does all steps.
   - If kind works for you, this is the simplest way of testing.
   - Once we find a substitute for kind this would be even more useful.

2. Medium, the script will do 2. and 4. step and the user would provide 1. and 3.
   - This would enable easier testing if 1. isn't working.
   - It would also make testing on different clusters easier.
   - This is bunched together since cluster needs to be configured to be able to use private repository, so we can't just start local repository. (Minikube is maybe an exception)

3. Smallest, the script would just do 2. step.
   - Useful for more faster testing/debugging of one test

@MOZGIII @LucioFranco what do you think?

Also what are your experiences with different Kubernetes cluster meant for testing? Like minikube, microk8s, kind, and the like.

[LucioFranco]

The major thing we need implemented is this some script that can build vector, push an image to some registry, and run a set of integration tests against some kube cluster. The config for this should be some kubeconfig and a registry url. This should then allow us to test against any kube cluster whether it is kind, minikube or an actual cluster running somewhere in the cloud. I would not worry about the script setting up a cluster because locally that is very trivially done via minikube.

[ktff]

I agree, then to keep it simple we could only support that.

[MOZGIII]

I think it's cool that we can provide a registry. It would be great to have a script for that, maybe to run ad-hoc.

We can also consider using an empty container for vector, and copying compiled files into it before running the test (skaffold/ksync style). This way we don't need a registry.

Also what are your experiences with different Kubernetes cluster meant for testing? Like minikube, microk8s, kind, and the like.

I've been using minikube mostly. k3d looks interesting though.

[MOZGIII]

Also, I'm looking for the tooling that I could call manually. I'm not opposed to having a script that implements a "whole process", but it would be great if it simply outlines the logic while being composed of smaller individually callable components. I.e. so that I can invoke building vector docker image manually.
The reason I want that is I feel like having make kubernetes-tests, while being an excellent command to run the tests, is not the only thing I'd want to do while working on k8s-related functionality. From practice, I had to run vector manually to test some of my assumptions regarding a bug. This kind of troubleshooting would be simplified if I had a set of tools for manual use to do what make kubernetes-tests does internally.
This is on a nice-to-have level though.

[LucioFranco]

I don't think we have users running enough tests for kube to make that worth it. Ideally, CI should cover the majority case and users that need to do so locally should be able to stand up what they need.

[MOZGIII]

The utilities are not for our users, they're for us.

CI should cover the cases that CI is built for, however, when developing, if I want to try to run the code I make at k8s I have to build the image myself.

So, I'm all about having an option to do

bin/build-docker-image mozgiii/vector-tests:mytest taget/release/vector

instead of

docker build -t "mozgiii/vector-tests:mytest" -f - . << EOF
FROM buildpack-deps:18.04-curl
COPY ./taget/release/vector /usr/local/bin
ENTRYPOINT ["/usr/local/bin/vector"]
EOF

Take a look at scripts/kubernetes-test.sh, it has a bunch of cool useful subroutines that could be easily exposed as commands and then reused.

[LucioFranco]

@MOZGIII that seems totally fine, I would say that we can expect developers to have docker and a valid kubeconfig. From there we can write a script as you said to build the image and run tests.

[MOZGIII]

Fixed via #2702.