Support WSL and minikube

[F21]

Expected behavior

Skaffold should work under WSL when used with minikube on windows

Actual behavior

Running skaffold dev in WSL (windows subsystem for linux) with minikube does not work at the moment.

This is because skaffold calls minikube docker-env --shell none to get the docker configuration, but the docker_cert_path points to a windows path:

DOCKER_HOST=tcp://192.168.1.113:2376
DOCKER_CERT_PATH=C:\Users\user\.minikube\certs
DOCKER_API_VERSION=1.23

Information

Skaffold version: version: a0879920e6ba86a997a02b4e28ea1e6b84dbdc46
Operating system: Windows 10 1804 64-bit with WSL
Content of skaffold.yaml: n/a

Some possible solutions:

• Detect if the environment is WSL and use wslpath to convert DOCKER_CERT_PATH into a linux path.
• Use the certificates configured in configuration in the ~/.kube folder.
• Implement configuration for skaffold (under ~/.skaffold?)

[remoe]

@F21, please try the master branch.

[F21]

@remoe I am not sure why, but master does not appear to be looking for minikube:

2018/06/20 09:46:42 Unable to read "/home/user/.docker/config.json", falling back on anonymous: open /home/user/.docker/config.json: no such file or directory
Starting build...
FATA[0009] first build: exiting dev mode because the first build failed: running build for artifact: running build: docker build: Cannot connect to the Docker daemon at tcp://0.0.0.0:2375. Is the docker daemon running?

My minikube.exe is available in my PATH and I have symlinked /usr/local/bin/minikube to minikube.exe:

$ which minikube
/usr/local/bin/minikube
$ which minikube.exe
/c/Minikube/minikube.exe

[remoe]

you don't need minikube on WSL. It should find your windows installation: "minikube.exe". you don't need a symlink.

[F21]

My minikube is not installed in WSL. It's installed in Windows, but because I prefer using 1 shell, I made the symlink so that I can execute minikube from WSL without typing minikube.exe.

[remoe]

do you have a wrong docker-env?

I can't reproduce on my side:

> uname -a
Linux x-PC 4.4.0-17134-Microsoft #112-Microsoft x86_64 x86_64 x86_64 GNU/Linux

> cat ~/.minikube/docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<ip>:2376"
export DOCKER_CERT_PATH="/mnt/d/Users/<username>/.minikube/certs"
export DOCKER_API_VERSION="1.35"

> cat ~/.kube/config

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /mnt/d/Users/<username>/.minikube/ca.crt
    server: https://<ip>:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /mnt/d/Users/<username>/.minikube/client.crt
    client-key: /mnt/d/Users/<username>/.minikube/client.key

> which minikube
> which minikube.exe
/mnt/d/apps/minikube/minikube.exe

[F21]

Thanks! That helped a lot.

My problem is that I did not have a .kube/config in WSL. It worked when I symlinked /c/Users/user/.kube to ~/.kube. However, because the paths inside .kube/config were Windows paths, WSL was not able to resolve them:

ARN[0005] error retrieving kubernetes client: Error creating kubeConfig: invalid configuration: [unable to read client-cert /home/user/.kube/C:\Users\user\.minikube\client.crt for minikube due to open
/home/user/.kube/C:\Users\user\.minikube\client.crt: no such file or directory, unable to read client-key /home/user/.kube/C:\Users\user\.minikube\client.key for minikube due to open /home/user/.kube/C:\Users\user\.minikube\client.key: no such file or directory, unable to read certificate-authority /home/user/.kube/C:\Users\user\.minikube\ca.crt for minikube due to open /home/user/.kube/C:\Users\user\.minikube\ca.crt: no such file or directory]
FATA[0005] first build: deploying manifests: exit status 1

Skaffold currently loads the kubernetes config here:

skaffold/pkg/skaffold/kubernetes/context/context.go

Line 33 in 64ecf0b

func CurrentConfig() (clientcmdapi.Config, error) {

A possible solution could be to transform the paths here https://github.com/GoogleContainerTools/skaffold/blob/master/pkg/skaffold/kubernetes/client.go#L32), however, I am not sure if that's the best way to do this.

The reason why I wanted to avoid creating a .kube/config inside WSL is because the IP address of the minikube VM changes when you delete and recreate the VM. This meant that I would have to update the kubernetes config in WSL when I delete and create the VM.

[r2d4]

@F21 You can also set KUBECONFIG environment variable to point to your kubeconfig that skaffold should resepect

[F21]

@r2d4 I am hoping that I can avoid maintaining 2 sets of kube configs (1 in WSL and 1 on Windows). The main reason is that when you delete a minikube machine and recreate it, the ip address can change. The kubeconfig in WSL would be out of date and skaffold would stop working.

However, if I symlink the kubeconfig on Windows into ~/.kube in WSL, the kube config is available, but the paths the certificates would be Windows paths, which would fail in WSL.

One other solution would be to run the Windows version of skaffold (using skaffold.exe). However, job control does not work in WSL yet, so if you terminate skaffold using ctrl + c, skaffold will be terminated straight away and will not be able to clean up the containers in kube.

[tstromberg]

As this issue is nearing its 2yr anniversary without much in the way of updates, I'm going to close this now as stale. I believe the right way to go about this is:

• Use minikube from WSL
• or add a flag to minikube to render the correct paths.