[BUG] k3d does not honor DOCKER_TLS

[mkubaczyk]

What did you do

I'm having Docker with TLS enabled on Gitlab CI:

  DOCKER_TLS: 1
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  DOCKER_HOST: "tcp://localhost:2376"

Then, when running simple cluster creation command:

bash-5.1# DOCKER_HOST="tcp://localhost:2376" DOCKER_CERT_PATH="/certs" DOCKER_TLS=1 DOCKER_CERT_PATH="/certs/client" k3d cluster create

ERRO[0000] Failed to get nodes for cluster 'k3s-default': docker failed to get containers with labels 'map[k3d.cluster:k3s-default]': failed to list containers: Error response from daemon: Client sent an HTTP request to an HTTPS server. 

I'm getting Error like k3d would be ignoring the env vars I'm using to let it know it's via TLS.

What did you expect to happen

k3d should honour these env vars making the docker runtime work via TLS.

Having simple test being run in the same place as the command above, I'm getting:

bash-5.1# docker version
Client:
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:03:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:07:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

bash-5.1# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
eddd48245ab6   bridge    bridge    local
410456b84d13   host      host      local
59e8740b2061   none      null      local

so the TLS Docker does work fine.

Which OS & Architecture

• Running docker:20.10.9-dind together with Alpine-based container with k3d installed

Which version of k3d

bash-5.1# k3d version
k3d version v5.0.1
k3s version v1.21.5-k3s2 (default)

Which version of docker

bash-5.1# docker version
Client:
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:03:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:07:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
bash-5.1# docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.129+
 Operating System: Alpine Linux v3.14 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 14.65GiB
 Name: runner-k1lv3ewh-project-3160-concurrent-0hwspv
 ID: 36HT:SJ67:XBK6:HSMW:KN6A:JSJE:APS6:Y7N3:G5P6:LNLP:JAG3:LKMP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[erikgb]

I am also observing this behavior with k3d version 5. It works on k3d version 4 - with the exact same setup, except k3d version OFC.

[iwilltry42]

Hi @mkubaczyk , thanks for opening this issue!
Apparently this seemed to work with k3d v4 and we first have to figure out the exact cause that broke this in v5.
However, using (remote) docker with TLS works with docker contexts as per #674.
E.g. as per the PR description: docker context update dindsec --description "dind local secure" --docker "host=tcp://127.0.0.1:3376,ca=/tmp/dockercerts/client/ca.pem,cert=/tmp/dockercerts/client/cert.pem,key=/tmp/dockercerts/client/key.pem" creates a secure context that works with k3d.