kind should be able to start the control plane when /dev/kmsg is not available

[hoegaarden]

What happened:

kind is not able to start the control plane (with the provided node images) when /dev/kmsg is not available in the node container.

kind then fails with:

 ✗ Starting control-plane 🕹️
Error: failed to create cluster: failed to init node with kubeadm: exit status 1

Checking on the node, I can see the following error:

# docker exec -ti kind-control-plane sh -c 'journalctl -u kubelet | grep -m1 -C3 kmsg'
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.522248  375565 kuberuntime_manager.go:205] Container runtime containerd initialized, version: 1.2.6-0ubuntu1, apiVersion: v1alpha2
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.525786  375565 server.go:1083] Started kubelet
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.526254  375565 server.go:144] Starting to listen on 0.0.0.0:10250
Jun 26 08:18:59 kind-control-plane kubelet[375565]: F0626 08:18:59.532066  375565 kubelet.go:1405] Failed to start OOM watcher open /dev/kmsg: no such file or directory
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.535478  375565 server.go:350] Adding debug handlers to kubelet server.
Jun 26 08:18:59 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Jun 26 08:18:59 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.

For the node images I bake myself, I have the following dirty hack in place:

• I create the image with kind build node-image ...
• I add another layer which adds a systemd file which sets a ExecStartPre for the kubelet which just does something like [ -e /dev/kmsg ] || ln -s /dev/console /dev/kmsg

After that kind create cluster ... successfully starts the control plane.

What you expected to happen:

kind should be able to bring up the control plane, at least when /dev/console is available.

How to reproduce it (as minimally and precisely as possible):

Run kind on an environment without /dev/kmsg, eg. concourse (via kind-on-c for example) and with the 1.15 node image provided by kind.

Anything else we need to know?:

I currently (need to dig deeper) have no idea what the downsides are when linking /dev/kmsg to /dev/console or what to do when /dev/console is not available either (when would that be the case). My tests seem to suggest that the cluster works just fine when doing this linking. Would the kind maintainers be open for a PR which sets such a ExecStartPre (or makes /dev/kmsg available via some other means for that matter)?

Environment:

• kind version: (use kind version):

  kind version v0.4.0
  

• Kubernetes version: (use kubectl version):

  v1.15.0 (the version that is baked into kindest/node:v1.15.0)

• Docker version: (use docker info): ``

  Containers: 1
   Running: 1
   Paused: 0
   Stopped: 0
  Images: 1
  Server Version: 18.09.6
  Storage Driver: overlay2
   Backing Filesystem: extfs
   Supports d_type: true
   Native Overlay Diff: true
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins:
  WARNING: bridge-nf-call-iptables is disabled
  WARNING: bridge-nf-call-ip6tables is disabled
   Volume: local
   Network: bridge host macvlan null overlay
   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
  runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
  init version: fec3683
  Security Options:
   apparmor
   seccomp
    Profile: default
  Kernel Version: 4.15.0-50-generic
  Operating System: Alpine Linux v3.10 (containerized)
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 15.66GiB
  Name: dd95e405-f2d2-4d82-54b3-36f4d1287a53
  ID: YMS4:B7WR:ZESV:VMM6:5CE3:OMGK:7U2Q:3GAS:EGEL:RPTL:35MW:EFPF
  Docker Root Dir: /scratch/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  Labels:
  Experimental: false
  Insecure Registries:
   127.0.0.0/8
  Live Restore Enabled: false
  Product License: Community Engine
  

• OS (e.g. from /etc/os-release):

  NAME="Alpine Linux"
  ID=alpine
  VERSION_ID=3.10.0
  PRETTY_NAME="Alpine Linux v3.10"
  HOME_URL="https://alpinelinux.org/"
  BUG_REPORT_URL="https://bugs.alpinelinux.org/"
  

[hoegaarden]

maybe related: #455

[BenTheElder]

We should definitely try to tolerate environments, I'm a bit curious how your system doesn't have a /dev/kmsg though and if these are really comparable. Taking a look...

[hoegaarden]

• I run the thing in a garden container, on concourse (same behaviour on concourse deployed as a BOSH release on VMs and as a helm chart on k8s/GKE)
• I have the hack with linking /dev/kmsg to /dev/console from some LXC mailinglists, IIRC
• My best guess is, that with Migrate oom watcher not relying on cAdviosr's API any more kubernetes/kubernetes#74942 this was introduced; in 1.14 this was not the case yet, kind ran fine without linking /dev/kmsg

[BenTheElder]

what does /dev/console give you in your environment? I'm not sure that using /dev/console as /dev/kmsg is any different from just placing an empty file at /dev/kmsg. As far as I can tell console is a tty and kmsg is a virtual file... so reading /dev/console as a file just hangs and prints nothing.

[BenTheElder]

https://www.kernel.org/doc/html/latest/admin-guide/serial-console.html
https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg

[hoegaarden]

lxc has a config setting for that, see man lxc.container.conf:

[...]
   ENABLE KMSG SYMLINK
       Enable creating /dev/kmsg as symlink to /dev/console. This defaults to 0.

       lxc.kmsg
              Set this to 1 to enable /dev/kmsg symlinking.
[...]

... so it is not completely uncommon. 🤷‍♂️

[BenTheElder]

interesting, thanks! that works for me :-)

[BenTheElder]

fixed in #664

[BenTheElder]

thanks!

[nmvega]

interesting, thanks! that works for me :-)

@hoegaarden @BenTheElder

Just in case someone comes looking, for my Fedora-30 LXC O/S container (and probably for somewhat newer and older versions of Fedora, as well as for CentOS), this directive was removed (deprecated): lxc.kmsg = 0|1. Container configuration files with that entry will prevent it from starting. See this.

However poking around a little bit, I found and verified that this now works ...

Bind host /dev/kmsg for OOM killer:

lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file

See this (and scroll down a bit).

I hope this helps others.

[BenTheElder]

I missed your comment previously @nmvega, I think we should be doing something else in kind as well, I just stumbled across https://www.mail-archive.com/[email protected]/msg02241.html while rethinking this and came to post about the removal in LXC :-)

I'm not sure kind should be responsible for ensuring /dev/kmsg exists anyhow vs leaving it to the host environment ...