kubeadm init fails with NewSession: EOF

[hemani19]

Environment: CentOS 7 64 Bit, VM version 13

VirtualBox Version: Oracle VM VirtualBox VM Selector v6.0.4

Minikube version (use minikube version): v0.35.0

• OS (e.g. from /etc/os-release): CentOS 7
• VM Driver virtualbox

# minikube start --vm-driver=virtualbox --bootstrapper=kubeadm
😄  minikube v0.35.0 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿  Downloading Minikube ISO ...
 184.42 MB / 184.42 MB [============================================] 100.00% 0s
📶  "minikube" IP address is 192.168.99.103
🐳  Configuring Docker as the container runtime ...
✨  Preparing Kubernetes environment ...
💾  Downloading kubeadm v1.13.4
💾  Downloading kubelet v1.13.4
🚜  Pulling images required by Kubernetes v1.13.4 ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout: [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.13.4
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.13.4
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.13.4
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.13.4
[config/images] Pulled k8s.gcr.io/pause:3.1

stderr: : wait: remote command exited without exit status or exit signal
🚀  Launching Kubernetes v1.13.4 using kubeadm ... 
E0320 10:42:17.364109   51664 logs.go:141] Failed to list containers for "kube-apiserver": NewSession: EOF
E0320 10:42:17.546170   51664 logs.go:141] Failed to list containers for "coredns": NewSession: EOF
E0320 10:42:17.546201   51664 logs.go:141] Failed to list containers for "kube-scheduler": NewSession: EOF
💣  Error starting cluster: kubeadm init: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI 


: NewSession: EOF

Tried deleting and starting from scratch and also tried other drivers with no luck.

[tstromberg]

The errors indicate that the VM is closing the SSH connection on minikube, though I don't know why. Do you mind including the output of:

minikube ssh dmesg

Thanks!

[hemani19]

Thanks, @tstromberg.

minikube ssh dmesg
ssh: "minikube" is not running

Also, I would like to mention that I could get it working with --vm-driver=none.

[tstromberg]

Could you run: minikube start -p testvm2 followed immediately by minikube ssh -p testvm2 dmesg?

I'd like to sort out if VirtualBox is killing the VM, or if the VM is killing itself. Thanks!

[recepayaz]

hello @tstromberg ,
I am getting the same issue: I am running Ubuntu vmware on windows 10.

Environment: Linux ubuntu 4.18.0-15-generic #16~18.04.1-Ubuntu SMP Thu Feb 7 14:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

**@ubuntu:~$ minikube start **
😄 minikube v1.0.1 on linux (amd64)
🤹 Downloading Kubernetes v1.14.1 images in the background ...
💡 Tip: Use 'minikube start -p ' to create a new cluster, or 'minikube delete' to delete this one.
🔄 Restarting existing virtualbox VM for "minikube" ...
⌛ Waiting for SSH access ...
📶 "minikube" IP address is 192.168.99.100
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.3-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
❌ Unable to load cached images: loading cached images: loading image /home/recep/.minikube/cache/images/k8s.gcr.io/kube-addon-manager_v9.0: Docker load /tmp/kube-addon-manager_v9.0: NewSession: EOF

💣 Failed to update cluster: downloading binaries: copying kubelet: copy: pre-copy: NewSession: EOF

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new
r@ubuntu:~$ minikube version
minikube version: v1.0.1

more detailed logs

I0511 18:08:54.940389 5971 cache_images.go:236] Successfully loaded image /home/recep/.minikube/cache/images/k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64_1.14.13 from cache
I0511 18:08:54.946214 5971 docker.go:86] Loading image: /tmp/k8s-dns-sidecar-amd64_1.14.13
I0511 18:08:54.946255 5971 ssh_runner.go:101] SSH: docker load -i /tmp/k8s-dns-sidecar-amd64_1.14.13
I0511 18:09:03.800461 5971 utils.go:240] > Loaded image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.13
I0511 18:09:03.818958 5971 ssh_runner.go:101] SSH: sudo rm -rf /tmp/k8s-dns-sidecar-amd64_1.14.13
I0511 18:09:04.383963 5971 cache_images.go:236] Successfully loaded image /home/recep/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13 from cache
I0511 18:09:04.389334 5971 docker.go:86] Loading image: /tmp/coredns_1.3.1
I0511 18:09:04.389980 5971 ssh_runner.go:101] SSH: docker load -i /tmp/coredns_1.3.1
I0511 18:12:31.264744 5971 docker.go:86] Loading image: /tmp/k8s-dns-kube-dns-amd64_1.14.13
I0511 18:12:31.266440 5971 ssh_runner.go:101] SSH: docker load -i /tmp/k8s-dns-kube-dns-amd64_1.14.13
I0511 18:12:31.275404 5971 docker.go:86] Loading image: /tmp/storage-provisioner_v1.8.1
I0511 18:12:31.276487 5971 ssh_runner.go:101] SSH: docker load -i /tmp/storage-provisioner_v1.8.1
W0511 18:12:31.276544 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: Docker load /tmp/storage-provisioner_v1.8.1: NewSession: EOF
I0511 18:12:31.279880 5971 docker.go:86] Loading image: /tmp/kube-scheduler_v1.14.1
I0511 18:12:31.279917 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kube-scheduler_v1.14.1
W0511 18:12:31.279951 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kube-scheduler_v1.14.1: Docker load /tmp/kube-scheduler_v1.14.1: NewSession: EOF
I0511 18:12:31.281473 5971 docker.go:86] Loading image: /tmp/kube-proxy_v1.14.1
I0511 18:12:31.281530 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kube-proxy_v1.14.1
W0511 18:12:31.281716 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kube-proxy_v1.14.1: Docker load /tmp/kube-proxy_v1.14.1: NewSession: EOF
I0511 18:12:31.282654 5971 docker.go:86] Loading image: /tmp/kube-addon-manager_v9.0
I0511 18:12:31.282679 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kube-addon-manager_v9.0
W0511 18:12:31.283318 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kube-addon-manager_v9.0: Docker load /tmp/kube-addon-manager_v9.0: NewSession: EOF
I0511 18:12:31.284119 5971 docker.go:86] Loading image: /tmp/kube-controller-manager_v1.14.1
I0511 18:12:31.284150 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kube-controller-manager_v1.14.1
W0511 18:12:31.284396 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kube-controller-manager_v1.14.1: Docker load /tmp/kube-controller-manager_v1.14.1: NewSession: EOF
I0511 18:12:31.285259 5971 docker.go:86] Loading image: /tmp/kubernetes-dashboard-amd64_v1.10.1
I0511 18:12:31.285294 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kubernetes-dashboard-amd64_v1.10.1
W0511 18:12:31.259208 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/coredns_1.3.1: Docker load /tmp/coredns_1.3.1: command failed: docker load -i /tmp/coredns_1.3.1
stdout:
stderr: : wait: remote command exited without exit status or exit signal
W0511 18:12:31.272336 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/k8s-dns-kube-dns-amd64_1.14.13: Docker load /tmp/k8s-dns-kube-dns-amd64_1.14.13: NewSession: EOF
W0511 18:12:31.289607 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kubernetes-dashboard-amd64_v1.10.1: Docker load /tmp/kubernetes-dashboard-amd64_v1.10.1: NewSession: EOF
I0511 18:12:31.290199 5971 docker.go:86] Loading image: /tmp/kube-apiserver_v1.14.1
I0511 18:12:31.290898 5971 ssh_runner.go:101] SSH: docker load -i /tmp/kube-apiserver_v1.14.1
W0511 18:12:31.291443 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/kube-apiserver_v1.14.1: Docker load /tmp/kube-apiserver_v1.14.1: NewSession: EOF
I0511 18:12:31.292239 5971 docker.go:86] Loading image: /tmp/etcd_3.3.10
I0511 18:12:31.292261 5971 ssh_runner.go:101] SSH: docker load -i /tmp/etcd_3.3.10
W0511 18:12:31.292287 5971 cache_images.go:104] Failed to load /home/recep/.minikube/cache/images/k8s.gcr.io/etcd_3.3.10: Docker load /tmp/etcd_3.3.10: NewSession: EOF
❌ Unable to load cached images: loading cached images: loading image /home/recep/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: Docker load /tmp/storage-provisioner_v1.8.1: NewSession: EOF
I0511 18:12:31.329550 5971 kubeadm.go:447] kubelet v1.14.1 config:

[Unit]
Wants=docker.socket

[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --allow-privileged=true --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests

[Install]
I0511 18:12:31.341556 5971 cache_binaries.go:61] Not caching binary, using https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/kubeadm
I0511 18:12:31.343541 5971 ssh_runner.go:101] SSH: sudo rm -f /usr/bin/kubeadm
I0511 18:12:31.344258 5971 cache_binaries.go:61] Not caching binary, using https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/kubelet
I0511 18:12:31.344311 5971 ssh_runner.go:101] SSH: sudo rm -f /usr/bin/kubelet
W0511 18:12:31.530423 5971 exit.go:99] Failed to update cluster: downloading binaries: copying kubeadm: copy: pre-copy: NewSession: EOF

💣 Failed to update cluster: downloading binaries: copying kubeadm: copy: pre-copy: NewSession: EOF

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new

and last as you asked here is the output of (minikube start -p testvm2 followed immediately by minikube ssh -p testvm2 dmesg)

recep@ubuntu:~$ minikube start -p testvm2
😄 minikube v1.0.1 on linux (amd64)
🤹 Downloading Kubernetes v1.14.1 images in the background ...
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶 "testvm2" IP address is 192.168.99.100
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.3-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
❌ Unable to load cached images: loading cached images: loading image /home/recep/.minikube/cache/images/k8s.gcr.io/kube-apiserver_v1.14.1: transferring cached image: wait: remote command exited without exit status or exit signal

💣 Failed to update cluster: downloading binaries: copying kubelet: copy: pre-copy: NewSession: write tcp 127.0.0.1:33748->127.0.0.1:44545: write: connection reset by peer

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new
recep@ubuntu:~$ minikube ssh -p testvm2 dmesg
ssh: "testvm2" is not running

would u please advise ? thanks in advance.

[tstromberg]

From the sound of it, VirtualBox is killing the minikube VM for some reason. I'm not sure how to investigate why, however.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

I'm closing this issue as it hasn't seen activity in awhile, and it's unclear if this issue still exists. If this issue does continue to exist in the most recent release of minikube, please feel free to re-open it.

Thank you for opening the issue!