Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

hetong07 · 2021-02-17T23:21:50Z

Hello,
I tried to run the minikube start on my Debian machine and got

 minikube v1.17.1 on Debian rodete
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=15900MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
cb0d6b78bb1f729dc2b4ba385bee0ddbdab7ab58d999c47a49e21c854f2cb52c

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-cb0d6b78bb1f729dc2b4ba385bee0ddbdab7ab58d999c47a49e21c854f2cb52c.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-cb0d6b78bb1f729dc2b4ba385bee0ddbdab7ab58d999c47a49e21c854f2cb52c.scope/memory.swap.max: permission denied: unknown.

🤷  docker "minikube" container is missing, will recreate.

I tried to run minikube delete --all and restart my computer, but that problem remains.

Below is the full log with '--alsologtostderr'

I0217 15:01:02.402828   19706 out.go:229] Setting OutFile to fd 1 ...
I0217 15:01:02.403094   19706 out.go:281] isatty.IsTerminal(1) = false
I0217 15:01:02.403105   19706 out.go:242] Setting ErrFile to fd 2...
I0217 15:01:02.403111   19706 out.go:281] isatty.IsTerminal(2) = false
I0217 15:01:02.403211   19706 root.go:291] Updating PATH: /usr/local/google/home/tonghe/.minikube/bin
I0217 15:01:02.403471   19706 out.go:236] Setting JSON to false
I0217 15:01:02.424842   19706 start.go:106] hostinfo: {"hostname":"tonghe.mtv.corp.google.com","uptime":1359,"bootTime":1613601503,"procs":402,"os":"linux","platform":"debian","platformFamily":"debian","platformVersion":"rodete","kernelVersion":"5.7.17-1rodete5-amd64","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"ea5f3350-297d-e911-b2d8-1ca0b8760815"}
I0217 15:01:02.424973   19706 start.go:116] virtualization: kvm host
I0217 15:01:02.436299   19706 out.go:119] * minikube v1.17.1 on Debian rodete
* minikube v1.17.1 on Debian rodete
I0217 15:01:02.436576   19706 notify.go:126] Checking for updates...
I0217 15:01:02.437007   19706 driver.go:315] Setting default libvirt URI to qemu:///system
I0217 15:01:02.501528   19706 docker.go:115] docker version: linux-20.10.2
I0217 15:01:02.501761   19706 cli_runner.go:111] Run: docker system info --format "{{json .}}"
I0217 15:01:02.589848   19706 info.go:253] docker info: {ID:LMAP:N3UY:RYPB:X2IG:XPJY:44ZG:U6HO:U47K:DEE6:LRAV:3Q6H:NZYS Containers:2 ContainersRunning:0 ContainersPaused:0 ContainersStopped:2 Images:17 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:true NFd:22 OomKillDisable:false NGoroutines:34 SystemTime:2021-02-17 15:01:02.533250224 -0800 PST LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:5.7.17-1rodete5-amd64 OperatingSystem:Debian GNU/Linux rodete OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[https://mirror.gcr.io/] Secure:true Official:true}} Mirrors:[https://mirror.gcr.io/]} NCPU:12 MemTotal:67086561280 GenericResources:<nil> DockerRootDir:/usr/local/google/docker HTTPProxy: HTTPSProxy: NoProxy: Name:tonghe.mtv.corp.google.com Labels:[] ExperimentalBuild:false ServerVersion:20.10.2 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:269548fa27e0089a8b8278fc4fc781d7f65a939b Expected:269548fa27e0089a8b8278fc4fc781d7f65a939b} RuncCommit:{ID:ff819c7e9184c13b7c2607fe6c30ae19403a7aff Expected:ff819c7e9184c13b7c2607fe6c30ae19403a7aff} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default name=cgroupns] ProductLicense: Warnings:[WARNING: No kernel memory TCP limit support WARNING: No oom kill disable support WARNING: Support for cgroup v2 is experimental] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Build with BuildKit Vendor:Docker Inc. Version:v0.5.1-docker]] Warnings:<nil>}}
I0217 15:01:02.590056   19706 docker.go:145] overlay module found
I0217 15:01:02.596507   19706 out.go:119] * Using the docker driver based on existing profile
* Using the docker driver based on existing profile
I0217 15:01:02.596574   19706 start.go:279] selected driver: docker
I0217 15:01:02.596589   19706 start.go:702] validating driver "docker" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:15900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false}
I0217 15:01:02.596699   19706 start.go:713] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc:}
I0217 15:01:02.596849   19706 cli_runner.go:111] Run: docker system info --format "{{json .}}"
I0217 15:01:02.690132   19706 info.go:253] docker info: {ID:LMAP:N3UY:RYPB:X2IG:XPJY:44ZG:U6HO:U47K:DEE6:LRAV:3Q6H:NZYS Containers:2 ContainersRunning:0 ContainersPaused:0 ContainersStopped:2 Images:17 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:true NFd:22 OomKillDisable:false NGoroutines:34 SystemTime:2021-02-17 15:01:02.63481078 -0800 PST LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:5.7.17-1rodete5-amd64 OperatingSystem:Debian GNU/Linux rodete OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[https://mirror.gcr.io/] Secure:true Official:true}} Mirrors:[https://mirror.gcr.io/]} NCPU:12 MemTotal:67086561280 GenericResources:<nil> DockerRootDir:/usr/local/google/docker HTTPProxy: HTTPSProxy: NoProxy: Name:tonghe.mtv.corp.google.com Labels:[] ExperimentalBuild:false ServerVersion:20.10.2 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:269548fa27e0089a8b8278fc4fc781d7f65a939b Expected:269548fa27e0089a8b8278fc4fc781d7f65a939b} RuncCommit:{ID:ff819c7e9184c13b7c2607fe6c30ae19403a7aff Expected:ff819c7e9184c13b7c2607fe6c30ae19403a7aff} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default name=cgroupns] ProductLicense: Warnings:[WARNING: No kernel memory TCP limit support WARNING: No oom kill disable support WARNING: Support for cgroup v2 is experimental] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Build with BuildKit Vendor:Docker Inc. Version:v0.5.1-docker]] Warnings:<nil>}}
I0217 15:01:02.692395   19706 start_flags.go:390] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:15900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false}
I0217 15:01:02.699055   19706 out.go:119] * Starting control plane node minikube in cluster minikube
* Starting control plane node minikube in cluster minikube
I0217 15:01:02.753081   19706 image.go:92] Found gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e in local docker daemon, skipping pull
I0217 15:01:02.753116   19706 cache.go:116] gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e exists in daemon, skipping pull
I0217 15:01:02.753125   19706 preload.go:97] Checking if preload exists for k8s version v1.20.2 and runtime docker
I0217 15:01:02.753158   19706 preload.go:105] Found local preload: /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4
I0217 15:01:02.753163   19706 cache.go:54] Caching tarball of preloaded images
I0217 15:01:02.753176   19706 preload.go:131] Found /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0217 15:01:02.753181   19706 cache.go:57] Finished verifying existence of preloaded tar for  v1.20.2 on docker
I0217 15:01:02.753273   19706 profile.go:148] Saving config to /usr/local/google/home/tonghe/.minikube/profiles/minikube/config.json ...
I0217 15:01:02.753449   19706 cache.go:185] Successfully downloaded all kic artifacts
I0217 15:01:02.753472   19706 start.go:313] acquiring machines lock for minikube: {Name:mkc82393027f1672ccfb71113021bcb9bfb7a9bd Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0217 15:01:02.753595   19706 start.go:317] acquired machines lock for "minikube" in 106.097µs
I0217 15:01:02.753623   19706 start.go:93] Skipping create...Using existing machine configuration
I0217 15:01:02.753631   19706 fix.go:54] fixHost starting: 
I0217 15:01:02.753827   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:02.797653   19706 fix.go:107] recreateIfNeeded on minikube: state= err=<nil>
I0217 15:01:02.797713   19706 fix.go:112] machineExists: false. err=machine does not exist
I0217 15:01:02.801509   19706 out.go:119] * docker "minikube" container is missing, will recreate.
* docker "minikube" container is missing, will recreate.
I0217 15:01:02.801553   19706 delete.go:124] DEMOLISHING minikube ...
I0217 15:01:02.801693   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:02.843186   19706 stop.go:79] host is in state 
I0217 15:01:02.843239   19706 main.go:119] libmachine: Stopping "minikube"...
I0217 15:01:02.843305   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:02.888079   19706 kic_runner.go:94] Run: systemctl --version
I0217 15:01:02.888128   19706 kic_runner.go:115] Args: [docker exec --privileged minikube systemctl --version]
I0217 15:01:02.933782   19706 kic_runner.go:94] Run: sudo service kubelet stop
I0217 15:01:02.933827   19706 kic_runner.go:115] Args: [docker exec --privileged minikube sudo service kubelet stop]
I0217 15:01:02.974208   19706 openrc.go:141] stop output: 
** stderr ** 
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running

** /stderr **
W0217 15:01:02.974248   19706 kic.go:421] couldn't stop kubelet. will continue with stop anyways: sudo service kubelet stop: exit status 1
stdout:

stderr:
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running
I0217 15:01:02.974702   19706 kic_runner.go:94] Run: sudo service kubelet stop
I0217 15:01:02.974722   19706 kic_runner.go:115] Args: [docker exec --privileged minikube sudo service kubelet stop]
I0217 15:01:03.015287   19706 openrc.go:141] stop output: 
** stderr ** 
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running

** /stderr **
W0217 15:01:03.015334   19706 kic.go:423] couldn't force stop kubelet. will continue with stop anyways: sudo service kubelet stop: exit status 1
stdout:

stderr:
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running
I0217 15:01:03.015479   19706 kic_runner.go:94] Run: docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}
I0217 15:01:03.015497   19706 kic_runner.go:115] Args: [docker exec --privileged minikube docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}]
I0217 15:01:03.056940   19706 kic.go:434] unable list containers : docker: docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}: exit status 1
stdout:

stderr:
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running
I0217 15:01:03.056965   19706 kic.go:444] successfully stopped kubernetes!
I0217 15:01:03.057028   19706 kic_runner.go:94] Run: pgrep kube-apiserver
I0217 15:01:03.057036   19706 kic_runner.go:115] Args: [docker exec --privileged minikube pgrep kube-apiserver]
I0217 15:01:03.138498   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:06.181932   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:09.301968   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:12.421459   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:15.543037   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:18.664705   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:21.786321   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:24.909862   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:28.023390   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:31.148369   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:34.262807   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:37.383298   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:40.505063   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:43.620049   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:46.735072   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:49.849763   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:52.962052   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:56.081844   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:01:59.197679   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:02.317398   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:05.431646   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:08.548684   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:11.662976   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:14.817169   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:17.944038   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:21.070996   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:24.120916   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:27.237180   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:30.357477   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:33.484913   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:36.600407   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:39.717916   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:42.834774   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:45.957272   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:49.073805   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:52.187029   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:55.301274   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:02:58.422622   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:01.545867   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:04.659838   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:07.773207   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:10.889945   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:14.009487   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:17.126836   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:20.252617   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:23.377542   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:26.497231   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:29.611090   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:32.724768   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:35.837958   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:38.953430   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:42.068358   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:45.189723   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:48.310349   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:51.429639   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:54.550961   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:03:57.663526   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:00.783431   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:03.899565   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:07.015140   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:10.133804   19706 stop.go:59] stop err: Maximum number of retries (60) exceeded
I0217 15:04:10.133929   19706 delete.go:129] stophost failed (probably ok): Temporary Error: stop: Maximum number of retries (60) exceeded
I0217 15:04:10.134874   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
W0217 15:04:10.251875   19706 delete.go:135] deletehost failed: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
I0217 15:04:10.252019   19706 cli_runner.go:111] Run: docker container inspect -f {{.Id}} minikube
I0217 15:04:10.302794   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:10.345879   19706 cli_runner.go:111] Run: docker exec --privileged -t minikube /bin/bash -c "sudo init 0"
W0217 15:04:10.391632   19706 cli_runner.go:149] docker exec --privileged -t minikube /bin/bash -c "sudo init 0" returned with exit code 1
I0217 15:04:10.391695   19706 oci.go:600] error shutdown minikube: docker exec --privileged -t minikube /bin/bash -c "sudo init 0": exit status 1
stdout:

stderr:
Error response from daemon: Container 6752e2aee123283c87e98383cc02a556077a3c6737668ef272b18ef07103da11 is not running
I0217 15:04:11.392073   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:11.442321   19706 oci.go:614] temporary error: container minikube status is  but expect it to be exited
I0217 15:04:11.442387   19706 oci.go:620] Successfully shutdown container minikube
I0217 15:04:11.442499   19706 cli_runner.go:111] Run: docker rm -f -v minikube
I0217 15:04:11.503067   19706 cli_runner.go:111] Run: docker container inspect -f {{.Id}} minikube
W0217 15:04:11.546554   19706 cli_runner.go:149] docker container inspect -f {{.Id}} minikube returned with exit code 1
I0217 15:04:11.546742   19706 cli_runner.go:111] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
I0217 15:04:11.586187   19706 cli_runner.go:111] Run: docker network rm minikube
W0217 15:04:11.736929   19706 delete.go:139] delete failed (probably ok) <nil>
I0217 15:04:11.736965   19706 fix.go:119] Sleeping 1 second for extra luck!
I0217 15:04:12.737340   19706 start.go:126] createHost starting for "" (driver="docker")
I0217 15:04:12.744265   19706 out.go:140] * Creating docker container (CPUs=2, Memory=15900MB) ...
* Creating docker container (CPUs=2, Memory=15900MB) ...| I0217 15:04:12.744510   19706 start.go:160] libmachine.API.Create for "minikube" (driver="docker")
I0217 15:04:12.744576   19706 client.go:168] LocalClient.Create starting
I0217 15:04:12.744741   19706 main.go:119] libmachine: Reading certificate data from /usr/local/google/home/tonghe/.minikube/certs/ca.pem
I0217 15:04:12.744816   19706 main.go:119] libmachine: Decoding PEM data...
I0217 15:04:12.744852   19706 main.go:119] libmachine: Parsing certificate...
I0217 15:04:12.745087   19706 main.go:119] libmachine: Reading certificate data from /usr/local/google/home/tonghe/.minikube/certs/cert.pem
I0217 15:04:12.745140   19706 main.go:119] libmachine: Decoding PEM data...
I0217 15:04:12.745169   19706 main.go:119] libmachine: Parsing certificate...
I0217 15:04:12.745715   19706 cli_runner.go:111] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
W0217 15:04:12.796215   19706 cli_runner.go:149] docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}" returned with exit code 1
I0217 15:04:12.796323   19706 network_create.go:249] running [docker network inspect minikube] to gather additional debugging logs...
I0217 15:04:12.796346   19706 cli_runner.go:111] Run: docker network inspect minikube
W0217 15:04:12.840811   19706 cli_runner.go:149] docker network inspect minikube returned with exit code 1
I0217 15:04:12.840877   19706 network_create.go:252] error running [docker network inspect minikube]: docker network inspect minikube: exit status 1
stdout:
[]

stderr:
Error: No such network: minikube
I0217 15:04:12.840909   19706 network_create.go:254] output of [docker network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr ** 
Error: No such network: minikube

** /stderr **
I0217 15:04:12.841007   19706 cli_runner.go:111] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
��WW���[K�[K/ I0217 15:04:12.877693   19706 network_create.go:104] attempt to create network 192.168.49.0/24 with subnet: minikube and gateway 192.168.49.1 and MTU of 1500 ...
I0217 15:04:12.877834   19706 cli_runner.go:111] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true minikube
��WW���[K�[K- I0217 15:04:12.968438   19706 kic.go:100] calculated static IP "192.168.49.2" for the "minikube" container
I0217 15:04:12.968517   19706 cli_runner.go:111] Run: docker ps -a --format {{.Names}}
I0217 15:04:12.999012   19706 cli_runner.go:111] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0217 15:04:13.036775   19706 oci.go:102] Successfully created a docker volume minikube
I0217 15:04:13.036893   19706 cli_runner.go:111] Run: docker run --rm --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -d /var/lib
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ I0217 15:04:13.723541   19706 oci.go:106] Successfully prepared a docker volume minikube
W0217 15:04:13.723612   19706 oci.go:159] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0217 15:04:13.723635   19706 oci.go:201] Your kernel does not support CPU cfs period/quota or the cgroup is not mounted.
I0217 15:04:13.723674   19706 preload.go:97] Checking if preload exists for k8s version v1.20.2 and runtime docker
I0217 15:04:13.723694   19706 cli_runner.go:111] Run: docker info --format "'{{json .SecurityOptions}}'"
I0217 15:04:13.723729   19706 preload.go:105] Found local preload: /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4
I0217 15:04:13.723742   19706 kic.go:163] Starting extracting preloaded images to volume ...
I0217 15:04:13.723804   19706 cli_runner.go:111] Run: docker run --rm --entrypoint /usr/bin/tar -v /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -I lz4 -xf /preloaded.tar -C /extractDir
��WW���[K�[K- I0217 15:04:13.838712   19706 cli_runner.go:111] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| W0217 15:04:14.357297   19706 cli_runner.go:149] docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e returned with exit code 126
I0217 15:04:14.357356   19706 client.go:171] LocalClient.Create took 1.612761581s
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:04:16.358457   19706 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0217 15:04:16.358595   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:04:16.460829   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:16.460937   19706 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:04:16.737592   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:04:16.777637   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:16.777802   19706 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:04:17.318634   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| I0217 15:04:17.716989   19706 cli_runner.go:155] Completed: docker run --rm --entrypoint /usr/bin/tar -v /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -I lz4 -xf /preloaded.tar -C /extractDir: (3.993104258s)
I0217 15:04:17.717055   19706 kic.go:172] duration metric: took 3.993305 seconds to extract preloaded images to volume
W0217 15:04:17.723992   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:17.724120   19706 retry.go:31] will retry after 655.06503ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- I0217 15:04:18.380289   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K\ W0217 15:04:18.429523   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
W0217 15:04:18.429734   19706 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

W0217 15:04:18.429797   19706 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
I0217 15:04:18.429817   19706 start.go:129] duration metric: createHost completed in 5.692401413s
I0217 15:04:18.429928   19706 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0217 15:04:18.430002   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0217 15:04:18.474878   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:18.475042   19706 retry.go:31] will retry after 234.428547ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K| ��WW���[K�[K/ I0217 15:04:18.710029   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K- W0217 15:04:18.764762   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:18.764901   19706 retry.go:31] will retry after 346.739061ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ I0217 15:04:19.112508   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K- W0217 15:04:19.182545   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:19.182720   19706 retry.go:31] will retry after 667.587979ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ I0217 15:04:19.850657   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0217 15:04:19.905664   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:04:19.905843   19706 retry.go:31] will retry after 477.256235ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- I0217 15:04:20.384056   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K\ W0217 15:04:20.439414   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
W0217 15:04:20.439608   19706 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

W0217 15:04:20.439636   19706 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
I0217 15:04:20.439655   19706 fix.go:56] fixHost completed within 3m17.686022086s
I0217 15:04:20.439671   19706 start.go:80] releasing machines lock for "minikube", held for 3m17.686066637s
W0217 15:04:20.439715   19706 start.go:377] error starting host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max: permission denied: unknown.
W0217 15:04:20.439997   19706 out.go:181] ! StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max: permission denied: unknown.

��WW���[K�[K
! StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0.scope/memory.swap.max: permission denied: unknown.

I0217 15:04:20.440086   19706 start.go:392] Will try again in 5 seconds ...
I0217 15:04:25.441832   19706 start.go:313] acquiring machines lock for minikube: {Name:mkc82393027f1672ccfb71113021bcb9bfb7a9bd Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0217 15:04:25.442112   19706 start.go:317] acquired machines lock for "minikube" in 177.732µs
I0217 15:04:25.442175   19706 start.go:93] Skipping create...Using existing machine configuration
I0217 15:04:25.442194   19706 fix.go:54] fixHost starting: 
I0217 15:04:25.442728   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:25.555370   19706 fix.go:107] recreateIfNeeded on minikube: state= err=<nil>
I0217 15:04:25.555458   19706 fix.go:112] machineExists: false. err=machine does not exist
I0217 15:04:25.566465   19706 out.go:119] * docker "minikube" container is missing, will recreate.
* docker "minikube" container is missing, will recreate.
I0217 15:04:25.566526   19706 delete.go:124] DEMOLISHING minikube ...
I0217 15:04:25.566667   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:25.612453   19706 stop.go:79] host is in state 
I0217 15:04:25.612521   19706 main.go:119] libmachine: Stopping "minikube"...
I0217 15:04:25.612671   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:25.657602   19706 kic_runner.go:94] Run: sudo service kubelet stop
I0217 15:04:25.657660   19706 kic_runner.go:115] Args: [docker exec --privileged minikube sudo service kubelet stop]
I0217 15:04:25.703845   19706 openrc.go:141] stop output: 
** stderr ** 
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running

** /stderr **
W0217 15:04:25.703886   19706 kic.go:421] couldn't stop kubelet. will continue with stop anyways: sudo service kubelet stop: exit status 1
stdout:

stderr:
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running
I0217 15:04:25.704023   19706 kic_runner.go:94] Run: sudo service kubelet stop
I0217 15:04:25.704097   19706 kic_runner.go:115] Args: [docker exec --privileged minikube sudo service kubelet stop]
I0217 15:04:25.751326   19706 openrc.go:141] stop output: 
** stderr ** 
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running

** /stderr **
W0217 15:04:25.751369   19706 kic.go:423] couldn't force stop kubelet. will continue with stop anyways: sudo service kubelet stop: exit status 1
stdout:

stderr:
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running
I0217 15:04:25.751508   19706 kic_runner.go:94] Run: docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}
I0217 15:04:25.751526   19706 kic_runner.go:115] Args: [docker exec --privileged minikube docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}]
I0217 15:04:25.796491   19706 kic.go:434] unable list containers : docker: docker ps -a --filter=name=k8s_.*_(kube-system|kubernetes-dashboard|storage-gluster|istio-operator)_ --format={{.ID}}: exit status 1
stdout:

stderr:
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running
I0217 15:04:25.796535   19706 kic.go:444] successfully stopped kubernetes!
I0217 15:04:25.796668   19706 kic_runner.go:94] Run: pgrep kube-apiserver
I0217 15:04:25.796685   19706 kic_runner.go:115] Args: [docker exec --privileged minikube pgrep kube-apiserver]
I0217 15:04:25.880765   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:28.924197   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:32.043109   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:35.157660   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:38.271962   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:41.391020   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:44.509846   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:47.638478   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:50.755340   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:53.868896   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:04:56.985874   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:00.101858   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:03.220473   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:06.337289   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:09.449477   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:12.565079   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:15.685884   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:18.804297   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:21.923054   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:25.041900   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:28.159479   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:31.281717   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:34.395035   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:37.510573   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:40.629955   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:43.745249   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:46.868774   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:49.990218   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:53.114251   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:56.250845   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:05:59.372173   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:02.490827   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:05.605100   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:08.720422   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:11.833838   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:14.949908   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:18.071048   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:21.190681   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:24.309879   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:27.428356   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:30.545293   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:33.661802   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:36.782571   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:39.907267   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:43.023001   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:46.142762   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:49.264183   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:52.385545   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:55.500243   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:06:58.621145   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:01.739956   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:04.857866   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:07.975044   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:11.096771   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:14.215141   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:17.331715   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:20.453973   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:23.569817   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:26.703397   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:29.826848   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:32.941406   19706 stop.go:59] stop err: Maximum number of retries (60) exceeded
I0217 15:07:32.941540   19706 delete.go:129] stophost failed (probably ok): Temporary Error: stop: Maximum number of retries (60) exceeded
I0217 15:07:32.942437   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
W0217 15:07:33.065477   19706 delete.go:135] deletehost failed: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
I0217 15:07:33.065681   19706 cli_runner.go:111] Run: docker container inspect -f {{.Id}} minikube
I0217 15:07:33.114549   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:33.163302   19706 cli_runner.go:111] Run: docker exec --privileged -t minikube /bin/bash -c "sudo init 0"
W0217 15:07:33.210020   19706 cli_runner.go:149] docker exec --privileged -t minikube /bin/bash -c "sudo init 0" returned with exit code 1
I0217 15:07:33.210083   19706 oci.go:600] error shutdown minikube: docker exec --privileged -t minikube /bin/bash -c "sudo init 0": exit status 1
stdout:

stderr:
Error response from daemon: Container 9aeacb348b8dd506e17d7a21f5b78446bacbe069c3670058ceaf818e2d7fe6c0 is not running
I0217 15:07:34.210349   19706 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0217 15:07:34.322826   19706 oci.go:614] temporary error: container minikube status is  but expect it to be exited
I0217 15:07:34.322885   19706 oci.go:620] Successfully shutdown container minikube
I0217 15:07:34.322984   19706 cli_runner.go:111] Run: docker rm -f -v minikube
I0217 15:07:34.382741   19706 cli_runner.go:111] Run: docker container inspect -f {{.Id}} minikube
W0217 15:07:34.433116   19706 cli_runner.go:149] docker container inspect -f {{.Id}} minikube returned with exit code 1
I0217 15:07:34.433282   19706 cli_runner.go:111] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
I0217 15:07:34.480678   19706 cli_runner.go:111] Run: docker network rm minikube
W0217 15:07:34.673786   19706 delete.go:139] delete failed (probably ok) <nil>
I0217 15:07:34.673835   19706 fix.go:119] Sleeping 1 second for extra luck!
I0217 15:07:35.674265   19706 start.go:126] createHost starting for "" (driver="docker")
I0217 15:07:35.695071   19706 out.go:140] * Creating docker container (CPUs=2, Memory=15900MB) ...
* Creating docker container (CPUs=2, Memory=15900MB) ...| I0217 15:07:35.695317   19706 start.go:160] libmachine.API.Create for "minikube" (driver="docker")
I0217 15:07:35.695383   19706 client.go:168] LocalClient.Create starting
I0217 15:07:35.695539   19706 main.go:119] libmachine: Reading certificate data from /usr/local/google/home/tonghe/.minikube/certs/ca.pem
I0217 15:07:35.695628   19706 main.go:119] libmachine: Decoding PEM data...
I0217 15:07:35.695666   19706 main.go:119] libmachine: Parsing certificate...
I0217 15:07:35.695884   19706 main.go:119] libmachine: Reading certificate data from /usr/local/google/home/tonghe/.minikube/certs/cert.pem
I0217 15:07:35.695937   19706 main.go:119] libmachine: Decoding PEM data...
I0217 15:07:35.695967   19706 main.go:119] libmachine: Parsing certificate...
I0217 15:07:35.696444   19706 cli_runner.go:111] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
W0217 15:07:35.747947   19706 cli_runner.go:149] docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}" returned with exit code 1
I0217 15:07:35.748084   19706 network_create.go:249] running [docker network inspect minikube] to gather additional debugging logs...
I0217 15:07:35.748118   19706 cli_runner.go:111] Run: docker network inspect minikube
W0217 15:07:35.793919   19706 cli_runner.go:149] docker network inspect minikube returned with exit code 1
I0217 15:07:35.793969   19706 network_create.go:252] error running [docker network inspect minikube]: docker network inspect minikube: exit status 1
stdout:
[]

stderr:
Error: No such network: minikube
I0217 15:07:35.793989   19706 network_create.go:254] output of [docker network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr ** 
Error: No such network: minikube

** /stderr **
I0217 15:07:35.794080   19706 cli_runner.go:111] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}"
��WW���[K�[K/ I0217 15:07:35.835834   19706 network_create.go:104] attempt to create network 192.168.49.0/24 with subnet: minikube and gateway 192.168.49.1 and MTU of 1500 ...
I0217 15:07:35.835968   19706 cli_runner.go:111] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true minikube
��WW���[K�[K- I0217 15:07:35.923397   19706 kic.go:100] calculated static IP "192.168.49.2" for the "minikube" container
I0217 15:07:35.923475   19706 cli_runner.go:111] Run: docker ps -a --format {{.Names}}
I0217 15:07:35.956977   19706 cli_runner.go:111] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0217 15:07:35.994476   19706 oci.go:102] Successfully created a docker volume minikube
I0217 15:07:35.994587   19706 cli_runner.go:111] Run: docker run --rm --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -d /var/lib
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ I0217 15:07:36.668200   19706 oci.go:106] Successfully prepared a docker volume minikube
W0217 15:07:36.668300   19706 oci.go:159] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0217 15:07:36.668339   19706 preload.go:97] Checking if preload exists for k8s version v1.20.2 and runtime docker
W0217 15:07:36.668377   19706 oci.go:201] Your kernel does not support CPU cfs period/quota or the cgroup is not mounted.
I0217 15:07:36.668387   19706 preload.go:105] Found local preload: /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4
I0217 15:07:36.668419   19706 kic.go:163] Starting extracting preloaded images to volume ...
I0217 15:07:36.668467   19706 cli_runner.go:111] Run: docker info --format "'{{json .SecurityOptions}}'"
I0217 15:07:36.668500   19706 cli_runner.go:111] Run: docker run --rm --entrypoint /usr/bin/tar -v /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -I lz4 -xf /preloaded.tar -C /extractDir
��WW���[K�[K- I0217 15:07:36.755964   19706 cli_runner.go:111] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| W0217 15:07:37.402352   19706 cli_runner.go:149] docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e returned with exit code 126
I0217 15:07:37.402440   19706 client.go:171] LocalClient.Create took 1.707042448s
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| I0217 15:07:39.402966   19706 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0217 15:07:39.403086   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K/ W0217 15:07:39.508073   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:39.508268   19706 retry.go:31] will retry after 164.129813ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K- ��WW���[K�[K\ I0217 15:07:39.673011   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:07:39.719750   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:39.719869   19706 retry.go:31] will retry after 223.863569ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- I0217 15:07:39.944674   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0217 15:07:39.992719   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:39.992804   19706 retry.go:31] will retry after 450.512921ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ I0217 15:07:40.443647   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K- I0217 15:07:40.530096   19706 cli_runner.go:155] Completed: docker run --rm --entrypoint /usr/bin/tar -v /usr/local/google/home/tonghe/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e -I lz4 -xf /preloaded.tar -C /extractDir: (3.861556559s)
I0217 15:07:40.530138   19706 kic.go:172] duration metric: took 3.861716 seconds to extract preloaded images to volume
W0217 15:07:40.548396   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:40.548561   19706 retry.go:31] will retry after 856.585655ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:07:41.406068   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0217 15:07:41.464030   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
W0217 15:07:41.464225   19706 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

W0217 15:07:41.464277   19706 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
I0217 15:07:41.464301   19706 start.go:129] duration metric: createHost completed in 5.789952461s
I0217 15:07:41.464449   19706 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0217 15:07:41.464533   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:07:41.514174   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:41.514345   19706 retry.go:31] will retry after 328.409991ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:07:41.843984   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:07:41.899527   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:41.899694   19706 retry.go:31] will retry after 267.848952ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- I0217 15:07:42.168821   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K\ W0217 15:07:42.221233   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:42.221436   19706 retry.go:31] will retry after 495.369669ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| I0217 15:07:42.718007   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0217 15:07:42.766278   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
I0217 15:07:42.766460   19706 retry.go:31] will retry after 690.236584ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ ��WW���[K�[K| ��WW���[K�[K/ ��WW���[K�[K- ��WW���[K�[K\ I0217 15:07:43.457643   19706 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
��WW���[K�[K| W0217 15:07:43.552523   19706 cli_runner.go:149] docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 1
W0217 15:07:43.552714   19706 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

W0217 15:07:43.552746   19706 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil
I0217 15:07:43.552766   19706 fix.go:56] fixHost completed within 3m18.110572206s
I0217 15:07:43.552787   19706 start.go:80] releasing machines lock for "minikube", held for 3m18.110640072s
W0217 15:07:43.553124   19706 out.go:181] * Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max: permission denied: unknown.

��WW���[K�[K
* Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max: permission denied: unknown.

I0217 15:07:43.569988   19706 out.go:119] 

W0217 15:07:43.570247   19706 out.go:181] X Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max: permission denied: unknown.

X Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=15900mb --memory-swap=15900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-4f72870f4e3d4b04680e56e1795de9d3030b4bc2a2bcb1d481d23eb06b325d43.scope/memory.swap.max: permission denied: unknown.

W0217 15:07:43.570340   19706 out.go:181] * 
* 
W0217 15:07:43.570442   19706 out.go:181] * If the above advice does not help, please let us know: 
* If the above advice does not help, please let us know: 
W0217 15:07:43.570506   19706 out.go:181]   - https://github.com/kubernetes/minikube/issues/new/choose
  - https://github.com/kubernetes/minikube/issues/new/choose
I0217 15:07:43.580369   19706 out.go:119]

The text was updated successfully, but these errors were encountered:

hetong07 · 2021-02-17T23:42:16Z

However, if I run with minikube start --driver=none, it starts correctly.

hetong07 · 2021-02-18T01:07:24Z

Dig up a little bit:

After the start failure, I immediately check the docker and got:

$ docker ps -a
CONTAINER ID   IMAGE                                 COMMAND                  CREATED         STATUS    PORTS                                                                                                      NAMES
e6d6770fedf4   gcr.io/k8s-minikube/kicbase:v0.0.17   "/usr/local/bin/entr…"   4 minutes ago   Created   127.0.0.1:49196->22/tcp, 127.0.0.1:49195->2376/tcp, 127.0.0.1:49194->5000/tcp, 127.0.0.1:49193->8443/tcp   minikube

If I then use docker container start e6d6770fedf4 , I will receive:

Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-e6d6770fedf4f9dab9331df44b5f644956013d6b765cabe540e33cac665b1f21.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-e6d6770fedf4f9dab9331df44b5f644956013d6b765cabe540e33cac665b1f21.scope/memory.swap.max: permission denied: unknown
Error: failed to start containers: e6d6770fedf4

So the problem seems to be the docker container has been created, but somehow could not be started?

afbjorklund · 2021-02-18T16:59:35Z

Looks related to cgroups v2 (can you confirm?), so probably the same issue as: #10371

$ docker info | grep "Cgroup Version"
 Cgroup Version: 1
WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support

If indeed --memory-swap doesn't work anymore, we should make that flag conditional...

$ docker run --memory 256m --memory-swap 256m busybox echo Hello World
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Hello World

afbjorklund · 2021-02-18T17:06:13Z

However, if I run with minikube start --driver=none, it starts correctly.

Then the kubelet runs on the node, and there is no privileged node container.

The kubelet might have similar issues with restricting memory, haven't checked.

It's related to the Debian kernel having certain features disabled by default:

https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

hetong07 · 2021-02-18T22:12:06Z

@afbjorklund Thank you for you reply...

I tried commands you mentioned and below are the outputs:

$ docker info | grep "Cgroup Version"                                                                                                                                    
Cgroup Version: 2
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: Support for cgroup v2 is experimental

$ docker run --memory 256m --memory-swap 256m busybox echo Hello World                                                                                                    
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
4c892f00285e: Pull complete 
Digest: sha256:e1488cb900233d035575f0a7787448cb1fa93bed0ccc0d4efc1963d7d72a8f17
Status: Downloaded newer image for busybox:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-cd70bce022b084fbd59f5aff3af9f8fd238cc637bae1e968d8c1db1467e21646.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-cd70bce022b084fbd59f5aff3af9f8fd238cc637bae1e968d8c1db1467e21646.scope/memory.swap.max: permission denied: unknown.
ERRO[0001] error waiting for container: context canceled

So the solution is to check the Cgroup version and make the parameter optional? It so, I can help make that change....

tstromberg · 2021-02-18T22:22:56Z

This issue appears to be a duplicate of #10371, do you mind if we move the conversation there?

Ths way we can centralize the content relating to the issue. If you feel that this issue is not in fact a duplicate, please re-open it using /reopen. If you have additional information to share, please add it to the new issue.

Thank you for reporting this!

hetong07 changed the title ~~Minikube start failure due to 'minikube' docker container missing~~ Failed to start minikube on Debian Feb 17, 2021

afbjorklund added co/docker-driver Issues related to kubernetes in container os/linux kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Feb 18, 2021

tstromberg changed the title ~~Failed to start minikube on Debian~~ Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. Feb 18, 2021

tstromberg closed this as completed Feb 18, 2021

ReyRen mentioned this issue May 7, 2022

Failed to initialize NVML: Unknown Error without any kublet update(cpu-manager-policy is default none) NVIDIA/nvidia-docker#1618

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

hetong07 commented Feb 17, 2021 •

edited

Loading

hetong07 commented Feb 17, 2021

hetong07 commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021

hetong07 commented Feb 18, 2021 •

edited

Loading

tstromberg commented Feb 18, 2021

Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

Comments

hetong07 commented Feb 17, 2021 • edited Loading

hetong07 commented Feb 17, 2021

hetong07 commented Feb 18, 2021 • edited Loading

afbjorklund commented Feb 18, 2021 • edited Loading

afbjorklund commented Feb 18, 2021

hetong07 commented Feb 18, 2021 • edited Loading

tstromberg commented Feb 18, 2021

hetong07 commented Feb 17, 2021 •

edited

Loading

hetong07 commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

hetong07 commented Feb 18, 2021 •

edited

Loading