Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minikube with podman on macOS darwin/amd64 doesn't work #12658

Closed
kgibm opened this issue Oct 5, 2021 · 32 comments · Fixed by #14290
Closed

minikube with podman on macOS darwin/amd64 doesn't work #12658

kgibm opened this issue Oct 5, 2021 · 32 comments · Fixed by #14290
Labels
co/podman-driver podman driver issues kind/support Categorizes issue or PR as a support question. long-term-support Long-term support issues that can't be fixed in code os/macos

Comments

@kgibm
Copy link
Contributor

kgibm commented Oct 5, 2021

Steps to reproduce the issue:

  1. Install minikube with brew:
    $ minikube version
    minikube version: v1.23.2
    commit: 0a0ad764652082477c00d51d2475284b5d39ceed
    
  2. Following the minikube podman instructions, create and start the VM with podman machine:
    $ podman machine ls
    NAME                    VM TYPE     CREATED        LAST UP            CPUS        MEMORY      DISK SIZE
    podman-machine-default  qemu        7 minutes ago  Currently running  4           8.59GB      53.69GB
    
  3. Start minikube with the podman driver and CRI-O container runtime fails:
    $ minikube start --driver=podman --container-runtime=cri-o 
    😄  minikube v1.23.2 on Darwin 11.6
    ✨  Using the podman (experimental) driver based on user configuration
    
    ❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'podman' is not supported on darwin/amd64
    
  4. Try the second method:
    $ minikube start --driver=podman
    😄  minikube v1.23.2 on Darwin 11.6
    ✨  Using the podman (experimental) driver based on user configuration
    
    ❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'podman' is not supported on darwin/amd64
    

Run minikube logs --file=logs.txt and drag and drop the log file into this issue

$ minikube logs --file=logs.txt
🤷  Profile "minikube" not found. Run "minikube profile list" to view all profiles.
👉  To start a cluster, run: "minikube start"

If this is a known issue, maybe it should be added to the "Known Issues" section?

Full output of failed command if not minikube start:

% minikube start --alsologtostderr -v=7 --driver=podman --container-runtime=cri-o
I1005 09:39:38.863234   21025 out.go:298] Setting OutFile to fd 1 ...
I1005 09:39:38.863516   21025 out.go:350] isatty.IsTerminal(1) = true
I1005 09:39:38.863524   21025 out.go:311] Setting ErrFile to fd 2...
I1005 09:39:38.863531   21025 out.go:350] isatty.IsTerminal(2) = true
I1005 09:39:38.863622   21025 root.go:313] Updating PATH: /Users/kevin/.minikube/bin
W1005 09:39:38.863741   21025 root.go:291] Error reading config file at /Users/kevin/.minikube/config/config.json: open /Users/kevin/.minikube/config/config.json: no such file or directory
I1005 09:39:38.863940   21025 out.go:305] Setting JSON to false
I1005 09:39:38.900145   21025 start.go:111] hostinfo: {"hostname":"Kevins-MBP.attlocal.net","uptime":1008238,"bootTime":1632443740,"procs":464,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"11.6","kernelVersion":"20.6.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"c2591f4e-ee82-33cc-8c59-db81d9ad80dd"}
W1005 09:39:38.900322   21025 start.go:119] gopshost.Virtualization returned error: not implemented yet
I1005 09:39:38.922270   21025 out.go:177] 😄  minikube v1.23.2 on Darwin 11.6
😄  minikube v1.23.2 on Darwin 11.6
I1005 09:39:38.922492   21025 notify.go:169] Checking for updates...
I1005 09:39:38.923318   21025 driver.go:343] Setting default libvirt URI to qemu:///system
I1005 09:39:39.426347   21025 podman.go:121] podman version: 3.3.1
I1005 09:39:39.448592   21025 out.go:177] ✨  Using the podman (experimental) driver based on user configuration
✨  Using the podman (experimental) driver based on user configuration
I1005 09:39:39.448637   21025 start.go:278] selected driver: podman
I1005 09:39:39.448643   21025 start.go:751] validating driver "podman" against <nil>
I1005 09:39:39.491992   21025 out.go:177] 

W1005 09:39:39.492219   21025 out.go:242] ❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'podman' is not supported on darwin/amd64
❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'podman' is not supported on darwin/amd64
@afbjorklund
Copy link
Collaborator

afbjorklund commented Oct 5, 2021

See #12547 (and PR #12579 for "allowing" it to run). You probably need a newer/unrelased version of minikube

@afbjorklund afbjorklund added co/podman-driver podman driver issues os/macos co/runtime/crio CRIO related issues and removed co/runtime/crio CRIO related issues labels Oct 5, 2021
@kgibm
Copy link
Contributor Author

kgibm commented Oct 5, 2021

Ok, thanks. Tried building from source and received "error storing credentials in temporary auth file":

$ git clone https://github.com/kubernetes/minikube.git
[...]
$ cd minikube
$ make darwin
[...]
$ ./out/minikube-darwin-amd64 start --driver=podman --container-runtime=cri-o
😄  minikube v1.23.2 on Darwin 11.6
✨  Using the podman (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.22.2 preload ...
    > gcr.io/k8s-minikube/kicbase...: 355.78 MiB / 355.78 MiB  100.00% 4.93 MiB
    > preloaded-images-k8s-v13-v1...: 579.88 MiB / 579.88 MiB  100.00% 6.68 MiB
E1005 09:57:43.652236   22596 cache.go:201] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=4000MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: setting up container node: preparing volume for minikube container: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -d /var/lib: exit status 125
stdout:

stderr:
Error: failed to parse "X-Registry-Auth" header for /v3.4.0/libpod/images/pull?alltags=false&arch=&authfile=&os=&password=&policy=missing&quiet=false&reference=gcr.io%2Fk8s-minikube%2Fkicbase-builds%3Av0.0.27-1633027942-12081&skiptlsverify=false&tlsVerify=true&username=&variant=: error storing credentials in temporary auth file (server: "https://index.docker.io/v1/", user: ""): key https://index.docker.io/v1/ contains http[s]:// prefix

🔄  Restarting existing podman container for "minikube" ...
😿  Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"


❌  Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f  minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"


╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

Same result without CRI-O.

@kgibm
Copy link
Contributor Author

kgibm commented Oct 5, 2021

Full log with --alsologtostderr -v=7:


$ ./out/minikube-darwin-amd64 start --alsologtostderr -v=7 --driver=podman --container-runtime=cri-o
I1005 10:00:55.064735   22635 out.go:298] Setting OutFile to fd 1 ...
I1005 10:00:55.065028   22635 out.go:350] isatty.IsTerminal(1) = true
I1005 10:00:55.065036   22635 out.go:311] Setting ErrFile to fd 2...
I1005 10:00:55.065042   22635 out.go:350] isatty.IsTerminal(2) = true
I1005 10:00:55.065150   22635 root.go:313] Updating PATH: /Users/kevin/.minikube/bin
W1005 10:00:55.065266   22635 root.go:291] Error reading config file at /Users/kevin/.minikube/config/config.json: open /Users/kevin/.minikube/config/config.json: no such file or directory
I1005 10:00:55.065470   22635 out.go:305] Setting JSON to false
I1005 10:00:55.101661   22635 start.go:111] hostinfo: {"hostname":"Kevins-MBP.attlocal.net","uptime":1009515,"bootTime":1632443740,"procs":470,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"11.6","kernelVersion":"20.6.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"c2591f4e-ee82-33cc-8c59-db81d9ad80dd"}
W1005 10:00:55.101792   22635 start.go:119] gopshost.Virtualization returned error: not implemented yet
I1005 10:00:55.125416   22635 out.go:177] 😄  minikube v1.23.2 on Darwin 11.6
😄  minikube v1.23.2 on Darwin 11.6
I1005 10:00:55.125626   22635 notify.go:169] Checking for updates...
I1005 10:00:55.126418   22635 config.go:177] Loaded profile config "minikube": Driver=podman, ContainerRuntime=crio, KubernetesVersion=v1.22.2
I1005 10:00:55.148529   22635 driver.go:343] Setting default libvirt URI to qemu:///system
I1005 10:00:55.634660   22635 podman.go:121] podman version: 3.3.1
I1005 10:00:55.656333   22635 out.go:177] ✨  Using the podman (experimental) driver based on existing profile
✨  Using the podman (experimental) driver based on existing profile
I1005 10:00:55.656374   22635 start.go:278] selected driver: podman
I1005 10:00:55.656382   22635 start.go:751] validating driver "podman" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:4000 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s}
I1005 10:00:55.656497   22635 start.go:762] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:}
I1005 10:00:55.656615   22635 cli_runner.go:115] Run: podman system info --format json
I1005 10:00:55.856573   22635 info.go:285] podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7557607424 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:30m 58.38s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:00:55.857009   22635 cni.go:93] Creating CNI manager for ""
I1005 10:00:55.857029   22635 cni.go:160] "podman" driver + crio runtime found, recommending kindnet
I1005 10:00:55.857039   22635 start_flags.go:280] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:4000 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s}
I1005 10:00:55.902217   22635 out.go:177] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I1005 10:00:55.902360   22635 cache.go:118] Beginning downloading kic base image for podman with crio
I1005 10:00:55.924058   22635 out.go:177] 🚜  Pulling base image ...
🚜  Pulling base image ...
I1005 10:00:55.924362   22635 preload.go:131] Checking if preload exists for k8s version v1.22.2 and runtime crio
I1005 10:00:55.924401   22635 cache.go:146] Downloading gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c to local cache
I1005 10:00:55.924567   22635 preload.go:147] Found local preload: /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4
I1005 10:00:55.924653   22635 cache.go:57] Caching tarball of preloaded images
I1005 10:00:55.925051   22635 preload.go:173] Found /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4 in cache, skipping download
I1005 10:00:55.925068   22635 image.go:59] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory
I1005 10:00:55.925120   22635 image.go:62] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory, skipping pull
I1005 10:00:55.925129   22635 cache.go:60] Finished verifying existence of preloaded tar for  v1.22.2 on crio
I1005 10:00:55.925135   22635 image.go:103] gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c exists in cache, skipping pull
I1005 10:00:55.925197   22635 cache.go:149] successfully saved gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c as a tarball
I1005 10:00:55.925312   22635 profile.go:148] Saving config to /Users/kevin/.minikube/profiles/minikube/config.json ...
E1005 10:00:55.947015   22635 cache.go:201] Error downloading kic artifacts:  not yet implemented, see issue #8426
I1005 10:00:55.947038   22635 cache.go:206] Successfully downloaded all kic artifacts
I1005 10:00:55.947081   22635 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1005 10:00:55.947224   22635 start.go:317] acquired machines lock for "minikube" in 115.333µs
I1005 10:00:55.947276   22635 start.go:93] Skipping create...Using existing machine configuration
I1005 10:00:55.947290   22635 fix.go:55] fixHost starting: 
I1005 10:00:55.947645   22635 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
W1005 10:00:56.113421   22635 cli_runner.go:162] podman container inspect minikube --format={{.State.Status}} returned with exit code 125
I1005 10:00:56.113515   22635 fix.go:108] recreateIfNeeded on minikube: state= err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:00:56.113560 22635 fix.go:113] machineExists: true. err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W1005 10:00:56.113571 22635 fix.go:134] unexpected machine state, will restart: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:00:56.157438 22635 out.go:177] 🔄 Restarting existing podman container for "minikube" ...
🔄 Restarting existing podman container for "minikube" ...
I1005 10:00:56.157568 22635 cli_runner.go:115] Run: podman start minikube
W1005 10:00:56.325863 22635 cli_runner.go:162] podman start minikube returned with exit code 125
I1005 10:00:56.326020 22635 cli_runner.go:115] Run: podman inspect minikube
I1005 10:00:56.490318 22635 errors.go:84] Postmortem inspect ("podman inspect minikube"): -- stdout --
[
{
"Name": "minikube",
"Driver": "local",
"Mountpoint": "/var/lib/containers/storage/volumes/minikube/_data",
"CreatedAt": "2021-10-05T16:57:45.176823624Z",
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"name.minikube.sigs.k8s.io": "minikube"
},
"Scope": "local",
"Options": {}
}
]

-- /stdout --
I1005 10:00:56.490493 22635 cli_runner.go:115] Run: podman logs --timestamps minikube
W1005 10:00:56.652366 22635 cli_runner.go:162] podman logs --timestamps minikube returned with exit code 125
W1005 10:00:56.652403 22635 errors.go:89] Failed to get postmortem logs. podman logs --timestamps minikube :podman logs --timestamps minikube: exit status 125
stdout:

stderr:
Error: channel "123" found, 0-3 supported: lost synchronization with multiplexed stream
I1005 10:00:56.652510 22635 cli_runner.go:115] Run: podman system info --format json
I1005 10:00:56.853063 22635 info.go:285] podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7557095424 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:30m 59.38s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:00:56.853150 22635 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7557095424 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:30m 59.38s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:00:56.853319 22635 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
I1005 10:00:56.853365 22635 cli_runner.go:115] Run: podman network inspect minikube
I1005 10:00:57.009281 22635 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[
{
"args": {
"podman_labels": {
"created_by.minikube.sigs.k8s.io": "true"
}
},
"cniVersion": "0.4.0",
"name": "minikube",
"plugins": [
{
"bridge": "cni-podman1",
"hairpinMode": true,
"ipMasq": true,
"ipam": {
"ranges": [
[
{
"gateway": "192.168.49.1",
"subnet": "192.168.49.0/24"
}
]
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"type": "host-local"
},
"isGateway": true,
"type": "bridge"
},
{
"capabilities": {
"portMappings": true
},
"type": "portmap"
},
{
"backend": "",
"type": "firewall"
},
{
"type": "tuning"
},
{
"capabilities": {
"aliases": true
},
"domainName": "dns.podman",
"type": "dnsname"
},
{
"capabilities": {
"portMappings": true
},
"type": "podman-machine"
}
]
}
]

-- /stdout --
I1005 10:00:57.009470 22635 cli_runner.go:115] Run: podman system info --format json
I1005 10:00:57.202219 22635 info.go:285] podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7556485120 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:30m 59.73s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:00:57.203396 22635 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1005 10:00:57.359836 22635 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1005 10:00:57.360011 22635 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I1005 10:00:57.360107 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:00:57.547995 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:00:57.705620 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1005 10:00:57.705746 22635 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:00:57.982523 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:00:58.179573 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:00:58.332459 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1005 10:00:58.332588 22635 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:00:58.874840 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:00:59.075148 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:00:59.237946 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
W1005 10:00:59.238075 22635 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W1005 10:00:59.238112 22635 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:00:59.238132 22635 fix.go:57] fixHost completed within 3.290750691s
I1005 10:00:59.238141 22635 start.go:80] releasing machines lock for "minikube", held for 3.290807925s
W1005 10:00:59.238181 22635 start.go:522] error starting host: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W1005 10:00:59.238366 22635 out.go:242] 🤦 StartHost failed, but will try again: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

🤦 StartHost failed, but will try again: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

I1005 10:00:59.238391 22635 start.go:537] Will try again in 5 seconds ...
I1005 10:01:04.242762 22635 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1005 10:01:04.242999 22635 start.go:317] acquired machines lock for "minikube" in 180.321µs
I1005 10:01:04.243039 22635 start.go:93] Skipping create...Using existing machine configuration
I1005 10:01:04.243046 22635 fix.go:55] fixHost starting:
I1005 10:01:04.243681 22635 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
W1005 10:01:04.555304 22635 cli_runner.go:162] podman container inspect minikube --format={{.State.Status}} returned with exit code 125
I1005 10:01:04.555359 22635 fix.go:108] recreateIfNeeded on minikube: state= err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:01:04.555370 22635 fix.go:113] machineExists: true. err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W1005 10:01:04.555391 22635 fix.go:134] unexpected machine state, will restart: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:01:04.577232 22635 out.go:177] 🔄 Restarting existing podman container for "minikube" ...
🔄 Restarting existing podman container for "minikube" ...
I1005 10:01:04.577473 22635 cli_runner.go:115] Run: podman start minikube
W1005 10:01:04.763694 22635 cli_runner.go:162] podman start minikube returned with exit code 125
I1005 10:01:04.763823 22635 cli_runner.go:115] Run: podman inspect minikube
I1005 10:01:04.919321 22635 errors.go:84] Postmortem inspect ("podman inspect minikube"): -- stdout --
[
{
"Name": "minikube",
"Driver": "local",
"Mountpoint": "/var/lib/containers/storage/volumes/minikube/_data",
"CreatedAt": "2021-10-05T16:57:45.176823624Z",
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"name.minikube.sigs.k8s.io": "minikube"
},
"Scope": "local",
"Options": {}
}
]

-- /stdout --
I1005 10:01:04.919439 22635 cli_runner.go:115] Run: podman logs --timestamps minikube
W1005 10:01:05.069699 22635 cli_runner.go:162] podman logs --timestamps minikube returned with exit code 125
W1005 10:01:05.069729 22635 errors.go:89] Failed to get postmortem logs. podman logs --timestamps minikube :podman logs --timestamps minikube: exit status 125
stdout:

stderr:
Error: channel "123" found, 0-3 supported: lost synchronization with multiplexed stream
I1005 10:01:05.069845 22635 cli_runner.go:115] Run: podman system info --format json
I1005 10:01:05.274836 22635 info.go:285] podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7556333568 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:31m 7.8s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:01:05.274901 22635 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7556333568 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:31m 7.8s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:01:05.275029 22635 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
I1005 10:01:05.275048 22635 cli_runner.go:115] Run: podman network inspect minikube
I1005 10:01:05.421064 22635 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[
{
"args": {
"podman_labels": {
"created_by.minikube.sigs.k8s.io": "true"
}
},
"cniVersion": "0.4.0",
"name": "minikube",
"plugins": [
{
"bridge": "cni-podman1",
"hairpinMode": true,
"ipMasq": true,
"ipam": {
"ranges": [
[
{
"gateway": "192.168.49.1",
"subnet": "192.168.49.0/24"
}
]
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"type": "host-local"
},
"isGateway": true,
"type": "bridge"
},
{
"capabilities": {
"portMappings": true
},
"type": "portmap"
},
{
"backend": "",
"type": "firewall"
},
{
"type": "tuning"
},
{
"capabilities": {
"aliases": true
},
"domainName": "dns.podman",
"type": "dnsname"
},
{
"capabilities": {
"portMappings": true
},
"type": "podman-machine"
}
]
}
]

-- /stdout --
I1005 10:01:05.421199 22635 cli_runner.go:115] Run: podman system info --format json
I1005 10:01:05.600424 22635 info.go:285] podman info: {Host:{BuildahVersion:1.22.3 CgroupVersion:v2 Conmon:{Package:conmon-2.0.29-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.29, commit: } Distribution:{Distribution:fedora Version:34} MemFree:7556325376 MemTotal:8325619712 OCIRuntime:{Name:crun Package:crun-1.0-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.13.16-200.fc34.x86_64 Os:linux Rootless:false Uptime:31m 8.13s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1005 10:01:05.601522 22635 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1005 10:01:05.749324 22635 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1005 10:01:05.749518 22635 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I1005 10:01:05.749593 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:01:05.937270 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:01:06.089342 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1005 10:01:06.089479 22635 retry.go:31] will retry after 234.428547ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:01:06.325812 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:01:06.508992 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:01:06.662312 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1005 10:01:06.662471 22635 retry.go:31] will retry after 346.739061ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:01:07.013417 22635 cli_runner.go:115] Run: podman version --format {{.Version}}
I1005 10:01:07.208187 22635 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1005 10:01:07.356677 22635 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
W1005 10:01:07.356812 22635 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W1005 10:01:07.356838 22635 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1005 10:01:07.356853 22635 fix.go:57] fixHost completed within 3.113719634s
I1005 10:01:07.356863 22635 start.go:80] releasing machines lock for "minikube", held for 3.113760034s
W1005 10:01:07.357150 22635 out.go:242] 😿 Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

😿 Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

I1005 10:01:07.400943 22635 out.go:177]

W1005 10:01:07.401211 22635 out.go:242] ❌ Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

❌ Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W1005 10:01:07.401244 22635 out.go:242]

W1005 10:01:07.402513 22635 out.go:242] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I1005 10:01:07.466854 22635 out.go:177]

@afbjorklund
Copy link
Collaborator

afbjorklund commented Oct 5, 2021

Podman stumbles on some Docker config, so you need to move that aside...

You might want to try some simpler containers first, before trying kicbase

podman run docker.io/hello-world

@kgibm
Copy link
Contributor Author

kgibm commented Oct 5, 2021

Thanks. Now I'm hitting podman run returns error "unexpected end of JSON input" so I'll wait for that fix to roll out.

@spowelljr spowelljr added the kind/support Categorizes issue or PR as a support question. label Oct 6, 2021
@kgibm
Copy link
Contributor Author

kgibm commented Oct 13, 2021

Solved the "unexpected end of JSON input" issue with the latest CoreOS so I can now run podman, and re-built from latest minikube code, but still can't start minikube. The error seems to be "Error: the requested cgroup controller cpu is not available: OCI runtime error". Below is output with --alsologtostderr -v=7. I also tried without --container-runtime=cri-o with the same error and I do not have a ~/.docker/config.json file.


% ./out/minikube-darwin-amd64 start --alsologtostderr -v=7 --driver=podman --container-runtime=cri-o
I1013 07:09:22.387309   75820 out.go:297] Setting OutFile to fd 1 ...
I1013 07:09:22.387583   75820 out.go:349] isatty.IsTerminal(1) = true
I1013 07:09:22.387593   75820 out.go:310] Setting ErrFile to fd 2...
I1013 07:09:22.387600   75820 out.go:349] isatty.IsTerminal(2) = true
I1013 07:09:22.387727   75820 root.go:313] Updating PATH: /Users/kevin/.minikube/bin
I1013 07:09:22.388662   75820 out.go:304] Setting JSON to false
I1013 07:09:22.431306   75820 start.go:111] hostinfo: {"hostname":"Kevins-MBP.attlocal.net","uptime":74015,"bootTime":1634060147,"procs":462,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"11.6","kernelVersion":"20.6.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"c2591f4e-ee82-33cc-8c59-db81d9ad80dd"}
W1013 07:09:22.431560   75820 start.go:119] gopshost.Virtualization returned error: not implemented yet
I1013 07:09:22.457693   75820 out.go:176] 😄  minikube v1.23.2 on Darwin 11.6
😄  minikube v1.23.2 on Darwin 11.6
I1013 07:09:22.457864   75820 notify.go:169] Checking for updates...
I1013 07:09:22.458258   75820 driver.go:343] Setting default libvirt URI to qemu:///system
I1013 07:09:23.068096   75820 podman.go:121] podman version: 3.4.0
I1013 07:09:23.090221   75820 out.go:176] ✨  Using the podman (experimental) driver based on user configuration
✨  Using the podman (experimental) driver based on user configuration
I1013 07:09:23.090274   75820 start.go:278] selected driver: podman
I1013 07:09:23.090281   75820 start.go:751] validating driver "podman" against 
I1013 07:09:23.090296   75820 start.go:762] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:}
I1013 07:09:23.090437   75820 cli_runner.go:115] Run: podman system info --format json
I1013 07:09:23.342773   75820 info.go:285] podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:34} MemFree:5589291008 MemTotal:10426978304 OCIRuntime:{Name:crun Package:crun-1.2-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.2
commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.14.9-200.fc34.x86_64 Os:linux Rootless:false Uptime:17h 46m 20.87s (Approximately 0.71 days)} Registries:{Search:[docker.io]} Store:{ConfigFile:/var/home/core/.config/containers/storage.conf ContainerStore:{Number:1} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/home/core/.local/share/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:16} RunRoot:/run/user/1000/containers VolumePath:/var/home/core/.local/share/containers/storage/volumes}}
I1013 07:09:23.343040   75820 start_flags.go:266] no existing cluster config was found, will generate one from the flags 
I1013 07:09:23.343265   75820 start_flags.go:347] Using suggested 4000MB memory alloc based on sys=16384MB, container=9943MB
I1013 07:09:23.343356   75820 start_flags.go:727] Wait components to verify : map[apiserver:true system_pods:true]
I1013 07:09:23.343377   75820 cni.go:93] Creating CNI manager for ""
I1013 07:09:23.343387   75820 cni.go:160] "podman" driver + crio runtime found, recommending kindnet
I1013 07:09:23.343396   75820 start_flags.go:275] Found "CNI" CNI - setting NetworkPlugin=cni
I1013 07:09:23.343408   75820 start_flags.go:280] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:4000 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s}
I1013 07:09:23.387724   75820 out.go:176] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I1013 07:09:23.387792   75820 cache.go:118] Beginning downloading kic base image for podman with crio
I1013 07:09:23.409683   75820 out.go:176] 🚜  Pulling base image ...
🚜  Pulling base image ...
I1013 07:09:23.409790   75820 preload.go:130] Checking if preload exists for k8s version v1.22.2 and runtime crio
I1013 07:09:23.409778   75820 cache.go:146] Downloading gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c to local cache
I1013 07:09:23.409886   75820 preload.go:146] Found local preload: /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4
I1013 07:09:23.409905   75820 cache.go:57] Caching tarball of preloaded images
I1013 07:09:23.410063   75820 image.go:59] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory
I1013 07:09:23.410071   75820 preload.go:172] Found /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4 in cache, skipping download
I1013 07:09:23.410094   75820 image.go:62] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory, skipping pull
I1013 07:09:23.410102   75820 cache.go:60] Finished verifying existence of preloaded tar for  v1.22.2 on crio
I1013 07:09:23.410106   75820 image.go:103] gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c exists in cache, skipping pull
I1013 07:09:23.410137   75820 cache.go:149] successfully saved gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c as a tarball
I1013 07:09:23.410452   75820 profile.go:147] Saving config to /Users/kevin/.minikube/profiles/minikube/config.json ...
I1013 07:09:23.410512   75820 lock.go:35] WriteFile acquiring /Users/kevin/.minikube/profiles/minikube/config.json: {Name:mke28f4a3b281daafb742f57c9cfb025182acec4 Clock:{} Delay:500ms Timeout:1m0s Cancel:}
E1013 07:09:23.431100   75820 cache.go:201] Error downloading kic artifacts:  not yet implemented, see issue #8426
I1013 07:09:23.431136   75820 cache.go:206] Successfully downloaded all kic artifacts
I1013 07:09:23.431184   75820 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1013 07:09:23.431339   75820 start.go:317] acquired machines lock for "minikube" in 135.715µs
I1013 07:09:23.431386   75820 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:4000 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}
I1013 07:09:23.431539   75820 start.go:126] createHost starting for "" (driver="podman")
I1013 07:09:23.453835   75820 out.go:203] 🔥  Creating podman container (CPUs=2, Memory=4000MB) ...
🔥  Creating podman container (CPUs=2, Memory=4000MB) ...| I1013 07:09:23.454233   75820 start.go:160] libmachine.API.Create for "minikube" (driver="podman")
I1013 07:09:23.454285   75820 client.go:168] LocalClient.Create starting
I1013 07:09:23.454490   75820 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/ca.pem
I1013 07:09:23.455099   75820 main.go:130] libmachine: Decoding PEM data...
I1013 07:09:23.455157   75820 main.go:130] libmachine: Parsing certificate...
I1013 07:09:23.455311   75820 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/cert.pem
I1013 07:09:23.455857   75820 main.go:130] libmachine: Decoding PEM data...
I1013 07:09:23.455896   75820 main.go:130] libmachine: Parsing certificate...
I1013 07:09:23.457068   75820 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
- W1013 07:09:23.686404   75820 cli_runner.go:162] podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}" returned with exit code 125
I1013 07:09:23.686534   75820 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
I1013 07:09:23.686561   75820 cli_runner.go:115] Run: podman network inspect minikube
| W1013 07:09:23.895954   75820 cli_runner.go:162] podman network inspect minikube returned with exit code 125
I1013 07:09:23.896004   75820 network_create.go:256] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: error inspecting object: no such network "minikube"
I1013 07:09:23.896029 75820 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: error inspecting object: no such network "minikube"

** /stderr **
I1013 07:09:23.896205 75820 cli_runner.go:115] Run: podman network inspect podman --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"

  • I1013 07:09:24.115577 75820 network.go:288] reserving subnet 192.168.49.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[] amended:true}} dirty:map[192.168.49.0:0xc000a3c178] misses:0}
    I1013 07:09:24.115628 75820 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
    I1013 07:09:24.115649 75820 network_create.go:106] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
    I1013 07:09:24.115755 75820 cli_runner.go:115] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true minikube
    | I1013 07:09:24.323378 75820 network_create.go:90] podman network minikube 192.168.49.0/24 created
    I1013 07:09:24.323429 75820 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
    I1013 07:09:24.323551 75820 cli_runner.go:115] Run: podman ps -a --format {{.Names}}
  • I1013 07:09:24.545189 75820 cli_runner.go:115] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
    | I1013 07:09:24.771285 75820 oci.go:102] Successfully created a podman volume minikube
    I1013 07:09:24.771460 75820 cli_runner.go:115] Run: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -d /var/lib
    / I1013 07:09:26.034838 75820 cli_runner.go:168] Completed: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -d /var/lib: (1.263301002s)
    I1013 07:09:26.034908 75820 oci.go:106] Successfully prepared a podman volume minikube
    I1013 07:09:26.034984 75820 preload.go:130] Checking if preload exists for k8s version v1.22.2 and runtime crio
    I1013 07:09:26.035014 75820 kic.go:179] Starting extracting preloaded images to volume ...
    I1013 07:09:26.035074 75820 cli_runner.go:115] Run: podman info --format "'{{json .SecurityOptions}}'"
    I1013 07:09:26.035204 75820 cli_runner.go:115] Run: podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir
    | W1013 07:09:26.339483 75820 cli_runner.go:162] podman info --format "'{{json .SecurityOptions}}'" returned with exit code 125
    I1013 07:09:26.339755 75820 cli_runner.go:115] Run: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081
    W1013 07:09:26.386646 75820 cli_runner.go:162] podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir returned with exit code 125
    I1013 07:09:26.386719 75820 kic.go:186] Unable to extract preloaded tarball to volume: podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir: exit status 125
    stdout:

stderr:
Error: statfs /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4: no such file or directory
/ W1013 07:09:27.678357 75820 cli_runner.go:162] podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 returned with exit code 126
I1013 07:09:27.678400 75820 cli_runner.go:168] Completed: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081: (1.338529066s)
I1013 07:09:27.678457 75820 client.go:171] LocalClient.Create took 4.224198613s
| I1013 07:09:29.679568 75820 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I1013 07:09:29.679715 75820 cli_runner.go:115] Run: podman version --format {{.Version}}
\ I1013 07:09:29.992263 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
/ I1013 07:09:30.204365 75820 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:46335 SSHKeyPath:/Users/kevin/.minikube/machines/minikube/id_rsa Username:docker}
I1013 07:09:30.204687 75820 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/Users/kevin/.minikube/machines/minikube/id_rsa]}: open /Users/kevin/.minikube/machines/minikube/id_rsa: no such file or directory
| I1013 07:09:30.482764 75820 cli_runner.go:115] Run: podman version --format {{.Version}}
\ I1013 07:09:30.764575 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
/ I1013 07:09:30.994874 75820 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:46335 SSHKeyPath:/Users/kevin/.minikube/machines/minikube/id_rsa Username:docker}
I1013 07:09:30.995093 75820 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/Users/kevin/.minikube/machines/minikube/id_rsa]}: open /Users/kevin/.minikube/machines/minikube/id_rsa: no such file or directory

  • I1013 07:09:31.538828 75820 cli_runner.go:115] Run: podman version --format {{.Version}}
    / I1013 07:09:31.813576 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    \ I1013 07:09:32.024317 75820 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:46335 SSHKeyPath:/Users/kevin/.minikube/machines/minikube/id_rsa Username:docker}
    W1013 07:09:32.024551 75820 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/Users/kevin/.minikube/machines/minikube/id_rsa]}: open /Users/kevin/.minikube/machines/minikube/id_rsa: no such file or directory
    W1013 07:09:32.024641 75820 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/Users/kevin/.minikube/machines/minikube/id_rsa]}: open /Users/kevin/.minikube/machines/minikube/id_rsa: no such file or directory
    I1013 07:09:32.024706 75820 start.go:129] duration metric: createHost completed in 8.593227285s
    I1013 07:09:32.024723 75820 start.go:80] releasing machines lock for "minikube", held for 8.59344186s
    W1013 07:09:32.024785 75820 start.go:522] error starting host: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081: exit status 126
    stdout:

stderr:
Error: the requested cgroup controller cpu is not available: OCI runtime error
I1013 07:09:32.025500 75820 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
/ I1013 07:09:32.235502 75820 stop.go:39] StopHost: minikube
W1013 07:09:32.236060 75820 register.go:129] "Stopping" was not found within the registered steps for "Initial Minikube Setup": [Initial Minikube Setup Selecting Driver Downloading Artifacts Starting Node Updating Driver Pulling Base Image Running on Localhost Local OS Release Creating Container Creating VM Running Remotely Preparing Kubernetes Generating certificates Booting control plane Configuring RBAC rules Configuring CNI Configuring Localhost Environment Verifying Kubernetes Enabling Addons Done]
I1013 07:09:32.261313 75820 out.go:176] ✋ Stopping node "minikube" ...

✋ Stopping node "minikube" ...
I1013 07:09:32.261507 75820 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:09:32.472234 75820 stop.go:79] host is in state Stopped
I1013 07:09:32.472330 75820 main.go:130] libmachine: Stopping "minikube"...
I1013 07:09:32.472513 75820 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:09:32.683187 75820 stop.go:59] stop err: Machine "minikube" is already stopped.
I1013 07:09:32.683233 75820 stop.go:62] host is already stopped
W1013 07:09:33.686217 75820 register.go:129] "Stopping" was not found within the registered steps for "Initial Minikube Setup": [Initial Minikube Setup Selecting Driver Downloading Artifacts Starting Node Updating Driver Pulling Base Image Running on Localhost Local OS Release Creating Container Creating VM Running Remotely Preparing Kubernetes Generating certificates Booting control plane Configuring RBAC rules Configuring CNI Configuring Localhost Environment Verifying Kubernetes Enabling Addons Done]
I1013 07:09:33.709157 75820 out.go:176] 🔥 Deleting "minikube" in podman ...
🔥 Deleting "minikube" in podman ...
I1013 07:09:33.709378 75820 cli_runner.go:115] Run: podman container inspect -f {{.Id}} minikube
I1013 07:09:33.939610 75820 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:09:34.150644 75820 cli_runner.go:115] Run: podman exec --privileged -t minikube /bin/bash -c "sudo init 0"
W1013 07:09:34.355944 75820 cli_runner.go:162] podman exec --privileged -t minikube /bin/bash -c "sudo init 0" returned with exit code 125
I1013 07:09:34.355989 75820 oci.go:635] error shutdown minikube: podman exec --privileged -t minikube /bin/bash -c "sudo init 0": exit status 125
stdout:

stderr:
Error: can only create exec sessions on running containers: container state improper
I1013 07:09:35.357677 75820 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:09:35.580954 75820 oci.go:643] container minikube status is Stopped
I1013 07:09:35.581013 75820 oci.go:655] Successfully shutdown container minikube
I1013 07:09:35.581252 75820 cli_runner.go:115] Run: podman rm -f -v minikube
I1013 07:09:35.801487 75820 cli_runner.go:115] Run: podman container inspect -f {{.Id}} minikube
W1013 07:09:36.012355 75820 cli_runner.go:162] podman container inspect -f {{.Id}} minikube returned with exit code 125
I1013 07:09:36.012599 75820 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
I1013 07:09:36.249735 75820 cli_runner.go:115] Run: podman network rm minikube
W1013 07:09:36.459922 75820 out.go:241] 🤦 StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081: exit status 126
stdout:

stderr:
Error: the requested cgroup controller cpu is not available: OCI runtime error

🤦 StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=4000mb --memory=4000mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081: exit status 126
stdout:

stderr:
Error: the requested cgroup controller cpu is not available: OCI runtime error

I1013 07:09:36.460002 75820 start.go:537] Will try again in 5 seconds ...
I1013 07:09:41.460546 75820 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1013 07:09:41.460711 75820 start.go:317] acquired machines lock for "minikube" in 121.824µs
I1013 07:09:41.460739 75820 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:4000 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}
I1013 07:09:41.460856 75820 start.go:126] createHost starting for "" (driver="podman")
I1013 07:09:41.508614 75820 out.go:203] 🔥 Creating podman container (CPUs=2, Memory=4000MB) ...
🔥 Creating podman container (CPUs=2, Memory=4000MB) ...| I1013 07:09:41.508768 75820 start.go:160] libmachine.API.Create for "minikube" (driver="podman")
I1013 07:09:41.508796 75820 client.go:168] LocalClient.Create starting
I1013 07:09:41.508933 75820 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/ca.pem
I1013 07:09:41.509086 75820 main.go:130] libmachine: Decoding PEM data...
I1013 07:09:41.509120 75820 main.go:130] libmachine: Parsing certificate...
I1013 07:09:41.509208 75820 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/cert.pem
I1013 07:09:41.509346 75820 main.go:130] libmachine: Decoding PEM data...
I1013 07:09:41.509371 75820 main.go:130] libmachine: Parsing certificate...
I1013 07:09:41.510138 75820 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
| W1013 07:09:42.016286 75820 cli_runner.go:162] podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}" returned with exit code 125
I1013 07:09:42.016407 75820 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
I1013 07:09:42.016430 75820 cli_runner.go:115] Run: podman network inspect minikube
\ W1013 07:09:42.235455 75820 cli_runner.go:162] podman network inspect minikube returned with exit code 125
I1013 07:09:42.235535 75820 network_create.go:256] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: error inspecting object: no such network "minikube"
I1013 07:09:42.235591 75820 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: error inspecting object: no such network "minikube"

** /stderr **
I1013 07:09:42.235762 75820 cli_runner.go:115] Run: podman network inspect podman --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
/ I1013 07:09:42.457796 75820 network.go:279] skipping subnet 192.168.49.0 that has unexpired reservation: &{mu:{state:0 sema:0} read:{v:{m:map[192.168.49.0:0xc000a3c178] amended:false}} dirty:map[] misses:0}
I1013 07:09:42.457834 75820 network.go:238] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I1013 07:09:42.458281 75820 network.go:288] reserving subnet 192.168.58.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[192.168.49.0:0xc000a3c178] amended:true}} dirty:map[192.168.49.0:0xc000a3c178 192.168.58.0:0xc0003084c0] misses:0}
I1013 07:09:42.458312 75820 network.go:235] using free private subnet 192.168.58.0/24: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I1013 07:09:42.458323 75820 network_create.go:106] attempt to create podman network minikube 192.168.58.0/24 with gateway 192.168.58.1 and MTU of 0 ...
I1013 07:09:42.458449 75820 cli_runner.go:115] Run: podman network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 --label=created_by.minikube.sigs.k8s.io=true minikube
\ I1013 07:09:42.688042 75820 network_create.go:90] podman network minikube 192.168.58.0/24 created
I1013 07:09:42.688083 75820 kic.go:106] calculated static IP "192.168.58.2" for the "minikube" container
I1013 07:09:42.688220 75820 cli_runner.go:115] Run: podman ps -a --format {{.Names}}
/ I1013 07:09:42.926256 75820 cli_runner.go:115] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
\ W1013 07:09:43.138944 75820 cli_runner.go:162] podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true returned with exit code 125
I1013 07:09:43.139004 75820 client.go:171] LocalClient.Create took 1.630215631s
\ I1013 07:09:45.141245 75820 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I1013 07:09:45.141385 75820 cli_runner.go:115] Run: podman version --format {{.Version}}

  • I1013 07:09:45.426021 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    | W1013 07:09:45.634813 75820 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
    I1013 07:09:45.634927 75820 retry.go:31] will retry after 234.428547ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"

  • I1013 07:09:45.874497 75820 cli_runner.go:115] Run: podman version --format {{.Version}}
    / I1013 07:09:46.193498 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    \ W1013 07:09:46.412591 75820 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
    I1013 07:09:46.413116 75820 retry.go:31] will retry after 346.739061ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
\ I1013 07:09:46.761230 75820 cli_runner.go:115] Run: podman version --format {{.Version}}

  • I1013 07:09:47.118378 75820 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    | W1013 07:09:47.318931 75820 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
    W1013 07:09:47.319055 75820 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W1013 07:09:47.319076 75820 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:09:47.319092 75820 start.go:129] duration metric: createHost completed in 5.858276472s
I1013 07:09:47.319100 75820 start.go:80] releasing machines lock for "minikube", held for 5.858426985s
W1013 07:09:47.319333 75820 out.go:241] 😿 Failed to start podman container. Running "minikube delete" may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

😿 Failed to start podman container. Running "minikube delete" may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

I1013 07:09:47.367636 75820 out.go:176]

W1013 07:09:47.367828 75820 out.go:241] ❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

W1013 07:09:47.367852 75820 out.go:241]

W1013 07:09:47.368793 75820 out.go:241] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I1013 07:09:47.457658 75820 out.go:176]

@afbjorklund
Copy link
Collaborator

Did you run the "root" Podman ?
See podman system connection list

@kgibm
Copy link
Contributor Author

kgibm commented Oct 13, 2021

You're right, I was not using "root" podman. After switching, now there's an error no such network "minikube":


kevin@Kevins-MBP minikube % podman system connection default podman-machine-default-root
kevin@Kevins-MBP minikube % podman machine ssh whoami
Warning: Permanently added '[localhost]:59679' (ECDSA) to the list of known hosts.
root
kevin@Kevins-MBP minikube % ./out/minikube-darwin-amd64 delete                                                                                       
🔥  Deleting "minikube" in podman ...
🔥  Removing /Users/kevin/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
kevin@Kevins-MBP minikube % ./out/minikube-darwin-amd64 start --alsologtostderr -v=7 --driver=podman --container-runtime=cri-o --memory 8192 --cpus 2
I1013 07:52:10.867893   76994 out.go:297] Setting OutFile to fd 1 ...
I1013 07:52:10.868113   76994 out.go:349] isatty.IsTerminal(1) = true
I1013 07:52:10.868121   76994 out.go:310] Setting ErrFile to fd 2...
I1013 07:52:10.868127   76994 out.go:349] isatty.IsTerminal(2) = true
I1013 07:52:10.868224   76994 root.go:313] Updating PATH: /Users/kevin/.minikube/bin
I1013 07:52:10.868663   76994 out.go:304] Setting JSON to false
I1013 07:52:10.908046   76994 start.go:111] hostinfo: {"hostname":"Kevins-MBP.attlocal.net","uptime":76583,"bootTime":1634060147,"procs":466,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"11.6","kernelVersion":"20.6.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"c2591f4e-ee82-33cc-8c59-db81d9ad80dd"}
W1013 07:52:10.908212   76994 start.go:119] gopshost.Virtualization returned error: not implemented yet
I1013 07:52:10.930540   76994 out.go:176] 😄  minikube v1.23.2 on Darwin 11.6
😄  minikube v1.23.2 on Darwin 11.6
I1013 07:52:10.930790   76994 notify.go:169] Checking for updates...
I1013 07:52:10.931513   76994 driver.go:343] Setting default libvirt URI to qemu:///system
I1013 07:52:11.503365   76994 podman.go:121] podman version: 3.4.0
I1013 07:52:11.525626   76994 out.go:176] ✨  Using the podman (experimental) driver based on user configuration
✨  Using the podman (experimental) driver based on user configuration
I1013 07:52:11.525681   76994 start.go:278] selected driver: podman
I1013 07:52:11.525691   76994 start.go:751] validating driver "podman" against 
I1013 07:52:11.525709   76994 start.go:762] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:}
I1013 07:52:11.525869   76994 cli_runner.go:115] Run: podman system info --format json
I1013 07:52:11.730061   76994 info.go:285] podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc34.x86_64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:34} MemFree:8518746112 MemTotal:10426978304 OCIRuntime:{Name:crun Package:crun-1.2-1.fc34.x86_64 Path:/usr/bin/crun Version:crun version 1.2
commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:4 Eventlogger:journald Hostname:localhost Kernel:5.14.9-200.fc34.x86_64 Os:linux Rootless:false Uptime:25m 29.35s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:1} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I1013 07:52:11.730493   76994 start_flags.go:266] no existing cluster config was found, will generate one from the flags 
I1013 07:52:11.730613   76994 start_flags.go:727] Wait components to verify : map[apiserver:true system_pods:true]
I1013 07:52:11.730632   76994 cni.go:93] Creating CNI manager for ""
I1013 07:52:11.730641   76994 cni.go:160] "podman" driver + crio runtime found, recommending kindnet
I1013 07:52:11.730653   76994 start_flags.go:275] Found "CNI" CNI - setting NetworkPlugin=cni
I1013 07:52:11.730662   76994 start_flags.go:280] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:8192 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s}
I1013 07:52:11.752883   76994 out.go:176] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I1013 07:52:11.752938   76994 cache.go:118] Beginning downloading kic base image for podman with crio
I1013 07:52:11.774689   76994 out.go:176] 🚜  Pulling base image ...
🚜  Pulling base image ...
I1013 07:52:11.774828   76994 preload.go:130] Checking if preload exists for k8s version v1.22.2 and runtime crio
I1013 07:52:11.774853   76994 cache.go:146] Downloading gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c to local cache
I1013 07:52:11.774903   76994 preload.go:146] Found local preload: /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4
I1013 07:52:11.774919   76994 cache.go:57] Caching tarball of preloaded images
I1013 07:52:11.775139   76994 image.go:59] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory
I1013 07:52:11.775156   76994 preload.go:172] Found /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4 in cache, skipping download
I1013 07:52:11.775180   76994 image.go:62] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c in local cache directory, skipping pull
I1013 07:52:11.775191   76994 image.go:103] gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c exists in cache, skipping pull
I1013 07:52:11.775197   76994 cache.go:60] Finished verifying existence of preloaded tar for  v1.22.2 on crio
I1013 07:52:11.775245   76994 cache.go:149] successfully saved gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c as a tarball
I1013 07:52:11.775675   76994 profile.go:147] Saving config to /Users/kevin/.minikube/profiles/minikube/config.json ...
I1013 07:52:11.775750   76994 lock.go:35] WriteFile acquiring /Users/kevin/.minikube/profiles/minikube/config.json: {Name:mke28f4a3b281daafb742f57c9cfb025182acec4 Clock:{} Delay:500ms Timeout:1m0s Cancel:}
E1013 07:52:11.797534   76994 cache.go:201] Error downloading kic artifacts:  not yet implemented, see issue #8426
I1013 07:52:11.797560   76994 cache.go:206] Successfully downloaded all kic artifacts
I1013 07:52:11.797604   76994 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1013 07:52:11.797732   76994 start.go:317] acquired machines lock for "minikube" in 111.75µs
I1013 07:52:11.797777   76994 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:8192 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}
I1013 07:52:11.797932   76994 start.go:126] createHost starting for "" (driver="podman")
I1013 07:52:11.840885   76994 out.go:203] 🔥  Creating podman container (CPUs=2, Memory=8192MB) ...
🔥  Creating podman container (CPUs=2, Memory=8192MB) ...| I1013 07:52:11.841248   76994 start.go:160] libmachine.API.Create for "minikube" (driver="podman")
I1013 07:52:11.841298   76994 client.go:168] LocalClient.Create starting
I1013 07:52:11.841598   76994 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/ca.pem
I1013 07:52:11.841826   76994 main.go:130] libmachine: Decoding PEM data...
I1013 07:52:11.841874   76994 main.go:130] libmachine: Parsing certificate...
I1013 07:52:11.842004   76994 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/cert.pem
I1013 07:52:11.842100   76994 main.go:130] libmachine: Decoding PEM data...
I1013 07:52:11.842118   76994 main.go:130] libmachine: Parsing certificate...
I1013 07:52:11.842943   76994 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
- W1013 07:52:12.049202   76994 cli_runner.go:162] podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}" returned with exit code 125
I1013 07:52:12.049331   76994 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
I1013 07:52:12.049353   76994 cli_runner.go:115] Run: podman network inspect minikube
\ W1013 07:52:12.230293   76994 cli_runner.go:162] podman network inspect minikube returned with exit code 125
I1013 07:52:12.230378   76994 network_create.go:256] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: error inspecting object: no such network "minikube"
I1013 07:52:12.230472 76994 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: error inspecting object: no such network "minikube"

** /stderr **
I1013 07:52:12.230624 76994 cli_runner.go:115] Run: podman network inspect podman --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
/ I1013 07:52:12.405562 76994 network.go:288] reserving subnet 192.168.49.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[] amended:true}} dirty:map[192.168.49.0:0xc00069a488] misses:0}
I1013 07:52:12.405611 76994 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I1013 07:52:12.405628 76994 network_create.go:106] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
I1013 07:52:12.405714 76994 cli_runner.go:115] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true minikube
\ I1013 07:52:12.605463 76994 network_create.go:90] podman network minikube 192.168.49.0/24 created
I1013 07:52:12.605771 76994 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I1013 07:52:12.606006 76994 cli_runner.go:115] Run: podman ps -a --format {{.Names}}
/ I1013 07:52:12.786405 76994 cli_runner.go:115] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true

  • I1013 07:52:12.972345 76994 oci.go:102] Successfully created a podman volume minikube
    I1013 07:52:12.972516 76994 cli_runner.go:115] Run: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -d /var/lib
    | I1013 07:52:13.937545 76994 oci.go:106] Successfully prepared a podman volume minikube
    I1013 07:52:13.937631 76994 preload.go:130] Checking if preload exists for k8s version v1.22.2 and runtime crio
    I1013 07:52:13.937655 76994 kic.go:179] Starting extracting preloaded images to volume ...
    I1013 07:52:13.937733 76994 cli_runner.go:115] Run: podman info --format "'{{json .SecurityOptions}}'"
    I1013 07:52:13.937888 76994 cli_runner.go:115] Run: podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir
  • W1013 07:52:14.191131 76994 cli_runner.go:162] podman info --format "'{{json .SecurityOptions}}'" returned with exit code 125
    I1013 07:52:14.191363 76994 cli_runner.go:115] Run: podman run -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory-swap=8192mb --memory=8192mb --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081
    \ W1013 07:52:14.222284 76994 cli_runner.go:162] podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir returned with exit code 125
    I1013 07:52:14.222433 76994 kic.go:186] Unable to extract preloaded tarball to volume: podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir: exit status 125
    stdout:

stderr:
Error: statfs /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4: no such file or directory
/ I1013 07:52:14.893073 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Running}}
\ I1013 07:52:15.131239 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}

  • I1013 07:52:15.360040 76994 cli_runner.go:115] Run: podman exec minikube stat /var/lib/dpkg/alternatives/iptables
    / I1013 07:52:15.738341 76994 oci.go:281] the created container "minikube" has a running status.
    I1013 07:52:15.738393 76994 kic.go:210] Creating ssh key for kic: /Users/kevin/.minikube/machines/minikube/id_rsa...
    / I1013 07:52:16.082709 76994 vm_assets.go:109] NewFileAsset: /Users/kevin/.minikube/machines/minikube/id_rsa.pub -> /home/docker/.ssh/authorized_keys
    I1013 07:52:16.082919 76994 kic_runner.go:187] podman (temp): /Users/kevin/.minikube/machines/minikube/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
    I1013 07:52:16.088751 76994 kic_runner.go:272] Run: /usr/local/bin/podman exec -i minikube tee /home/docker/.ssh/authorized_keys
    | I1013 07:52:16.418802 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}

  • I1013 07:52:16.625862 76994 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
    I1013 07:52:16.625893 76994 kic_runner.go:114] Args: [podman exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys]
    / I1013 07:52:16.930099 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
    \ I1013 07:52:17.136585 76994 machine.go:88] provisioning docker machine ...
    I1013 07:52:17.136659 76994 ubuntu.go:169] provisioning hostname "minikube"
    I1013 07:52:17.136798 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
    / I1013 07:52:17.403839 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    \ I1013 07:52:17.607243 76994 main.go:130] libmachine: Using SSH client type: native
    I1013 07:52:17.607528 76994 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x43a00c0] 0x43a31a0 [] 0s} 127.0.0.1 37393 }
    I1013 07:52:17.607551 76994 main.go:130] libmachine: About to run SSH command:
    sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
    I1013 07:52:17.608808 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53930->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:52:20.612819 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53935->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:52:23.614212 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53936->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:52:26.616993 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53941->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:52:29.618918 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53946->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:52:32.623980 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53947->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:52:35.627841 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53953->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:52:38.631676 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53954->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:52:41.636032 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53959->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:52:44.639101 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53972->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:52:47.643942 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53973->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:52:50.645938 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53979->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:52:53.649452 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53980->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:52:56.655954 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53985->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:52:59.657670 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53990->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:02.663271 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53991->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:53:05.665897 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53996->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:53:08.671044 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53997->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:11.672465 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54002->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:53:14.677435 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54007->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:53:17.681422 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54008->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:53:20.686661 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54013->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:53:23.691269 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54014->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:53:26.696171 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54019->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:53:29.702504 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54024->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:32.706067 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54025->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:53:35.709976 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54030->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:53:38.716547 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54031->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:41.719896 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54036->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:53:44.726553 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54041->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:53:47.731382 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54042->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:50.736730 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54047->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:53:53.741814 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54048->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:53:56.745454 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54053->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:53:59.751532 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54059->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:02.752989 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54060->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:54:05.758857 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54065->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:54:08.763439 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54066->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:11.765377 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54071->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:54:14.768568 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54076->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:54:17.773899 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54077->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:54:20.778229 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54082->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:54:23.780258 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54083->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:54:26.785650 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54088->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:54:29.792445 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54093->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:32.798310 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54094->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:54:35.801051 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54099->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:54:38.806833 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54101->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:41.810329 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54106->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:54:44.813296 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54112->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:54:47.815849 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54113->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:50.817726 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54118->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:54:53.844605 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54119->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:54:56.850850 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54124->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:54:59.856502 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54130->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:55:02.858637 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54131->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:05.860636 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54136->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:55:08.861885 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54137->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:55:11.866718 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54142->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:14.873262 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54147->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:55:17.942673 76994 main.go:130] libmachine: SSH cmd err, output: :
    I1013 07:55:17.942805 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
    \ I1013 07:55:18.846310 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    / I1013 07:55:19.054928 76994 main.go:130] libmachine: Using SSH client type: native
    I1013 07:55:19.055264 76994 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x43a00c0] 0x43a31a0 [] 0s} 127.0.0.1 37393 }
    I1013 07:55:19.055295 76994 main.go:130] libmachine: About to run SSH command:

      if ! grep -xq '.*\sminikube' /etc/hosts; then
      	if grep -xq '127.0.1.1\s.*' /etc/hosts; then
      		sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
      	else 
      		echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts; 
      	fi
      fi
    

I1013 07:55:19.057113 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54156->127.0.0.1:37393: read: connection reset by peer

  • I1013 07:55:22.062866 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54161->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:55:25.066282 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54166->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:28.072047 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54167->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:55:31.074534 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54172->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:55:34.080138 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54173->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:37.081772 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54178->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:55:40.085275 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54183->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:55:43.091239 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54184->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:55:46.093356 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54189->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:49.098640 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54190->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:55:52.102811 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54195->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:55:55.107136 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54200->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:55:58.108918 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54201->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:01.114076 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54207->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:56:04.120856 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54208->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:56:07.125588 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54213->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:10.131474 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54218->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:56:13.137527 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54219->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:56:16.142286 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54224->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:19.143945 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54225->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:56:22.147921 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54230->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:56:25.154064 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54235->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:28.159261 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54236->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:56:31.163966 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54241->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:56:34.165666 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54242->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:56:37.171411 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54247->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:40.174434 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54252->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:56:43.179203 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54254->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:56:46.185198 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54259->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:49.190393 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54260->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:56:52.194497 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54265->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:56:55.200610 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54270->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:56:58.203010 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54271->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:57:01.206842 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54276->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:57:04.208445 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54277->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:07.210746 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54282->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:57:10.212375 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54287->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:57:13.214405 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54288->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:16.220207 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54293->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:57:19.224616 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54294->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:57:22.227022 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54299->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:25.230463 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54304->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:57:28.232063 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54305->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:57:31.236347 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54311->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:34.238324 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54312->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:57:37.243743 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54317->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:57:40.247898 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54323->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:57:43.254431 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54325->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:46.259881 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54330->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:57:49.262019 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54332->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:57:52.268071 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54341->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:57:55.269774 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54346->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:57:58.272566 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54347->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:58:01.277536 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54353->127.0.0.1:37393: read: connection reset by peer
    | I1013 07:58:04.283400 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54355->127.0.0.1:37393: read: connection reset by peer
    / I1013 07:58:07.287830 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54360->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:58:10.292802 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54365->127.0.0.1:37393: read: connection reset by peer
  • I1013 07:58:11.912143 76994 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
    I1013 07:58:11.912251 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
    / I1013 07:58:12.636578 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    | I1013 07:58:12.911821 76994 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:37393 SSHKeyPath:/Users/kevin/.minikube/machines/minikube/id_rsa Username:docker}
    W1013 07:58:12.913368 76994 sshutil.go:64] dial failure (will retry): ssh: handshake failed: read tcp 127.0.0.1:54372->127.0.0.1:37393: read: connection reset by peer
    I1013 07:58:12.913396 76994 retry.go:31] will retry after 276.165072ms: ssh: handshake failed: read tcp 127.0.0.1:54372->127.0.0.1:37393: read: connection reset by peer
  • W1013 07:58:13.195332 76994 sshutil.go:64] dial failure (will retry): ssh: handshake failed: read tcp 127.0.0.1:54373->127.0.0.1:37393: read: connection reset by peer
    I1013 07:58:13.195354 76994 retry.go:31] will retry after 540.190908ms: ssh: handshake failed: read tcp 127.0.0.1:54373->127.0.0.1:37393: read: connection reset by peer
    \ I1013 07:58:13.297485 76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:54374->127.0.0.1:37393: read: connection reset by peer
    | W1013 07:58:13.739876 76994 sshutil.go:64] dial failure (will retry): ssh: handshake failed: read tcp 127.0.0.1:54375->127.0.0.1:37393: read: connection reset by peer
    I1013 07:58:13.739902 76994 retry.go:31] will retry after 655.06503ms: ssh: handshake failed: read tcp 127.0.0.1:54375->127.0.0.1:37393: read: connection reset by peer
  • W1013 07:58:14.396779 76994 sshutil.go:64] dial failure (will retry): ssh: handshake failed: read tcp 127.0.0.1:54376->127.0.0.1:37393: read: connection reset by peer
    W1013 07:58:14.396897 76994 start.go:257] error running df -h /var: NewSession: new client: new client: ssh: handshake failed: read tcp 127.0.0.1:54376->127.0.0.1:37393: read: connection reset by peer
    W1013 07:58:14.396922 76994 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: ssh: handshake failed: read tcp 127.0.0.1:54376->127.0.0.1:37393: read: connection reset by peer
    I1013 07:58:14.396931 76994 start.go:129] duration metric: createHost completed in 6m2.532746389s
    I1013 07:58:14.396944 76994 start.go:80] releasing machines lock for "minikube", held for 6m2.53296203s
    W1013 07:58:14.397018 76994 start.go:522] error starting host: creating host: create host timed out in 360.000000 seconds
    I1013 07:58:14.397745 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
    | I1013 07:58:14.608649 76994 stop.go:39] StopHost: minikube
    W1013 07:58:14.609145 76994 register.go:129] "Stopping" was not found within the registered steps for "Initial Minikube Setup": [Initial Minikube Setup Selecting Driver Downloading Artifacts Starting Node Updating Driver Pulling Base Image Running on Localhost Local OS Release Creating Container Creating VM Running Remotely Preparing Kubernetes Generating certificates Booting control plane Configuring RBAC rules Configuring CNI Configuring Localhost Environment Verifying Kubernetes Enabling Addons Done]
    / I1013 07:58:14.652692 76994 out.go:176] ✋ Stopping node "minikube" ...

✋ Stopping node "minikube" ...
I1013 07:58:14.652959 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
W1013 07:58:14.839279 76994 register.go:129] "PowerOff" was not found within the registered steps for "Initial Minikube Setup": [Initial Minikube Setup Selecting Driver Downloading Artifacts Starting Node Updating Driver Pulling Base Image Running on Localhost Local OS Release Creating Container Creating VM Running Remotely Preparing Kubernetes Generating certificates Booting control plane Configuring RBAC rules Configuring CNI Configuring Localhost Environment Verifying Kubernetes Enabling Addons Done]
I1013 07:58:14.861296 76994 out.go:176] 🛑 Powering off "minikube" via SSH ...
🛑 Powering off "minikube" via SSH ...
I1013 07:58:14.861436 76994 cli_runner.go:115] Run: podman exec --privileged -t minikube /bin/bash -c "sudo init 0"
I1013 07:58:16.185931 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:58:16.302602 76994 main.go:130] libmachine: Error dialing TCP: dial tcp 127.0.0.1:37393: connect: connection refused
I1013 07:58:16.393693 76994 oci.go:643] container minikube status is Stopped
I1013 07:58:16.393722 76994 oci.go:655] Successfully shutdown container minikube
I1013 07:58:16.393728 76994 stop.go:88] shutdown container: err=
I1013 07:58:16.393748 76994 main.go:130] libmachine: Stopping "minikube"...
I1013 07:58:16.393897 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:58:16.582799 76994 stop.go:59] stop err: Machine "minikube" is already stopped.
I1013 07:58:16.582866 76994 stop.go:62] host is already stopped
W1013 07:58:17.585641 76994 register.go:129] "PowerOff" was not found within the registered steps for "Initial Minikube Setup": [Initial Minikube Setup Selecting Driver Downloading Artifacts Starting Node Updating Driver Pulling Base Image Running on Localhost Local OS Release Creating Container Creating VM Running Remotely Preparing Kubernetes Generating certificates Booting control plane Configuring RBAC rules Configuring CNI Configuring Localhost Environment Verifying Kubernetes Enabling Addons Done]
I1013 07:58:17.607846 76994 out.go:176] 🔥 Deleting "minikube" in podman ...
🔥 Deleting "minikube" in podman ...
I1013 07:58:17.608215 76994 cli_runner.go:115] Run: podman container inspect -f {{.Id}} minikube
I1013 07:58:17.809052 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:58:18.024298 76994 cli_runner.go:115] Run: podman exec --privileged -t minikube /bin/bash -c "sudo init 0"
W1013 07:58:18.203827 76994 cli_runner.go:162] podman exec --privileged -t minikube /bin/bash -c "sudo init 0" returned with exit code 125
I1013 07:58:18.203871 76994 oci.go:635] error shutdown minikube: podman exec --privileged -t minikube /bin/bash -c "sudo init 0": exit status 125
stdout:

stderr:
Error: can only create exec sessions on running containers: container state improper
I1013 07:58:19.207289 76994 cli_runner.go:115] Run: podman container inspect minikube --format={{.State.Status}}
I1013 07:58:19.303844 76994 main.go:130] libmachine: SSH cmd err, output: :
I1013 07:58:19.303886 76994 ubuntu.go:175] set auth options {CertDir:/Users/kevin/.minikube CaCertPath:/Users/kevin/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/kevin/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/kevin/.minikube/machines/server.pem ServerKeyPath:/Users/kevin/.minikube/machines/server-key.pem ClientKeyPath:/Users/kevin/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/kevin/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/kevin/.minikube}
I1013 07:58:19.303946 76994 ubuntu.go:177] setting up certificates
I1013 07:58:19.303969 76994 provision.go:83] configureAuth start
I1013 07:58:19.304065 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
I1013 07:58:19.415929 76994 oci.go:643] container minikube status is Stopped
I1013 07:58:19.415982 76994 oci.go:655] Successfully shutdown container minikube
I1013 07:58:19.416082 76994 cli_runner.go:115] Run: podman rm -f -v minikube
I1013 07:58:19.520600 76994 cli_runner.go:115] Run: podman container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I1013 07:58:19.657710 76994 cli_runner.go:115] Run: podman container inspect -f {{.Id}} minikube
W1013 07:58:19.730903 76994 cli_runner.go:162] podman container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube returned with exit code 125
I1013 07:58:19.730990 76994 provision.go:138] copyHostCerts
I1013 07:58:19.731037 76994 vm_assets.go:109] NewFileAsset: /Users/kevin/.minikube/certs/cert.pem -> /Users/kevin/.minikube/cert.pem
I1013 07:58:19.731164 76994 exec_runner.go:144] found /Users/kevin/.minikube/cert.pem, removing ...
I1013 07:58:19.731175 76994 exec_runner.go:207] rm: /Users/kevin/.minikube/cert.pem
I1013 07:58:19.732404 76994 exec_runner.go:151] cp: /Users/kevin/.minikube/certs/cert.pem --> /Users/kevin/.minikube/cert.pem (1119 bytes)
I1013 07:58:19.732699 76994 vm_assets.go:109] NewFileAsset: /Users/kevin/.minikube/certs/key.pem -> /Users/kevin/.minikube/key.pem
I1013 07:58:19.732909 76994 exec_runner.go:144] found /Users/kevin/.minikube/key.pem, removing ...
I1013 07:58:19.732920 76994 exec_runner.go:207] rm: /Users/kevin/.minikube/key.pem
I1013 07:58:19.733632 76994 exec_runner.go:151] cp: /Users/kevin/.minikube/certs/key.pem --> /Users/kevin/.minikube/key.pem (1679 bytes)
I1013 07:58:19.734304 76994 vm_assets.go:109] NewFileAsset: /Users/kevin/.minikube/certs/ca.pem -> /Users/kevin/.minikube/ca.pem
I1013 07:58:19.734458 76994 exec_runner.go:144] found /Users/kevin/.minikube/ca.pem, removing ...
I1013 07:58:19.734474 76994 exec_runner.go:207] rm: /Users/kevin/.minikube/ca.pem
I1013 07:58:19.735096 76994 exec_runner.go:151] cp: /Users/kevin/.minikube/certs/ca.pem --> /Users/kevin/.minikube/ca.pem (1074 bytes)
I1013 07:58:19.735867 76994 provision.go:112] generating server cert: /Users/kevin/.minikube/machines/server.pem ca-key=/Users/kevin/.minikube/certs/ca.pem private-key=/Users/kevin/.minikube/certs/ca-key.pem org=kevin.minikube san=[127.0.0.1 127.0.0.1 localhost 127.0.0.1 minikube minikube]
W1013 07:58:19.873734 76994 cli_runner.go:162] podman container inspect -f {{.Id}} minikube returned with exit code 125
I1013 07:58:19.873872 76994 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
I1013 07:58:19.908479 76994 provision.go:172] copyRemoteCerts
I1013 07:58:19.908658 76994 ssh_runner.go:152] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I1013 07:58:19.908787 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
I1013 07:58:20.090413 76994 cli_runner.go:115] Run: podman network rm minikube
I1013 07:58:20.234955 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1013 07:58:20.365591 76994 out.go:241] 🤦 StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
🤦 StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
I1013 07:58:20.365633 76994 start.go:537] Will try again in 5 seconds ...
W1013 07:58:20.482732 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1013 07:58:20.482796 76994 retry.go:31] will retry after 296.705768ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:20.783263 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
I1013 07:58:21.021570 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1013 07:58:21.260802 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1013 07:58:21.260911 76994 retry.go:31] will retry after 212.113882ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:21.476156 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
I1013 07:58:21.711870 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1013 07:58:21.886111 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1013 07:58:21.886187 76994 provision.go:86] duration metric: configureAuth took 2.582188879s
W1013 07:58:21.886193 76994 ubuntu.go:180] configureAuth failed: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:21.886211 76994 retry.go:31] will retry after 59.697µs: Temporary Error: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:21.886323 76994 provision.go:83] configureAuth start
I1013 07:58:21.886440 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:22.073910 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:22.073965 76994 provision.go:86] duration metric: configureAuth took 187.625037ms
W1013 07:58:22.073970 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.074015 76994 retry.go:31] will retry after 120.137µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.074197 76994 provision.go:83] configureAuth start
I1013 07:58:22.074320 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:22.253059 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:22.253121 76994 provision.go:86] duration metric: configureAuth took 178.903539ms
W1013 07:58:22.253131 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.253152 76994 retry.go:31] will retry after 228.423µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.253473 76994 provision.go:83] configureAuth start
I1013 07:58:22.253571 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:22.426012 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:22.426069 76994 provision.go:86] duration metric: configureAuth took 172.583638ms
W1013 07:58:22.426077 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.426113 76994 retry.go:31] will retry after 443.354µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.426723 76994 provision.go:83] configureAuth start
I1013 07:58:22.426825 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:22.597782 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:22.597831 76994 provision.go:86] duration metric: configureAuth took 171.08672ms
W1013 07:58:22.597836 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.597865 76994 retry.go:31] will retry after 361.596µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.598368 76994 provision.go:83] configureAuth start
I1013 07:58:22.598465 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:22.813098 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:22.813207 76994 provision.go:86] duration metric: configureAuth took 214.815421ms
W1013 07:58:22.813222 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.813285 76994 retry.go:31] will retry after 668.749µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:22.814109 76994 provision.go:83] configureAuth start
I1013 07:58:22.814378 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:23.031300 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:23.031349 76994 provision.go:86] duration metric: configureAuth took 217.172129ms
W1013 07:58:23.031356 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.031379 76994 retry.go:31] will retry after 931.819µs: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.032421 76994 provision.go:83] configureAuth start
I1013 07:58:23.032541 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:23.233156 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:23.233216 76994 provision.go:86] duration metric: configureAuth took 200.775642ms
W1013 07:58:23.233237 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.233271 76994 retry.go:31] will retry after 1.655438ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.235068 76994 provision.go:83] configureAuth start
I1013 07:58:23.235211 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:23.430918 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:23.430983 76994 provision.go:86] duration metric: configureAuth took 195.890833ms
W1013 07:58:23.430991 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.431019 76994 retry.go:31] will retry after 2.006829ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.433160 76994 provision.go:83] configureAuth start
I1013 07:58:23.433350 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:23.633113 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:23.633187 76994 provision.go:86] duration metric: configureAuth took 199.976012ms
W1013 07:58:23.633195 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.633220 76994 retry.go:31] will retry after 3.048947ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.637035 76994 provision.go:83] configureAuth start
I1013 07:58:23.637189 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:23.822744 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:23.822790 76994 provision.go:86] duration metric: configureAuth took 185.721786ms
W1013 07:58:23.822797 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.822817 76994 retry.go:31] will retry after 6.799191ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:23.829952 76994 provision.go:83] configureAuth start
I1013 07:58:23.830088 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:24.007531 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:24.007584 76994 provision.go:86] duration metric: configureAuth took 177.605036ms
W1013 07:58:24.007590 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.007610 76994 retry.go:31] will retry after 6.215303ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.014385 76994 provision.go:83] configureAuth start
I1013 07:58:24.014713 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:24.250792 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:24.250842 76994 provision.go:86] duration metric: configureAuth took 236.426759ms
W1013 07:58:24.250848 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.250871 76994 retry.go:31] will retry after 9.123584ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.261280 76994 provision.go:83] configureAuth start
I1013 07:58:24.261423 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:24.445320 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:24.445365 76994 provision.go:86] duration metric: configureAuth took 184.05758ms
W1013 07:58:24.445371 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.445402 76994 retry.go:31] will retry after 16.754222ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.464530 76994 provision.go:83] configureAuth start
I1013 07:58:24.464641 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:24.700909 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:24.701086 76994 provision.go:86] duration metric: configureAuth took 236.518936ms
W1013 07:58:24.701095 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.701149 76994 retry.go:31] will retry after 31.256057ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.735817 76994 provision.go:83] configureAuth start
I1013 07:58:24.736135 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:24.928695 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:24.928753 76994 provision.go:86] duration metric: configureAuth took 192.896751ms
W1013 07:58:24.928759 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.928782 76994 retry.go:31] will retry after 59.662612ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:24.991869 76994 provision.go:83] configureAuth start
I1013 07:58:24.992121 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W1013 07:58:25.265133 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:25.265236 76994 provision.go:86] duration metric: configureAuth took 273.334789ms
W1013 07:58:25.265246 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:25.265281 76994 retry.go:31] will retry after 52.094933ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:25.322410 76994 provision.go:83] configureAuth start
I1013 07:58:25.322610 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
I1013 07:58:25.369495 76994 start.go:313] acquiring machines lock for minikube: {Name:mkd8e9571f6c11de6ea65bb56cf73cade4c784c0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I1013 07:58:25.369728 76994 start.go:317] acquired machines lock for "minikube" in 166.804µs
I1013 07:58:25.369770 76994 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081@sha256:4780f1897569d2bf77aafb3d133a08d42b4fe61127f06fcfc90c2c5d902d893c Memory:8192 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.22.2 ControlPlane:true Worker:true}
I1013 07:58:25.369990 76994 start.go:126] createHost starting for "" (driver="podman")
I1013 07:58:25.414905 76994 out.go:203] 🔥 Creating podman container (CPUs=2, Memory=8192MB) ...
🔥 Creating podman container (CPUs=2, Memory=8192MB) ...| I1013 07:58:25.415156 76994 start.go:160] libmachine.API.Create for "minikube" (driver="podman")
I1013 07:58:25.415190 76994 client.go:168] LocalClient.Create starting
I1013 07:58:25.415376 76994 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/ca.pem
I1013 07:58:25.415528 76994 main.go:130] libmachine: Decoding PEM data...
I1013 07:58:25.415562 76994 main.go:130] libmachine: Parsing certificate...
I1013 07:58:25.415648 76994 main.go:130] libmachine: Reading certificate data from /Users/kevin/.minikube/certs/cert.pem
I1013 07:58:25.415723 76994 main.go:130] libmachine: Decoding PEM data...
I1013 07:58:25.415744 76994 main.go:130] libmachine: Parsing certificate...
I1013 07:58:25.416282 76994 cli_runner.go:115] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
/ W1013 07:58:25.540497 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I1013 07:58:25.540542 76994 provision.go:86] duration metric: configureAuth took 218.101852ms
W1013 07:58:25.540548 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:25.540580 76994 retry.go:31] will retry after 78.533383ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

  • I1013 07:58:25.622027 76994 provision.go:83] configureAuth start
    I1013 07:58:25.622250 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    W1013 07:58:25.657060 76994 cli_runner.go:162] podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}" returned with exit code 125
    I1013 07:58:25.657289 76994 network_create.go:253] running [podman network inspect minikube] to gather additional debugging logs...
    I1013 07:58:25.657342 76994 cli_runner.go:115] Run: podman network inspect minikube
    | W1013 07:58:25.883780 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:25.883859 76994 provision.go:86] duration metric: configureAuth took 261.756172ms
    W1013 07:58:25.883887 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:25.883915 76994 retry.go:31] will retry after 185.116578ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
/ W1013 07:58:25.935750 76994 cli_runner.go:162] podman network inspect minikube returned with exit code 125
I1013 07:58:25.935805 76994 network_create.go:256] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: error inspecting object: no such network "minikube"
I1013 07:58:25.935844 76994 network_create.go:258] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: error inspecting object: no such network "minikube"

** /stderr **
I1013 07:58:25.936024 76994 cli_runner.go:115] Run: podman network inspect podman --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"

  • I1013 07:58:26.073926 76994 provision.go:83] configureAuth start
    I1013 07:58:26.074137 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    \ I1013 07:58:26.163733 76994 network.go:284] reusing subnet 192.168.49.0 that has expired reservation: &{mu:{state:0 sema:0} read:{v:{m:map[192.168.49.0:0xc00069a488] amended:false}} dirty:map[] misses:0}
    I1013 07:58:26.163768 76994 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
    I1013 07:58:26.163781 76994 network_create.go:106] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
    I1013 07:58:26.163874 76994 cli_runner.go:115] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true minikube
    / W1013 07:58:26.355463 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:26.355514 76994 provision.go:86] duration metric: configureAuth took 281.555861ms
    W1013 07:58:26.355520 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:26.355554 76994 retry.go:31] will retry after 156.637769ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:26.439979 76994 network_create.go:90] podman network minikube 192.168.49.0/24 created
I1013 07:58:26.440008 76994 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I1013 07:58:26.440174 76994 cli_runner.go:115] Run: podman ps -a --format {{.Names}}

  • I1013 07:58:26.517101 76994 provision.go:83] configureAuth start
    I1013 07:58:26.517333 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    | I1013 07:58:26.672400 76994 cli_runner.go:115] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
    / W1013 07:58:26.825241 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:26.825319 76994 provision.go:86] duration metric: configureAuth took 308.128211ms
    W1013 07:58:26.825334 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:26.825388 76994 retry.go:31] will retry after 454.008516ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

  • W1013 07:58:26.949460 76994 cli_runner.go:162] podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true returned with exit code 125
    I1013 07:58:26.949528 76994 client.go:171] LocalClient.Create took 1.534314296s
  • I1013 07:58:27.280129 76994 provision.go:83] configureAuth start
    I1013 07:58:27.280349 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    | W1013 07:58:27.501722 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:27.501774 76994 provision.go:86] duration metric: configureAuth took 221.617615ms
    W1013 07:58:27.501781 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:27.501819 76994 retry.go:31] will retry after 596.909205ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

  • I1013 07:58:28.099308 76994 provision.go:83] configureAuth start
    I1013 07:58:28.099516 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    \ W1013 07:58:28.297268 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:28.297344 76994 provision.go:86] duration metric: configureAuth took 197.978598ms
    W1013 07:58:28.297352 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:28.297377 76994 retry.go:31] will retry after 766.003983ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

  • I1013 07:58:28.953582 76994 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
    I1013 07:58:28.953712 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
    \ I1013 07:58:29.067963 76994 provision.go:83] configureAuth start
    I1013 07:58:29.068161 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
    / I1013 07:58:29.302653 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    W1013 07:58:29.336593 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:29.336645 76994 provision.go:86] duration metric: configureAuth took 268.637017ms
    W1013 07:58:29.336652 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:29.336687 76994 retry.go:31] will retry after 592.900331ms: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
\ W1013 07:58:29.524416 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1013 07:58:29.524576 76994 retry.go:31] will retry after 164.582069ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
/ I1013 07:58:29.691988 76994 cli_runner.go:115] Run: podman version --format {{.Version}}
\ I1013 07:58:29.934444 76994 provision.go:83] configureAuth start
I1013 07:58:29.934617 76994 cli_runner.go:115] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
| I1013 07:58:29.995813 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube

  • W1013 07:58:30.174594 76994 cli_runner.go:162] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
    I1013 07:58:30.174635 76994 provision.go:86] duration metric: configureAuth took 240.161182ms
    W1013 07:58:30.174641 76994 ubuntu.go:180] configureAuth failed: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:30.174687 76994 retry.go:31] will retry after 1.8639611s: Temporary Error: error getting ip during provisioning: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W1013 07:58:30.243832 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I1013 07:58:30.243948 76994 retry.go:31] will retry after 553.215608ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
| I1013 07:58:30.798706 76994 cli_runner.go:115] Run: podman version --format {{.Version}}

  • I1013 07:58:31.052829 76994 cli_runner.go:115] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
    | W1013 07:58:31.244758 76994 cli_runner.go:162] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
    W1013 07:58:31.244947 76994 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
    stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W1013 07:58:31.244989 76994 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I1013 07:58:31.245008 76994 start.go:129] duration metric: createHost completed in 5.874936866s
I1013 07:58:31.245020 76994 start.go:80] releasing machines lock for "minikube", held for 5.875222124s
W1013 07:58:31.245339 76994 out.go:241] 😿 Failed to start podman container. Running "minikube delete" may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

😿 Failed to start podman container. Running "minikube delete" may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

I1013 07:58:31.289138 76994 out.go:176]

W1013 07:58:31.289363 76994 out.go:241] ❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

W1013 07:58:31.289389 76994 out.go:241]

W1013 07:58:31.290205 76994 out.go:241] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I1013 07:58:31.354161 76994 out.go:176]

@afbjorklund
Copy link
Collaborator

afbjorklund commented Oct 13, 2021

There needs to be a warning for that, similar to the one for docker. Just that running podman-remote is quite new.

Eventually, minikube and kubernetes will support rootless containers as well. But until it works, it should warn...

@afbjorklund
Copy link
Collaborator

Seems to be the volume (and possibly the network) from a previous run that are "in the way" ? Maybe remove manually.

Error: volume with name minikube already exists: volume already exists

@kgibm
Copy link
Contributor Author

kgibm commented Oct 13, 2021

Seems to be the volume (and possibly the network) from a previous run that are "in the way" ? Maybe remove manually.
Error: volume with name minikube already exists: volume already exists

I did a minikube delete before. Does that not clean it up?

It seems to me the volume error is because it tries again after failing the first time. The first error seems to be this though I'm not sure if it's benign:

    I1013 07:52:14.222433   76994 kic.go:186] Unable to extract preloaded tarball to volume: podman run --rm --entrypoint /usr/bin/tar -v /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.27-1633027942-12081 -I lz4 -xf /preloaded.tar -C /extractDir: exit status 125
    stdout:


stderr:
Error: statfs /Users/kevin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v13-v1.22.2-cri-o-overlay-amd64.tar.lz4: no such file or directory

Then it seems the next major error is:

I1013 07:52:17.607551   76994 main.go:130] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
I1013 07:52:17.608808   76994 main.go:130] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53930->127.0.0.1:37393: read: connection reset by peer

@toanju
Copy link

toanju commented Oct 27, 2021

Hi,

I started with minikube and ran into the same issue and switched to kind. Though I ran there into the same issue but I was able to troubleshoot it. I guess this could help here as well:

kubernetes-sigs/kind#2445 (comment)

hth

@ifplusor
Copy link
Contributor

ifplusor commented Nov 2, 2021

I also encountered the same issue, and stderr output below:

tcpproxy: for incoming conn 127.0.0.1:64275, error dialing "192.168.127.2:33131": connect tcp 192.168.127.2:33131: connection was refused

ss -nplt in podman machine indicate port 33131 is listened by conmon.

@ifplusor
Copy link
Contributor

ifplusor commented Nov 3, 2021

Port mapping bind 127.0.0.1, but vm of podman refuse connection from host to connect 127.0.0.1. I find an issue in podman project: containers/podman#11528

After change --publish=127.0.0.1::22 to --publish=22, ssh/handshake is ok, but another issue about kubeconfig present.

❌  Exiting due to GUEST_START: Failed to setup kubeconfig: RoutableHostIPFromInside is currently only implemented for linux

@kgibm
Copy link
Contributor Author

kgibm commented Nov 15, 2021

@ifplusor

After change --publish=127.0.0.1::22 to --publish=22, ssh/handshake is ok

Where do you change this?

@ifplusor
Copy link
Contributor

ifplusor commented Nov 16, 2021

@kgibm See this line

publish := fmt.Sprintf("--publish=%s::%d", pm.ListenAddress, pm.ContainerPort)

I change it to

publish := fmt.Sprintf("--publish=%d", pm.ContainerPort)

@klaases
Copy link
Contributor

klaases commented Dec 22, 2021

Hi @kgibm, was the comment by @ifplusor able to solve the issue?

@kgibm
Copy link
Contributor Author

kgibm commented Dec 22, 2021

@klaases

was the comment by @ifplusor able to solve the issue?

Yes. Is this a bug? Should the publish variable be evaluated differently based on the host operating system?

Also, I see the same error, "Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux" although it doesn't seem to be fatal.

Steps:

$ git clone https://github.com/kubernetes/minikube.git
[...]
$ cd minikube
$ sed -i.bak 's/fmt.Sprintf("--publish=%s::%d", pm.ListenAddress, pm.ContainerPort)/fmt.Sprintf("--publish=%d", pm.ContainerPort)/g' pkg/drivers/kic/oci/oci.go && rm pkg/drivers/kic/oci/oci.go.bak
$ make darwin
[...]
$ podman machine init --cpus 4 --memory 10240 --disk-size 50
[...]
$ podman machine start
[...]
$ podman machine ls
NAME                    VM TYPE     CREATED       LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default  qemu        27 hours ago  Currently running  4           10.74GB     53.69GB
$ podman system connection default podman-machine-default-root
[...]
$ podman system connection list
Name                          Identity                                  URI
podman-machine-default        /Users/kevin/.ssh/podman-machine-default  ssh://core@localhost:59823/run/user/1000/podman/podman.sock
podman-machine-default-root*  /Users/kevin/.ssh/podman-machine-default  ssh://root@localhost:59823/run/podman/podman.sock
$ ./out/minikube-darwin-amd64 delete
[...]
$ ./out/minikube-darwin-amd64 start --driver=podman --container-runtime=cri-o
😄  minikube v1.24.0 on Darwin 12.0.1
✨  Using the podman (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
E1222 12:13:39.958908   19600 cache.go:201] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=4000MB) ...
🎁  Preparing Kubernetes v1.23.1 on CRI-O 1.22.1 ...
E1222 12:13:50.705786   19600 start.go:124] Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl get node
NAME       STATUS   ROLES                  AGE   VERSION
minikube   Ready    control-plane,master   47s   v1.23.1

@kgibm
Copy link
Contributor Author

kgibm commented Dec 22, 2021

Also, stopping resulted in errors:

$ ./out/minikube-darwin-amd64 stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
E1222 12:25:05.277205   19735 kic.go:463] unable to kill containers : crictl: sudo /usr/bin/crictl rm aa8d15da600b9885deaba147fd47e49884fa46145747a23782e90cfbccd3c8a9 4cf56655bb5fd94b5b3dddfe5d5629d06f0445605a5b3c4ba66c2cddc7ac341a ffdc8eb38b0b2acfbc9a3ee9b4b328c853a08194414667756daf1e3ebb1fc0f7 f5b6aaa9b2ecce1eab578866828390f73e110a7913eb3dee996cd8d0a0f4b92a 539a5a26ab579b898a57bfe138bb58a3a363ab04eda8c3fcb0c43d27d02ea605 b10a65225ee051ff7a18d86ac6fc220beba6028a34776582cf5ed6ff4587188d 36db9c720882ceb6bef54859fccac5b6c7d1388c24fdb40fe6d70d151d86e7c3 2c8aedd1e284a427a7dc92e025a8d41d8878f347258710ca0f76674136aef9b7 a50a1fb645839877b3c562faf714043857815faa2b14008e34da8c3d33631090: exit status 143
stdout:

stderr:
sudo: unable to resolve host minikube: Temporary failure in name resolution
🛑  1 node stopped.

@egzonzeneli
Copy link

egzonzeneli commented Jan 6, 2022

Thank @kgibm,

I can confirm that, with that change I`ve been able to run minikube on Arm based Mac.

@kadursandy
Copy link

I get the same error

@kaliberr44
Copy link

I also have the same error but @kgibm change didn't help to resolve it.

@cdbattags
Copy link

Hi everyone, I'm getting the same ssh error.

I'll try with building minikube myself with that line change suggested hopefully some point this week.

Any more progress on this issue to know about?

@k2patel
Copy link

k2patel commented Feb 9, 2022

I had the same issue, suggestion from @kgibm works like charm. During multiple rebuild it is clear that important step not to miss is podman system connection default podman-machine-default-root.

@afbjorklund
Copy link
Collaborator

During multiple rebuild it is clear that important step not to miss is podman system connection default podman-machine-default-root.

This is being addressed:

If you want to run as rootless.

@kgibm
Copy link
Contributor Author

kgibm commented Feb 9, 2022

I had the issue even running as root. It was only resolved after the patch:

$ sed -i.bak 's/fmt.Sprintf("--publish=%s::%d", pm.ListenAddress, pm.ContainerPort)/fmt.Sprintf("--publish=%d", pm.ContainerPort)/g' pkg/drivers/kic/oci/oci.go && rm pkg/drivers/kic/oci/oci.go.bak

@MarcoR83
Copy link

can confirm the patch is necessary to make it work. patched and build the current stable. v1.25.1. with this minikube startup works fine. hope to see this fixed in a new release in the future :)

@spowelljr spowelljr added the long-term-support Long-term support issues that can't be fixed in code label May 4, 2022
@dwelch2344
Copy link

Just coming up to speed on this thread. @spowelljr does the tag you added mean we're SOL if in this boat?

@spowelljr
Copy link
Member

Hi @dwelch2344, it was just added because it's a support issue that's been up for more than 6 months.

I can easily change the line @kgibm suggested, I would only do it for macOS specifically as I don't want to change the behavior on Linux unless we can be sure that it works on Linux as well.

Anyone else is free to make the PR themselves if that interested them, but making this change in general shouldn't be an issue.

@dwelch2344
Copy link

Yeah, definitely makes sense to do it only for Mac. -le sigh-

Haven't pulled the project to try this myself as I found another solution for my current needs, but def would love to see this fixed for the future as I'm pitching a dev practice (including minikube) to an org right now.

Will take a look through the codebase asap. Is there a good example for handling platform specific scenarios like this someone could point me to? If not, I'll jungle around – just figure might someone might have a shortcut handy :D

@spowelljr
Copy link
Member

For something small like this, if runtime.GOOS == "darwin" is appropriate.

if runtime.GOOS == "darwin" {

kgibm added a commit to kgibm/minikube that referenced this issue Jun 7, 2022
@kgibm
Copy link
Contributor Author

kgibm commented Jun 7, 2022

Created PR #14290

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/podman-driver podman driver issues kind/support Categorizes issue or PR as a support question. long-term-support Long-term support issues that can't be fixed in code os/macos
Projects
None yet
Development

Successfully merging a pull request may close this issue.