Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linux|Podman|Docker: permission denied (selinux?) for /preloaded.tar #8016

Closed
elegos opened this issue May 6, 2020 · 4 comments · Fixed by #8017
Closed

Linux|Podman|Docker: permission denied (selinux?) for /preloaded.tar #8016

elegos opened this issue May 6, 2020 · 4 comments · Fixed by #8017
Labels
co/podman-driver podman driver issues kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@elegos
Copy link
Contributor

elegos commented May 6, 2020

Steps to reproduce the issue:

  1. minikube start --driver=podman

Full output of failed command:

~/Development/minikube(master) » minikube start --driver=podman --alsologtostderr                                  elegos@localhost
I0506 08:58:04.303362   54458 start.go:99] hostinfo: {"hostname":"localhost.localdomain","uptime":3072,"bootTime":1588745212,"procs":655,"os":"linux","platform":"fedora","platformFamily":"fedora","platformVersion":"32","kernelVersion":"5.6.8-300.fc32.x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostid":"5a3e2727-374c-4665-8b07-e67a1fc66448"}
I0506 08:58:04.304354   54458 start.go:109] virtualization: kvm host
😄  minikube v1.10.0-beta.2 on Fedora 32
I0506 08:58:04.304515   54458 driver.go:253] Setting default libvirt URI to qemu:///system
I0506 08:58:04.304571   54458 notify.go:125] Checking for updates...
I0506 08:58:04.356772   54458 podman.go:95] podman version: 1.9.1
✨  Using the podman (experimental) driver based on user configuration
I0506 08:58:04.356822   54458 start.go:215] selected driver: podman
I0506 08:58:04.356827   54458 start.go:594] validating driver "podman" against <nil>
I0506 08:58:04.356835   54458 start.go:600] status for podman: {Installed:true Healthy:true Error:<nil> Fix: Doc:}
I0506 08:58:04.356870   54458 start_flags.go:217] no existing cluster config was found, will generate one from the flags 
I0506 08:58:04.356962   54458 cli_runner.go:108] Run: sudo podman system info --format json
I0506 08:58:04.440685   54458 start_flags.go:231] Using suggested 3900MB memory alloc based on sys=15992MB, container=15992MB
I0506 08:58:04.440885   54458 start_flags.go:558] Wait components to verify : map[apiserver:true system_pods:true]
👍  Starting control plane node minikube in cluster minikube
I0506 08:58:04.441019   54458 cache.go:104] Beginning downloading kic artifacts for podman with docker
I0506 08:58:04.441030   54458 cache.go:122] Driver isn't docker, skipping base-image download
I0506 08:58:04.441045   54458 preload.go:81] Checking if preload exists for k8s version v1.18.1 and runtime docker
I0506 08:58:04.441083   54458 preload.go:96] Found local preload: /home/elegos/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4
I0506 08:58:04.441099   54458 cache.go:48] Caching tarball of preloaded images
I0506 08:58:04.441121   54458 preload.go:122] Found /home/elegos/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0506 08:58:04.441134   54458 cache.go:51] Finished verifying existence of preloaded tar for  v1.18.1 on docker
I0506 08:58:04.441559   54458 profile.go:156] Saving config to /home/elegos/.minikube/profiles/minikube/config.json ...
I0506 08:58:04.441715   54458 lock.go:35] WriteFile acquiring /home/elegos/.minikube/profiles/minikube/config.json: {Name:mkf0fc1747a7eda8f54bf02b38aabf21182a31cf Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0506 08:58:04.442117   54458 cache.go:132] Successfully downloaded all kic artifacts
I0506 08:58:04.442142   54458 start.go:223] acquiring machines lock for minikube: {Name:mk54bbd76b9ba071d84e6139eee3a3cd7ecc36f4 Clock:{} Delay:500ms Timeout:15m0s Cancel:<nil>}
I0506 08:58:04.442273   54458 start.go:227] acquired machines lock for "minikube" in 112.81µs
I0506 08:58:04.442298   54458 start.go:83] Provisioning new machine with config: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 Memory:3900 CPUs:2 DiskSize:20000 Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio KubernetesConfig:{KubernetesVersion:v1.18.1 ClusterName:minikube APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.18.1 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true]} {Name: IP: Port:8443 KubernetesVersion:v1.18.1 ControlPlane:true Worker:true}
I0506 08:58:04.442362   54458 start.go:104] createHost starting for "" (driver="podman")
🔥  Creating podman container (CPUs=2, Memory=3900MB) ...
I0506 08:58:04.442729   54458 start.go:140] libmachine.API.Create for "minikube" (driver="podman")
I0506 08:58:04.442768   54458 client.go:161] LocalClient.Create starting
I0506 08:58:04.442823   54458 main.go:110] libmachine: Reading certificate data from /home/elegos/.minikube/certs/ca.pem
I0506 08:58:04.442855   54458 main.go:110] libmachine: Decoding PEM data...
I0506 08:58:04.442873   54458 main.go:110] libmachine: Parsing certificate...
I0506 08:58:04.442990   54458 main.go:110] libmachine: Reading certificate data from /home/elegos/.minikube/certs/cert.pem
I0506 08:58:04.443015   54458 main.go:110] libmachine: Decoding PEM data...
I0506 08:58:04.443031   54458 main.go:110] libmachine: Parsing certificate...
I0506 08:58:04.443401   54458 cli_runner.go:108] Run: sudo podman ps -a --format {{.Names}}
I0506 08:58:04.508760   54458 cli_runner.go:108] Run: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0506 08:58:04.575751   54458 oci.go:98] Successfully created a podman volume minikube
W0506 08:58:04.575803   54458 oci.go:158] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0506 08:58:04.576019   54458 cli_runner.go:108] Run: sudo podman info --format "'{{json .SecurityOptions}}'"
I0506 08:58:04.575844   54458 preload.go:81] Checking if preload exists for k8s version v1.18.1 and runtime docker
I0506 08:58:04.576056   54458 preload.go:96] Found local preload: /home/elegos/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4
I0506 08:58:04.576063   54458 kic.go:133] Starting extracting preloaded images to volume ...
I0506 08:58:04.576094   54458 cli_runner.go:108] Run: sudo podman run --rm --entrypoint /usr/bin/tar -v /home/elegos/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 -I lz4 -xvf /preloaded.tar -C /extractDir
I0506 08:58:04.669075   54458 cli_runner.go:108] Run: sudo podman run --cgroup-manager cgroupfs -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var:exec --cpus=2 -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.10
I0506 08:58:04.964490   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.257314   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.299237   54458 kic.go:136] Unable to extract preloaded tarball to volume: sudo podman run --rm --entrypoint /usr/bin/tar -v /home/elegos/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 -I lz4 -xvf /preloaded.tar -C /extractDir: exit status 2
stdout:

stderr:
tar (child): /preloaded.tar: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
/usr/bin/tar: Child returned status 2
/usr/bin/tar: Error is not recoverable: exiting now
I0506 08:58:05.357448   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.449387   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.543753   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.653081   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:05.806868   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:06.031181   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:06.395715   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:06.705278   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:07.213428   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:07.896531   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:09.058715   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:10.451989   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:12.528473   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:17.072999   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:21.235810   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Running}}
I0506 08:58:21.310173   54458 client.go:164] LocalClient.Create took 16.867381712s
I0506 08:58:23.310362   54458 start.go:107] duration metric: createHost completed in 18.86798428s
I0506 08:58:23.310391   54458 start.go:74] releasing machines lock for "minikube", held for 18.868102221s
I0506 08:58:23.311243   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Status}}
I0506 08:58:23.386627   54458 stop.go:36] StopHost: minikube
✋  Stopping "minikube" in podman ...
I0506 08:58:23.387605   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Status}}
I0506 08:58:23.462207   54458 stop.go:76] host is in state Stopped
I0506 08:58:23.462250   54458 main.go:110] libmachine: Stopping "minikube"...
I0506 08:58:23.462307   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Status}}
I0506 08:58:23.533132   54458 stop.go:56] stop err: Machine "minikube" is already stopped.
I0506 08:58:23.533163   54458 stop.go:59] host is already stopped
🔥  Deleting "minikube" in podman ...
I0506 08:58:24.533519   54458 cli_runner.go:108] Run: sudo podman inspect -f {{.Id}} minikube
I0506 08:58:24.607173   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Status}}
I0506 08:58:24.677845   54458 cli_runner.go:108] Run: sudo podman exec --privileged -t minikube /bin/bash -c "sudo init 0"
I0506 08:58:24.740685   54458 oci.go:505] error shutdown minikube: sudo podman exec --privileged -t minikube /bin/bash -c "sudo init 0": exit status 255
stdout:

stderr:
Error: can only create exec sessions on running containers: container state improper
I0506 08:58:25.740823   54458 cli_runner.go:108] Run: sudo podman inspect minikube --format={{.State.Status}}
I0506 08:58:25.811602   54458 oci.go:513] container minikube status is Stopped
I0506 08:58:25.811632   54458 oci.go:525] Successfully shutdown container minikube
I0506 08:58:25.811693   54458 cli_runner.go:108] Run: sudo podman rm -f -v minikube
I0506 08:58:25.926191   54458 cli_runner.go:108] Run: sudo podman inspect -f {{.Id}} minikube
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: check container "minikube" running: temporary error created container "minikube" is not running yet
I0506 08:58:30.988354   54458 start.go:223] acquiring machines lock for minikube: {Name:mk54bbd76b9ba071d84e6139eee3a3cd7ecc36f4 Clock:{} Delay:500ms Timeout:15m0s Cancel:<nil>}
I0506 08:58:30.988625   54458 start.go:227] acquired machines lock for "minikube" in 234.481µs
I0506 08:58:30.988651   54458 start.go:83] Provisioning new machine with config: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 Memory:3900 CPUs:2 DiskSize:20000 Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio KubernetesConfig:{KubernetesVersion:v1.18.1 ClusterName:minikube APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.18.1 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true]} {Name: IP: Port:8443 KubernetesVersion:v1.18.1 ControlPlane:true Worker:true}
I0506 08:58:30.988707   54458 start.go:104] createHost starting for "" (driver="podman")
🔥  Creating podman container (CPUs=2, Memory=3900MB) ...
I0506 08:58:30.988838   54458 start.go:140] libmachine.API.Create for "minikube" (driver="podman")
I0506 08:58:30.988866   54458 client.go:161] LocalClient.Create starting
I0506 08:58:30.988893   54458 main.go:110] libmachine: Reading certificate data from /home/elegos/.minikube/certs/ca.pem
I0506 08:58:30.988923   54458 main.go:110] libmachine: Decoding PEM data...
I0506 08:58:30.988943   54458 main.go:110] libmachine: Parsing certificate...
I0506 08:58:30.989075   54458 main.go:110] libmachine: Reading certificate data from /home/elegos/.minikube/certs/cert.pem
I0506 08:58:30.989098   54458 main.go:110] libmachine: Decoding PEM data...
I0506 08:58:30.989115   54458 main.go:110] libmachine: Parsing certificate...
I0506 08:58:30.989441   54458 cli_runner.go:108] Run: sudo podman ps -a --format {{.Names}}
I0506 08:58:31.050702   54458 cli_runner.go:108] Run: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0506 08:58:31.112689   54458 client.go:164] LocalClient.Create took 123.811464ms
I0506 08:58:33.112880   54458 start.go:107] duration metric: createHost completed in 2.124159371s
I0506 08:58:33.112922   54458 start.go:74] releasing machines lock for "minikube", held for 2.124280732s
😿  Failed to start podman container. "minikube start" may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

I0506 08:58:33.113104   54458 exit.go:58] WithError(error provisioning host)=Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists
 called from:
goroutine 1 [running]:
runtime/debug.Stack(0x40c49a, 0x181f360, 0x1804900)
        /usr/lib/golang/src/runtime/debug/stack.go:24 +0x9d
k8s.io/minikube/pkg/minikube/exit.WithError(0x1a74591, 0x17, 0x1d34900, 0xc000891ae0)
        /home/elegos/Development/minikube/pkg/minikube/exit/exit.go:58 +0x34
k8s.io/minikube/cmd/minikube/cmd.runStart(0x2a5eae0, 0xc000111920, 0x0, 0x2)
        /home/elegos/Development/minikube/cmd/minikube/cmd/start.go:170 +0xac2
github.com/spf13/cobra.(*Command).execute(0x2a5eae0, 0xc000111900, 0x2, 0x2, 0x2a5eae0, 0xc000111900)
        /home/elegos/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x2a5db20, 0x0, 0x1, 0xc0000425e0)
        /home/elegos/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
        /home/elegos/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
k8s.io/minikube/cmd/minikube/cmd.Execute()
        /home/elegos/Development/minikube/cmd/minikube/cmd/root.go:108 +0x691
main.main()
        /home/elegos/Development/minikube/cmd/minikube/main.go:66 +0xe6
W0506 08:58:33.113440   54458 out.go:201] error provisioning host: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

💣  error provisioning host: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: sudo podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

The problem is being addressed (preloaded.tar has all the permissions scrumbled: "-?????????"), though I'm unsure about the cause. I suspect selinux being the problem, I'll propose a PR for this.
elegos added a commit to elegos/minikube that referenced this issue May 6, 2020
@elegos
Run podman in privileged mode extracting the kubernetes tarball into …
bd0b444 
…the volume. Fixes kubernetes#8016
@elegos elegos mentioned this issue May 6, 2020
Merged
@medyagh
Copy link
Member

medyagh commented May 6, 2020

Have u tried to turn of the selinux ?

@elegos
Copy link
Contributor Author

elegos commented May 6, 2020

@medyagh I can confirm you it's a selinux problem: temporarely disabling enforce via sudo setenforce 0 the expansion process inside of the container was able to finish correctly

@afbjorklund
Copy link
Collaborator

Actually this was known, commented on it in the original PR... Sorry for not adding a new issue.

#7961 (comment)

@afbjorklund afbjorklund added co/podman-driver podman driver issues kind/bug Categorizes issue or PR as related to a bug. labels May 6, 2020
@afbjorklund
Copy link
Collaborator

Rather unsure if SELinux works with the "podman" driver, we know it fails with the "none" driver

@elegos elegos mentioned this issue May 7, 2020
Closed
@sharifelgamal sharifelgamal added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label May 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
co/podman-driver podman driver issues kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Podman: disable selinux labels when extracting the tarball (permissions error) elegos/minikube
4 participants
@elegos @medyagh @afbjorklund @sharifelgamal