podman 2.0.1 on Fedora 32 Failed to mount tmpfs on /run: Operation not permitted #6920

space88man · 2020-07-10T02:44:22Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
podman 2.0.1 on Fedora 32 cannot run systemd based containers
Steps to reproduce the issue:

podman run --entrypoint /sbin/init centos:8

Describe the results you received:

# podman run -it --rm  --entrypoint /sbin/init --name testing centos:8
Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

Describe the results you expected:
Container runs with systemd as PID 1

Additional information you deem important (e.g. issue happens only occasionally):

Works with 1.8.2

Output of podman version:

# podman version
Version:      2.0.1
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 07:30:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:                      
  arch: amd64             
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:                                                          
    package: conmon-2.0.18-1.fc32.x86_64         
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8 
  distribution:                                                    
    distribution: fedora
    version: "32"    
  eventLogger: file  
  hostname: podman.localhost
  idMappings:
    gidmap: null  
    uidmap: null
  kernel: 5.7.7-200.fc32.x86_64
  linkmode: dynamic
  memFree: 4131205120
  memTotal: 33607065600
  ociRuntime:
    name: crun
    package: crun-0.14-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14
      commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock 
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 44h 34m 36.62s (Approximately 1.83 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 1
    stopped: 13
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 29
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 07:30:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman crun
podman-2.0.1-1.fc32.x86_64
crun-0.14-2.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

No SELinux AVCs; still doesn't work if setenforce 0
tried 2.0.2 from koji, same issue

The text was updated successfully, but these errors were encountered:

space88man · 2020-07-10T02:46:12Z

Downgrade to 1.8.2:

podman run -it --rm  --entrypoint /sbin/init --name testing centos:8
systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +X
Z +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Detected virtualization container-other.  
Detected architecture x86-64.

Welcome to CentOS Linux 8 (Core)!

Set hostname to <ccd6ae8a306d>.
Initializing machine ID from random generator.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Network is Online.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Swap.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Listening on Journal Socket.
         Starting Read and set NIS domainname from /etc/sysconfig/network...
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Dynamic Linker Cache...
         Starting Restore /run/initramfs on shutdown...
         Starting Rebuild Journal Catalog...
         Starting Rebuild Hardware Database...
[  OK  ] Started Forward Password Requests to Wall Directory Watch. 
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
         Starting Create System Users...
[  OK  ] Reached target Slices.
         Starting Journal Service...
[  OK  ] Started Read and set NIS domainname from /etc/sysconfig/network.
[  OK  ] Started Rebuild Journal Catalog.
[  OK  ] Started Create System Users.
[  OK  ] Started Rebuild Hardware Database.
[  OK  ] Started Restore /run/initramfs on shutdown.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Rebuild Dynamic Linker Cache.
         Starting Update is Completed...
[  OK  ] Started Update is Completed.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Crash recovery kernel arming...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily Cleanup of Temporary Directories.
         Starting Permit User Sessions...
[  OK  ] Started dnf makecache --timer.
[  OK  ] Reached target Timers.
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
[FAILED] Failed to start Crash recovery kernel arming.

f-bn · 2020-07-10T07:24:34Z

To run Podman container with systemd inside, you need to pass the --systemd flag in order to set correct tmpfs mounts for systemd. And don't forget the correct SELinux boolean : setsebool -P container_manage_cgroup true
http://docs.podman.io/en/latest/markdown/podman-run.1.html

vrothberg · 2020-07-10T11:35:56Z

It should work as well when the entrypoint is init or systemd. Can you try as suggested by @ruskofd with --systemd ?

space88man · 2020-07-10T13:23:54Z

@ruskofd @vrothberg It didnt work (koji's 2.0.2), getting the same error. It does work with 1.8.2 (i.e., the --systemd=true option)

podman run -it --rm   --log-level=debug --systemd=true --entrypoint /sbin/init  --name testing centos:8


INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e.scope 
DEBU[0000] Received: 132981                             
INFO[0000] Got Conmon PID as 132978                     
DEBU[0000] Created container df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e in OCI runtime 
DEBU[0000] Attaching to container df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e 
DEBU[0000] connecting to socket /var/run/libpod/socket/df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e/attach 
DEBU[0000] Received a resize event: {Width:134 Height:39} 
DEBU[0000] Starting container df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e with command [/sbin/init] 
DEBU[0000] Started container df9592482867ff5ad4a6d025a16df6dae14f5d5042c495f7ade02d7821f8bd6e 
DEBU[0000] Enabling signal proxying                     
Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

mheon · 2020-07-10T13:33:12Z

Confirmed locally.

mheon · 2020-07-10T13:38:18Z

Only happens with centos:8 and not fedora:32 - makes me think the systemd version is a problem?

mheon · 2020-07-10T13:42:52Z

Adding --privileged makes it work. Tried to narrow it down to Seccomp, SELinux, or Capabilities, but disabling just one of them doesn't fix it.

mheon · 2020-07-10T13:43:26Z

I do have an AVC from systemd in the journal

Jul 10 09:41:02 Marlborough.redhat.com audit[275766]: AVC avc:  denied  { mount } for  pid=275766 comm="systemd" name="/" dev="tmpfs" ino=2309759 scontext=system_u:system_r:container_t:s0:c166,c939 tcontext=system_u:object_r:tmpfs_t:s0 tclass=filesystem permissive=0

mheon · 2020-07-10T13:49:48Z

Hm. It seems like --systemd=true is not firing - we're not autodetecting that the container executable is systemd.

--systemd=always works perfectly.

space88man · 2020-07-10T14:19:54Z

I couldn't get it to work with fedora:32 after installing the systemd RPM. The image doesn't have /sbin/init.

I can confirm that --systemd=always is working for me, both centos:8 + fedora:32 (+ systemd installed)

mheon · 2020-07-10T18:25:59Z

Aha, think I've got it.

We aren't taking into account the entrypoint (only the command) when we determine if the container is running systemd - so we don't see your /sbin/init and don't enable systemd mode unless systemd=always is set.

We were only using the Command field in specgen when determining whether to enable systemd if systemd=true (the default) was used. This does not include the entrypoint, and does not include any entrypoint/command sourced from the image - so an image could be running systemd and we'd not correctly detect this. Using the full, final command resolves this and matches Podman v1.9.x behavior. Fixes containers#6920 Signed-off-by: Matthew Heon <[email protected]>

We were only using the Command field in specgen when determining whether to enable systemd if systemd=true (the default) was used. This does not include the entrypoint, and does not include any entrypoint/command sourced from the image - so an image could be running systemd and we'd not correctly detect this. Using the full, final command resolves this and matches Podman v1.9.x behavior. Fixes containers#6920 Signed-off-by: Matthew Heon <[email protected]> <MH: Fixed compile after backport> Signed-off-by: Matthew Heon <[email protected]>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 10, 2020

mheon mentioned this issue Jul 13, 2020

When determining systemd mode, use full command #6951

Merged

openshift-merge-robot closed this as completed in #6951 Jul 14, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman 2.0.1 on Fedora 32 Failed to mount tmpfs on /run: Operation not permitted #6920

podman 2.0.1 on Fedora 32 Failed to mount tmpfs on /run: Operation not permitted #6920

space88man commented Jul 10, 2020 •

edited

Loading

space88man commented Jul 10, 2020

f-bn commented Jul 10, 2020 •

edited

Loading

vrothberg commented Jul 10, 2020

space88man commented Jul 10, 2020 •

edited

Loading

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

space88man commented Jul 10, 2020

mheon commented Jul 10, 2020

podman 2.0.1 on Fedora 32 Failed to mount tmpfs on /run: Operation not permitted #6920

podman 2.0.1 on Fedora 32 Failed to mount tmpfs on /run: Operation not permitted #6920

Comments

space88man commented Jul 10, 2020 • edited Loading

space88man commented Jul 10, 2020

f-bn commented Jul 10, 2020 • edited Loading

vrothberg commented Jul 10, 2020

space88man commented Jul 10, 2020 • edited Loading

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

mheon commented Jul 10, 2020

space88man commented Jul 10, 2020

mheon commented Jul 10, 2020

space88man commented Jul 10, 2020 •

edited

Loading

f-bn commented Jul 10, 2020 •

edited

Loading

space88man commented Jul 10, 2020 •

edited

Loading