Podman 2.2.1 breaks buildah in Podman container.

[jonasbb]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I use Podman + Buildah in Gitlab CI to build docker images. I use quay.io/buildah/stable:latest as the build container and then run buildah --storage-driver=vfs bud . in the container to create the image.

Steps to reproduce the issue:

1. Create a Dockerfile with this content:

FROM alpine:3.12.1
RUN true

2. Start build container: podman run --rm -v $(pwd):/project:Z -it quay.io/buildah/stable:latest

3. Change into /project folder: cd /project

4. Run the build: buildah --storage-driver=vfs bud .

Describe the results you received:

Building the container fails with this error:

STEP 1: FROM alpine:3.12.1
STEP 2: RUN true
WARN could not bind mount "/sys/kernel/security", skipping: no such file or directory 
error running subprocess: error remounting /var/tmp/buildah692178324/mnt/rootfs/sys/fs/cgroup in mount namespace read-only: permission denied
                                                                                                                                                                                                                                   error building at STEP "RUN true": exit status 1
ERRO exit status 1                                

journalctl -fxe reports these SELinux errors in enforcing mode.

Dez 14 11:51:29 yoga audit[219706]: AVC avc:  denied  { remount } for  pid=219706 comm="5" scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:cgroup_t:s0 tclass=filesystem permissive=0

I switched the Fedora host system to permissive. Then building succeeds with some warnings, yet still SELinux messages in the log:

STEP 1: FROM alpine:3.12.1
STEP 2: RUN true
WARN could not bind mount "/sys/kernel/security", skipping: no such file or directory 
WARN could not bind mount "/sys/firmware/efi/efivars", skipping: no such file or directory 
WARN could not bind mount "/sys/kernel/tracing", skipping: no such file or directory 
WARN could not bind mount "/sys/kernel/debug", skipping: no such file or directory 
WARN could not bind mount "/sys/kernel/config", skipping: no such file or directory 
                                                                                                                                                                                               STEP 3: COMMIT
Getting image source signatures
Copying blob ace0eda3e3be skipped: already exists  
Copying blob 0ec69a82ecdc done  
Copying config c338c8819f done  
Writing manifest to image destination
Storing signatures
--> c338c8819f7
c338c8819f7f67e46ef4a845535353b626e44ba4220b6593edd9c56f2672561e

journalctl -fxe reports these SELinux errors after switching to permissive mode:

Dez 14 11:57:37 yoga dbus-broker-launch[2602]: avc:  received setenforce notice (enforcing=0)
Dez 14 11:57:41 yoga audit[231323]: AVC avc:  denied  { remount } for  pid=231323 comm="5" scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:cgroup_t:s0 tclass=filesystem permissive=1
Dez 14 11:57:41 yoga audit[231323]: AVC avc:  denied  { mounton } for  pid=231323 comm="5" path="/var/tmp/buildah267884588/mnt/rootfs/sys/fs/pstore" dev="pstore" ino=102 scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:pstore_t:s0 tclass=dir permissive=1
Dez 14 11:57:41 yoga audit[231323]: AVC avc:  denied  { remount } for  pid=231323 comm="5" scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:pstore_t:s0 tclass=filesystem permissive=1
Dez 14 11:57:41 yoga audit[231323]: AVC avc:  denied  { mounton } for  pid=231323 comm="5" path="/var/tmp/buildah267884588/mnt/rootfs/sys/fs/bpf" dev="bpf" ino=1 scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:bpf_t:s0 tclass=dir permissive=1
Dez 14 11:57:41 yoga audit[231323]: AVC avc:  denied  { remount } for  pid=231323 comm="5" scontext=system_u:system_r:container_t:s0:c28,c957 tcontext=system_u:object_r:bpf_t:s0 tclass=filesystem permissive=1
Dez 14 11:57:41 yoga audit[231336]: SECCOMP auid=1000 uid=1000 gid=1000 ses=3 subj=system_u:system_r:container_t:s0:c28,c957 pid=231336 comm="5" exe=2F6D656D66643A6275696C6461682D6368726F6F742D65786563202864656C6574656429 sig=0 arch=c000003e syscall=170 compat=0 ip=0x55d90ad2709b code=0x50000

Describe the results you expected:
The build should be able to succeed without any warnings printed.

Additional information you deem important (e.g. issue happens only occasionally):
I can reproduce this error reliable. I first noticed the problem on my Fedora 32 machine and then reproduced the results in Fedora 33.

Downgrading the packages podman, podman-plugins and containers-common solves this issue for me

containers-common 1:1.2.0-3.fc33 
podman            2:2.1.1-10.fc33
podman-plugins    2:2.1.1-10.fc33

Shortly before this problem appeared first my system upgraded podman from podman-2:2.2.1-1.fc32.x86_64 to podman-2:2.2.0-2.fc32.x86_64. So my best guess is that version 2.2.1 introduced this regression. Unfortunately I do not know how to downgrade to version 2.2.0 to test this.

EDIT: I downloaded the podman 2.2.0 packages from here (https://koji.fedoraproject.org/koji/buildinfo?buildID=1648168) and can confirm this version works. The regression is therefore in the 2.2.0 -> 2.2.1 update.

This bug might be related to #8711. However, I see different SELinux messages related to mountin instead of writing.

Output of podman version:

Fedora 33:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

Fedora 32:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.14.10
Built:        Tue Dec  8 15:37:43 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

Fedora 33:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: yoga
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.9.12-200.fc33.x86_64
  linkmode: dynamic
  memFree: 601866240
  memTotal: 16319234048
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 12500066304
  swapTotal: 12507406336
  uptime: 11h 52m 32s (Approximately 0.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/jbushart/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/jbushart/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/jbushart/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Fedora 32:

host:          
  arch: amd64                                
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: bushart
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 975
      size: 1
    - container_id: 1
      host_id: 2328224
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 975
      size: 1
    - container_id: 1
      host_id: 2328224
      size: 65536
  kernel: 5.8.16-200.fc32.x86_64
  linkmode: dynamic
  memFree: 7273857024
  memTotal: 33573601280
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL 
  os: linux
  remoteSocket:
    path: /run/user/975/podman/podman.sock
  rootless: true
  slirp4netns:                                                                                                                                                                                                                                
    executable: /usr/bin/slirp4netns                                                                                                                                                                                                          
    package: slirp4netns-1.1.8-1.fc32.x86_64                                                                                                                                                                                                  
    version: |-                                                                                                                                                                                                                               
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 26731732992
  swapTotal: 26741170176
  uptime: 1075h 46m 50.92s (Approximately 44.79 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /run/user/975/containers
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:     
  APIVersion: 2.1.0
  Built: 1607438263                
  BuiltTime: Tue Dec  8 15:37:43 2020                                                                                  
  GitCommit: ""
  GoVersion: go1.14.10
  OsArch: linux/amd64
  Version: 2.2.1                     

Package info (e.g. output of rpm -q podman or apt list podman):

Fedora 33:

podman-2.2.1-1.fc33.x86_64
containers-common-1.2.0-10.fc33.x86_64

Fedora 32:

podman-2.2.1-1.fc32.x86_64
containers-common-1.2.0-10.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

Could be #8561

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[jonasbb]

This is still an issue and so far the workaround is using an old version.

[rhatdan]

@jonasbb Any chance you can check this against the main branch?

[jonasbb]

@rhatdan I can check it with Fedora Rawhide and a recent build from bodhi. Would that suffice?

[rhatdan]

Rawhide should suffice.

[jonasbb]

@rhatdan This does not seem fixed yet. Could you re-open the issue, please?

Writing manifest to image destination
Storing signatures
STEP 2: RUN true
WARN could not bind mount "/sys/kernel/security", skipping: no such file or directory 
                                                                                      WARN could not bind mount "/sys/kernel/tracing", skipping: no such file or directory 
                                             WARN could not bind mount "/sys/kernel/debug", skipping: no such file or directory 
  WARN could not bind mount "/sys/kernel/config", skipping: no such file or directory 
                                                                                      STEP 3: COMMIT
Getting image source signatures

podman version 3.0.0-dev, Release 0.126.dev.git0ccc888.fc34

I can re-test after #8949 is available via bodhi.

[jonasbb]

This issue is fixed for me in Rawhide with Podman 0.144.dev.git73b036d.fc34 (downloaded from bodhi). So I assume #8949 fixed this issue. Thanks :)