Privileged containers cannot be restarted if host devices changed

[gcampax]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

If you create a privileged container, stop it, then change the set of devices attached to the machine (e.g. reboot and unplug a USB device), the container will fail to start with:

Error: unable to start container "e6dee751631505392ce8f01701bdf93dc9af9171497ce4631a6e4d6ef6cc965a": crun: error stat'ing file `/dev/cec0`: No such file or directory: OCI runtime attempted to invoke a command that was not found

(the device file will change, and there can be many such devices)

This is particularly annoying because it affects k8s clusters created with kind on a laptop, because the container is privileged, it needs to be restarted, not recreated (otherwise all k8s configs are gone), devices obviously change often, and the machine is rebooted often.

Steps to reproduce the issue:

1. Create a kind cluster on a laptop with some USB device attached.
2. Unplug that device and reboot.
3. Try to restart the control plane container of that cluster.

Describe the results you received:
Container fails to start.

Describe the results you expected:
Container starts normally.

Additional information you deem important (e.g. issue happens only occasionally):
I found this code:

podman/pkg/specgen/generate/config_linux.go

Line 25 in 54f808e

func addPrivilegedDevices(g *generate.Generator) error {

It seems podman will walk through all devices visible to the current user, and add them to the OCI spec. The OCI spec is only created when the container is created, not when the container is started. This clearly breaks because the set of visible devices changes.

I see three solutions to this:
a. podman should recreate the OCI spec from its own container config when starting a container
b. podman should have a flag to not mount all devices in privileged containers, and kind should use that flag (or ideally it would be a flag in containers.conf so it can be set globally without changes to kind)
c. some flag should be set in the OCI spec so missing devices do not fail starting the container

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.18beta2

Built:      Thu Mar  3 06:56:09 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 16
  distribution:
    distribution: fedora
    variant: silverblue
    version: "36"
  eventLogger: journald
  hostname: redacted
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.17.1-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 14018764800
  memTotal: 33396903936
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 25333587968
  swapTotal: 25333587968
  uptime: 57m 12.2s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/redacted/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.8.1-3.fc36.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.8.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/redacted/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 585
  runRoot: /run/user/1000
  volumePath: /var/home/redacted/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319369
  BuiltTime: Thu Mar  3 06:56:09 2022
  GitCommit: ""
  GoVersion: go1.18beta2
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.2-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[mheon]

This is a known issue (#4900). I'm not really sure why we closed that one - it's definitely not fixed.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[debarshiray]

I wonder what privileged means here. Toolbx containers are also created with --privileged and I couldn't reproduce this with Podman 3.4.7 on a Fedora 35 host:

• I plugged in a USB stick
• Created a new container with toolbox create
• Entered it with toolbox enter (that includes podman start)
• Got out
• Stopped it with podman stop
• Pulled out the USB stick
• Entered the container again with toolbox enter

[edsantiago]

It's imperfect, but one way to test might be:

dev=/dev/testdev-$(random_string)
mknod $dev c [SOMETHING SOMETHING]
podman run -n foo ... ls -l $dev
rm -f $dev
podman start foo
podman logs foo
...confirm that the last output line is ENOENT