Running rootless Podman in Systemd unit with PrivateTmp=true fails

[runiq]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman as a rootless user from a systemd system unit with PrivateTmp=true hoses the user's podman installation and requires podman system reset.

Steps to reproduce the issue:

1. Set up rootless podman for your user

2. sudo systemctl start ./foo.service, where foo.service is this:

   [Unit]
   Documentation=man:podman-generate-systemd(1)
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=my-user
   Group=my-user
   RuntimeDirectory=caddy
   Delegate=yes
   KillMode=mixed
   Environment=PODMAN_SYSTEMD_UNIT=%n
   Restart=on-abnormal
   TimeoutStopSec=70
   ExecStartPre=/bin/rm -f ${RUNTIME_DIRECTORY}/%n.ctr-id
   ExecStart=/usr/bin/podman run \
       --replace \
       --rm \
       --detach \
       --cidfile=${RUNTIME_DIRECTORY}/%n.ctr-id \
       --cgroups=split \
       --sdnotify=conmon \
       --name=foo \
       docker.io/alpine
   ExecStop=/usr/bin/podman stop --ignore --cidfile=${RUNTIME_DIRECTORY}/%n.ctr-id
   ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=${RUNTIME_DIRECTORY}/%n.ctr-id
   Type=notify
   NotifyAccess=all
   PrivateTmp=true
   
   [Install]
   WantedBy=default.target

3. Start the unit

4. Restart the unit

Describe the results you received:

• Unit is not able to restart, throws the following error:

  Error: open /var/tmp/009902369: no such file or directory
  

  Where /var/tmp/<numbers> is different every time (I mean the numbers are different).
• Once this has happened, any invocation of podman images from this account throws

  Error: error retrieving label for image "0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b": you may need to remove the image to resolve the error: open /var/tmp/728259659: no such file or directory
  

  Again, <numbers> is different on every invocation of podman images.
• The only way to not get this error is to either set $TMPDIR to a path outside of /var/tmp or /tmp or do podman system reset --force, which throws this:

  WARN[0000] error determining if an image is a parent: open /var/tmp/553603433: no such file or directory, ignoring the error
  WARN[0000] error determining parent of image: open /var/tmp/908895668: no such file or directory, ignoring the error
  

  Again, the <numbers> are random in both lines.

Describe the results you expected:

• The unit restarts fine
• Afterwards, I can interact with podman from the same account in the usual way, without needing to set TMPDIR or doing podman system reset

Additional information you deem important (e.g. issue happens only occasionally):

The unit was originally generated from podman systemd generate and augmented with cgroups=split, Delegate=true, KillMode=mixed from #6666.

Output of podman version:

Version:      3.4.7
API Version:  3.4.7
Go Version:   go1.16.15
Built:        Thu Apr 21 15:14:26 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "35"
  eventLogger: journald
  hostname: trabajo.itz.uni-halle.de
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2663
      size: 1
    - container_id: 1
      host_id: 65536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 29574
      size: 1
    - container_id: 1
      host_id: 65536
      size: 65536
  kernel: 5.17.4-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 15393599488
  memTotal: 33356550144
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/29574/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 25467805696
  swapTotal: 25467805696
  uptime: 16h 40m 25.35s (Approximately 0.67 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/ajaam/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/ajaam/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/user/29574/containers
  volumePath: /home/ajaam/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.7
  Built: 1650546866
  BuiltTime: Thu Apr 21 15:14:26 2022
  GitCommit: ""
  GoVersion: go1.16.15
  OsArch: linux/amd64
  Version: 3.4.7

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.7-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

I have checked the troubleshooting guide, but I have not tested with the latest podman.

Additional environment details (AWS, VirtualBox, physical, etc.):

Stock Fedora 35, up to date.

[runiq]

I just tried with the default options from podman generate systemd (i.e. --cgroups=no-conmon, no Delegate=yes, no KillMode=mixed) and the problem already happens on the first unit start, not only on restart. I'll leave the custom options in, though, since they result in Podman logging into the systemd journal, which is pretty helpful.

[vrothberg]

Thanks for reaching out, @runiq.

Before going into the details a warning. We can only support the exact units generated by podman generate systemd. If you customize them, you are leaving supported territory. There is just too much that can go wrong. Most notably the User= and Group= directives are not supported - there are open issues detailing the reasoning.

Back to your issue: I cannot reproduce when only adding PrivateTmp=true.

$ podman create --name=123 --replace alpine top
$ podman generate systemd --new --files 123
# Edit the unit and add "PrivateTmp=true"
$ systemctl --user daemon-reload
# Start, Restart etc.

Can you retry with the upper example?

[runiq]

We can only support the exact units generated by podman generate systemd. If you customize them, you are leaving supported territory. There is just too much that can go wrong.

Ah, I wasn't aware of that! I guess this issue can be closed then, as it only appears to happen with the above constellation (that is, a systemd system unit with User= and Group=).

Most notably the User= and Group= directives are not supported - there are open issues detailing the reasoning.

I was about to file a few more issues regarding systemd hardening options in this unit file, so it's good to know I don't have to. Sorry for failing to find these issues, I'll have another look through the issue list.

Back to your issue: I cannot reproduce when only adding PrivateTmp=true.

$ podman create --name=123 --replace alpine top
$ podman generate systemd --new --files 123
# Edit the unit and add "PrivateTmp=true"
$ systemctl --user daemon-reload
# Start, Restart etc.

Can you retry with the upper example?

Yes, that one works fine. I'll close this issue then.

[runiq]

Just a quick question: What's the perspective on supporting systemd system units with User=/Group= in the future? I'm probably not the only person to envision a great future for these kinds of units, especially on CoreOS, Silverblue and friends. :)

[vrothberg]

Here's the issue: #12778

Just a quick question: What's the perspective on supporting systemd system units with User=/Group= in the future? I'm probably not the only person to envision a great future for these kinds of units, especially on CoreOS, Silverblue and friends. :)

To me it looks like a blocker on the systemd-side of things. Podman cannot do much. Note that if you want to run the units as an ordinary user, you can do that systemctl --user etc. It's just not possible as root/a system service to user User=/Group=.

[runiq]

Here's the issue: #12778

Oh, right, I even commented in that one. I'll post my results in that issue then.

(By the way, also a big 'thank you!' for your Podman series on heise.de. Just went through it, it's a great introductory read.)

[vrothberg]

Very kind of you, thanks a lot :)