Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

topas-rec · 2020-11-28T20:33:21Z

/kind bug

Description

Generating systemd files for rootles container. Starting them makes the systemctl call not return. Though the container is started. The service status is still activating.

Steps to reproduce the issue:

Create a rootless container (with uidmap) and generate systemd file on Debian Bubster
Enable and start the service

Describe the results you received:
Starting them makes the systemctl call not return. Though the container is started. The service status is still activating.

tobias@tobias-pc:~$ systemctl --user start container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service
^C
tobias@tobias-pc:~$ podman ps -a
CONTAINER ID  IMAGE                               COMMAND               CREATED       STATUS             PORTS                 NAMES
181f37a4d457  docker.io/library/nextcloud:latest  apache2-foregroun...  12 hours ago  Up 12 seconds ago  0.0.0.0:8084->80/tcp  frosty_jang
tobias@tobias-pc:~$ systemctl --user status container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service
● container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service - Podman container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service
   Loaded: loaded (/home/tobias/.config/systemd/user/container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service; enabled; vendor preset: enabled)
   Active: activating (start) since Sat 2020-11-28 21:02:56 CET; 41s ago
     Docs: man:podman-generate-systemd(1)
  Process: 21849 ExecStart=/usr/bin/podman start 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (code=exited, status=0/SUCCESS)
   CGroup: /user.slice/user-1000.slice/[email protected]/container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service
           ├─21859 /usr/bin/fuse-overlayfs -o lowerdir=/home/tobias/.local/share/containers/storage/overlay/l/WCK4KK46DYYLXOISA5VO7HE3XR:/home/tobias/.local/share/containers/storage/overlay/l/7FZWWWXA34BTPE2H4UQ
           ├─21861 /usr/libexec/podman/conmon --api-version 1 -c 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 -u 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 -r /usr/bin/r
           ├─21872 apache2 -DFOREGROUND
           ├─21881 /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 21872 tap0
           ├─21883 containers-rootlessport
           ├─21891 containers-rootlessport-child
           ├─21932 apache2 -DFOREGROUND
           ├─21933 apache2 -DFOREGROUND
           ├─21934 apache2 -DFOREGROUND
           ├─21935 apache2 -DFOREGROUND
           └─21936 apache2 -DFOREGROUND

Nov 28 21:02:56 tobias-pc systemd[5225]: Starting Podman container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service...
Nov 28 21:02:56 tobias-pc podman[21849]: 2020-11-28 21:02:56.944338702 +0100 CET m=+0.309277367 container init 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud:
Nov 28 21:02:56 tobias-pc podman[21849]: 2020-11-28 21:02:56.965846843 +0100 CET m=+0.330785470 container start 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud
Nov 28 21:02:56 tobias-pc podman[21849]: 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65
Nov 28 21:02:56 tobias-pc systemd[5225]: container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service: Can't open PID file /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7f

syslog: (journalctl cropped the lines at the end)

Nov 28 21:01:24 tobias-pc systemd[5225]: Starting Podman container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service...
Nov 28 21:01:25 tobias-pc podman[21425]: 2020-11-28 21:01:25.219565372 +0100 CET m=+0.327012366 container init 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud:latest, name=frosty_jang)
Nov 28 21:01:25 tobias-pc podman[21425]: 2020-11-28 21:01:25.234144024 +0100 CET m=+0.341590961 container start 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud:latest, name=frosty_jang)
Nov 28 21:01:25 tobias-pc podman[21425]: 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65
Nov 28 21:01:25 tobias-pc systemd[5225]: container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service: Can't open PID file /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/conmon.pid (yet?) after start: Permission denied

Why is Can't open PID file /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/conmon.pid (yet?) after start: Permission denied happening?

Describe the results you expected:
Systemctl start command returns within a reasonable time (10 seconds max) and the container is started and the service is running.

Additional information you deem important (e.g. issue happens only occasionally):
always

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: tobias-pc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.19.0-6-amd64
  linkmode: dynamic
  memFree: 3835670528
  memTotal: 8314994688
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 8387555328
  swapTotal: 8387555328
  uptime: 1h 25m 4.66s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/tobias/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/tobias/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/tobias/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 ppc64el

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

topas-rec · 2020-11-28T21:27:31Z

From #8504

Ah, I forgot about that issue. Systemd is complaining about the ownership of the PID file - systemd, when run as root, wants all PID files to also be owned by root (for security reasons). This protection even applies to unit files that use the User directive, which are clearly not running as root (so you'd imagine they'd allow the user that ran the process to be the owner of the PID file?). We talked with the systemd team about this and they felt that their current behavior is correct.

@mheon and thats the end of the story? I cannot do anything to it?

Luap99 · 2020-11-28T21:49:38Z

Ah, I forgot about that issue. Systemd is complaining about the ownership of the PID file - systemd, when run as root, wants all PID files to also be owned by root (for security reasons). This protection even applies to unit files that use the User directive, which are clearly not running as root (so you'd imagine they'd allow the user that ran the process to be the owner of the PID file?). We talked with the systemd team about this and they felt that their current behavior is correct.

I don't think you run into this issue since you run your unit with systemctl --user. The error message for the problem mentioned by @mheon is container-<id>.service: New main PID 73266 does not belong to service, and PID file is not owned by root. Refusing

topas-rec · 2020-11-28T22:44:07Z

AppArmor?

topas-rec · 2020-11-28T23:24:33Z

I guess it has to do with uid mapping:

tobias@tobias-pc:~$ ls -l /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/
total 0
drwx------ 2 100000 100000 180 Nov 28 23:57 userdata
tobias@tobias-pc:~$ ls -l /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/
ls: cannot open directory '/run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/': Permission denied

The process is mapped to the user namespace (my user ID start at 100000)

tobias@tobias-pc:~$ sudo cat /etc/subuid
tobias:100000:65536
guest:165536:65536

topas-rec · 2020-11-29T07:50:57Z

The process conmon.pid itself is owned by my user (tobias). Just the directory userdata is owned by user 100000.
Is that something podman controls?

tobias@tobias-pc:~$ sudo ls -l /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/
total 20
-rw-r--r-- 1 tobias tobias   5 Nov 29 08:45 conmon.pid
-rw-r--r-- 1 100000 100000  12 Nov 29 08:45 hostname
-rw-r--r-- 1 100000 100000 183 Nov 29 08:45 hosts
-rw-r--r-- 1 tobias tobias   0 Nov 29 08:45 oci-log
-rw-r--r-- 1 tobias tobias   5 Nov 29 08:45 pidfile
-rw-r--r-- 1 tobias tobias  62 Nov 29 08:45 resolv.conf

Luap99 · 2020-11-29T14:52:43Z

This directory should be owned by your user, otherwise systemd can't access the pid file.

I think the use of --userns keep-id triggers the permission problem of the userdata directory. I don't know if that's a bug or intended.

topas-rec · 2020-11-29T15:02:27Z

I think the use of --userns keep-id triggers the permission problem of the userdata directory. I don't know if that's a bug or intended.

The container we're talking about is started without --userns keep-id. As mentioned above I use a custom mapping of users by option --uidmap.

Let me know if there is something specific that I can provide about the container.

Luap99 · 2020-11-29T15:16:16Z

@giuseppe PTAL

topas-rec · 2020-11-29T19:34:57Z

I uncommented the PIDFile line in the service file since it is recommended when forking is used but not needed.

For someone who stumbles into this issue: This is a workaround for the hanging systemctl command since it starts the service on my machine without issues. Container is also started without issues. Stopping also works.

The current service file I use:

# container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service
# autogenerated by Podman 2.1.1
# Sun Nov 29 19:29:01 CET 2020

[Unit]
Description=Podman container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.$
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa$
ExecStop=/usr/bin/podman stop -t 10 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda$
ExecStopPost=/usr/bin/podman stop -t 10 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b4340454$
#PIDFile=/run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3$
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

github-actions · 2020-12-30T00:28:05Z

A friendly reminder that this issue had no activity for 30 days.

topas-rec · 2020-12-30T05:36:20Z

Should still be present.

rhatdan · 2020-12-30T14:38:53Z

@giuseppe were you able to look at this?

giuseppe · 2021-01-04T10:28:32Z

opened a PR here: #8869

so that the PIDFile can be accessed also without being in the rootless user namespace. Closes: containers#8506 Signed-off-by: Giuseppe Scrivano <[email protected]>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 28, 2020

topas-rec mentioned this issue Nov 28, 2020

Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

Closed

github-actions bot added the stale-issue label Dec 30, 2020

rhatdan removed the stale-issue label Dec 30, 2020

giuseppe mentioned this issue Jan 4, 2021

systemd: make rundir always accessible #8869

Merged

giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 4, 2021

systemd: make rundir always accessible

898f57c

so that the PIDFile can be accessed also without being in the rootless user namespace. Closes: containers#8506 Signed-off-by: Giuseppe Scrivano <[email protected]>

openshift-merge-robot closed this as completed in #8869 Jan 4, 2021

runbmp mentioned this issue Mar 24, 2022

generated systemd unit file hangs on enable with rootless user #13649

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020

Luap99 commented Nov 28, 2020

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020

topas-rec commented Nov 29, 2020

Luap99 commented Nov 29, 2020

topas-rec commented Nov 29, 2020

Luap99 commented Nov 29, 2020

topas-rec commented Nov 29, 2020

github-actions bot commented Dec 30, 2020

topas-rec commented Dec 30, 2020

rhatdan commented Dec 30, 2020

giuseppe commented Jan 4, 2021

Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

Comments

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020

Luap99 commented Nov 28, 2020

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020

topas-rec commented Nov 29, 2020

Luap99 commented Nov 29, 2020

topas-rec commented Nov 29, 2020

Luap99 commented Nov 29, 2020

topas-rec commented Nov 29, 2020

github-actions bot commented Dec 30, 2020

topas-rec commented Dec 30, 2020

rhatdan commented Dec 30, 2020

giuseppe commented Jan 4, 2021