podman generate systemd with added User= directive for rootless container fails with PID file is not owned by root. error

[gczarnocki]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Good morning Podman team. I have the following problem. Trying to run a Kafka pod (a part of the Kafka cluster) as rootless user, inside the container image is also running within rootless user (UID=1001). I manage host mounts permissions via podman unshare and take user namespaces into consideration. So far so good - no problems with that at all.

I'd like to achieve the automatic restart of a pod when a machine reboots. In order to do that, I executed the following steps:

1. Enabled lingering for kafka user via loginctl enable-linger
2. Generated systems unit files with: podman generate systemd -n -f osm2dkafka1 --restart-policy=no -t 5
3. Manually added User=kafka and Group=kafka directives to generated files
4. Files renamed (podman-kafka-pod and podman-kafka-container-kafka) moved to /etc/systemd/system
5. sudo systemctl deamon-reload and attempting to start a podman-kafka-pod service.

It fails with the following error:

$ sudo systemctl status podman-kafka-pod
● podman-kafka-pod.service - Podman podman-kafka-pod.service
   Loaded: loaded (/etc/systemd/system/podman-kafka-pod.service; enabled; vendor preset: disabled)
   Active: failed (Result: protocol) since Mon 2022-01-03 23:18:49 UTC; 2s ago
     Docs: man:podman-generate-systemd(1)
  Process: 63173 ExecStopPost=/usr/bin/podman stop -t 5 a5e1bd93616d-infra (code=exited, status=0/SUCCESS)
  Process: 63036 ExecStart=/usr/bin/podman start a5e1bd93616d-infra (code=exited, status=0/SUCCESS)

Jan 03 23:18:47 <host> systemd[1]: Starting Podman podman-kafka-pod.service...
Jan 03 23:18:48 <host> podman[63036]: a5e1bd93616d-infra
Jan 03 23:18:48 <host> systemd[1]: podman-kafka-pod.service: New main PID 63133 does not belong to service, and PID file is not owned by root. Refusing.
Jan 03 23:18:48 <host> systemd[1]: podman-kafka-pod.service: New main PID 63133 does not belong to service, and PID file is not owned by root. Refusing.
Jan 03 23:18:49 <host> podman[63173]: cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3
Jan 03 23:18:49 <host> systemd[1]: podman-kafka-pod.service: Failed with result 'protocol'.
Jan 03 23:18:49 <host> systemd[1]: Failed to start Podman podman-kafka-pod.service.

When I first attempt to systemctl restart podman-kafka-pod, see the loop with results of presence of PIDFiles:

<after_running systemctl restart podman-kafka-pod; failed>
Mon Jan 3 23:35:35 UTC 2022 /run/user/1002/overlay-containers/0009f816c2d5ea8ef97dd308153835ff2418ccfc857bcb9e06b136e1d7ce67a8/userdata/conmon.pid Exists 66833
Mon Jan 3 23:35:36 UTC 2022 /run/user/1002/overlay-containers/cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3/userdata/conmon.pid NotExists ???
Mon Jan 3 23:35:36 UTC 2022 /run/user/1002/overlay-containers/0009f816c2d5ea8ef97dd308153835ff2418ccfc857bcb9e06b136e1d7ce67a8/userdata/conmon.pid Exists 66833
Mon Jan 3 23:35:37 UTC 2022 /run/user/1002/overlay-containers/cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3/userdata/conmon.pid NotExists ???
< attempting to run the same command again; PIDFile is created for a split second >
Mon Jan 3 23:35:37 UTC 2022 /run/user/1002/overlay-containers/0009f816c2d5ea8ef97dd308153835ff2418ccfc857bcb9e06b136e1d7ce67a8/userdata/conmon.pid Exists 66833
Mon Jan 3 23:35:37 UTC 2022 /run/user/1002/overlay-containers/cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3/userdata/conmon.pid Exists 71512
Mon Jan 3 23:35:38 UTC 2022 /run/user/1002/overlay-containers/0009f816c2d5ea8ef97dd308153835ff2418ccfc857bcb9e06b136e1d7ce67a8/userdata/conmon.pid Exists 66833
< podman-kafka-pod unit failed and removed PIDFile; file is gone again >
Mon Jan 3 23:35:38 UTC 2022 /run/user/1002/overlay-containers/cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3/userdata/conmon.pid NotExists ???
Mon Jan 3 23:35:38 UTC 2022 /run/user/1002/overlay-containers/0009f816c2d5ea8ef97dd308153835ff2418ccfc857bcb9e06b136e1d7ce67a8/userdata/conmon.pid Exists 66833
Mon Jan 3 23:35:39 UTC 2022 /run/user/1002/overlay-containers/cf4f6f7913ff9703ff98bb023b1c050bee393fa716eb8566051a49a06a6e89f3/userdata/conmon.pid NotExists ???

So PIDFile is gone, same as infra container:

CONTAINER ID  IMAGE              COMMAND               CREATED         STATUS                                                     NAMES
cf4f6f7913ff  ubi8/pause:latest                        28 minutes ago  Exited (0) 3 minutes ago  a5e1bd93616d-infra

PIDFile is recreated when I manually run podman start cf4f6f.

Describe the results you received:

I've used podman generate systemd to generate a service unit file for a rootless container on rootless user. Tried creating system service for that by adding User= and Group= directives to generated files.

Describe the results you expected:

I am not sure if it's possible to modify the service file to add User/Group=. I saw issue: #8504 and I know that I should probably create service files under my user $HOME directory and skip User= directives completely. I need a confirmation if that's what I should follow.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Version:      3.0.2-dev
API Version:  3.0.0
Go Version:   go1.15.13
Built:        Tue Jun  8 07:52:06 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.8
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 0f5bee61b18d4581668e5bf18b910cda3cff5081'
  cpus: 4
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: <host>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 4.18.0-348.7.1.el8_5.x86_64
  linkmode: dynamic
  memFree: 12649332736
  memTotal: 16601206784
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 0
  swapTotal: 0
  uptime: 8h 47m 18.4s (Approximately 0.33 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: <home>/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-1.module+el8.5.0+12582+56d94c81.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/podman/kafka
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1002
  volumePath: /var/podman/kafka/volumes
version:
  APIVersion: 3.0.0
  Built: 1623138726
  BuiltTime: Tue Jun  8 07:52:06 2021
  GitCommit: ""
  GoVersion: go1.15.13
  OsArch: linux/amd64
  Version: 3.0.2-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.1-7.module+el8.4.0+11311+9da8acfb.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

RHEL 8.4, FIPS enabled

[Luap99]

Yes you should move it to $HOME/.config/systemd/user and use systemctl --user ... as described in podman-generate-systemd(1).
I do not know the reasons for this but this came up several times already and the answer was always that this is not supported by systemd.

[gczarnocki]

@Luap99, if that was a duplicate - sorry. I will try with systemctl --user. Thank you for clarification and response.

[dyanakiev]

Hey @Luap99 Can i execute the command after running sudo su test-user or somehow specifying the user when running the command as root? With docker i need to ssh test-user@.. in order to enable the service, otherwise the command fails.