generated systemd unit file hangs on enable with rootless user

[runbmp]

/kind bug

Description

#8506 appears to be back

Steps to reproduce the issue:

1. podman-compose up -d && podman-compose stop

2. podman generate systemd --name caddy --files && mv container-caddy.service ~/.config/systemd/user/

3. systemctl --user enable --now container-caddy

Describe the results you received:
Enabling the unit hangs

[poduser@600sff userdata]$ systemctl --user enable --now container-caddy
Created symlink /home/poduser/.config/systemd/user/default.target.wants/container-caddy.service → /home/poduser/.config/systemd/user/container-caddy.service.
^C

Describe the results you expected:
Enabling the unit should not hang

Additional information you deem important (e.g. issue happens only occasionally):

PODMAN_USERNS=keep-id in my .bashrc

journalctl reports:
Mar 24 10:23:55 600sff systemd[1061]: container-caddy.service: Can't open PID file /run/user/1000/containers/overlay-containers/68941815877b980504e738423928f8545240f74b6bfb9fa1f5ddbcf947dfd4ec/userdata/conmon.pi>

[poduser@600sff userdata]$ ls
ls: cannot open directory '.': Permission denied
[poduser@600sff userdata]$ cd ../
[poduser@600sff de1c0da7a5d89f9a81c032d64ddbf15f8c55561a2794239f76ae507fd6316213]$ ls -la
total 0
drwx------ 3 poduser easystore  60 Mar 24 10:22 .
drwx------ 3 poduser easystore  60 Mar 24 10:22 ..
drwx--x--x 2  100000    100000 180 Mar 24 10:32 userdata

Output of podman version:

[poduser@600sff overlay-containers]$ podman version
Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.8
Git Commit:   c8b9a2e3ec3630e9172499e15205c11b823c8107
Built:        Sun Mar 13 05:44:23 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

[poduser@600sff overlay-containers]$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: 600sff
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.16-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 3233116160
  memTotal: 8238477312
  networkBackend: cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4108316672
  swapTotal: 4118802432
  uptime: 42h 34m 7.66s (Approximately 1.75 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/poduser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/poduser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  volumePath: /home/poduser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1647171863
  BuiltTime: Sun Mar 13 05:44:23 2022
  GitCommit: c8b9a2e3ec3630e9172499e15205c11b823c8107
  GoVersion: go1.17.8
  OsArch: linux/amd64
  Version: 4.0.1

[poduser@600sff overlay-containers]$ podman-compose --version
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.0.1
podman-composer version 1.0.4
podman --version
podman version 4.0.1
exit code: 0

[poduser@600sff caddy]$ cat docker-compose.yml
version: "3.4"
services:
  caddy:
    build:
      context: .
      dockerfile: /home/poduser/pods/caddy/Dockerfile
    container_name: caddy
    user: 1000:1002
    environment:
      - PUID=1000
      - PGID=1002
    cap_add:
      - NET_BIND_SERVICE
    ports:
      - 80:80
      - 443:443
      - 2019:2019
    volumes:
      - /home/poduser/pods/caddy/config:/config
      - /home/poduser/pods/caddy/data:/data
      - /home/poduser/pods/caddy/Caddyfile:/etc/caddy/Caddyfile
    restart: unless-stopped

[runbmp]

I did a podman-compose down followed by podman container prune and podman image prune and then a podman-compose up and podman-compose stop and regenerated the systemd units and they work just fine.