systemd managed pod: when one container restarts, all do

[kdknigga]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When one container in a systemd-managed pod is restarted, all containers in the pod are restarted.

Steps to reproduce the issue:

1. Create and start a pod with 2 more more containers.

2. Use podman generate systemd --files --new --name mypod to create systemd unit files for the pod and containers.

3. Stop the pod and containers via podman, and then start them using systemctl start pod-mypod.

4. Restart one of the containers via systemctl restart container-one and note that the other containers in the pod also restart.

Describe the results you expected:

I expect that only the container explicitly restarted would restart. This is behavior demonstrated here:

https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods#pods_and_container_management

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman --version
podman version 3.4.1-dev

Output of podman info --debug:

$ sudo podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.30-1.module_el8.6.0+944+d413f95e.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: e28f6ed9f4a6f18e27f3efdab92de483806e6b9c'
  cpus: 16
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: kdkworkstation
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-408.el8.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 14572630016
  memTotal: 25229623296
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 10636746752
  swapTotal: 10636746752
  uptime: 24h 47m 6.14s (Approximately 1.00 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 5
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.1-dev
  Built: 1634659902
  BuiltTime: Tue Oct 19 11:11:42 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.4.1-dev

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-3.4.1-3.module_el8.6.0+954+963caf36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

I have checked the troubleshooting guide, and I've searched through github issues. While there is a newer version of podman available for Centos Stream 8, it breaks my ability to use passed-in devices due to the issue fixed by 61f6e13. 3.4.1 appears to be the latest version easily available to me with which my containers work.

Additional environment details (AWS, VirtualBox, physical, etc.):

I'm running Centos Stream 8, fullly updated as of yesterday, with the exception of having the container-tools module version locked as follows due to the --device bug found in 4.0.0-4.0.2:

$ sudo dnf versionlock
Last metadata expiration check: 1:59:41 ago on Wed Aug  3 09:08:23 2022.
buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.*
cockpit-podman-0:35-1.module_el8.6.0+944+d413f95e.*
conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.*
container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.*
containernetworking-plugins-0:1.0.1-1.module_el8.6.0+944+d413f95e.*
containers-common-2:1-6.module_el8.6.0+954+963caf36.*
criu-0:3.15-3.module_el8.6.0+926+8bef8ae7.*
crun-0:1.2-1.module_el8.6.0+954+963caf36.*
fuse-overlayfs-0:1.7.1-1.module_el8.6.0+926+8bef8ae7.*
libslirp-0:4.4.0-1.module_el8.6.0+926+8bef8ae7.*
podman-1:3.4.1-3.module_el8.6.0+954+963caf36.*
podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.*
python3-podman-0:3.2.1-4.module_el8.6.0+954+963caf36.*
runc-0:1.0.2-1.module_el8.6.0+926+8bef8ae7.*
skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.*
slirp4netns-0:1.1.8-1.module_el8.6.0+926+8bef8ae7.*
toolbox-0:0.0.99.3-0.4.module_el8.6.0+944+d413f95e.*
udica-0:0.2.6-2.module_el8.6.0+944+d413f95e.*

Here are my actual unit files instead of the contrived example above:

$ systemctl --no-pager cat pod-iot
# /etc/systemd/system/pod-iot.service
# pod-iot.service
# autogenerated by Podman 3.4.1-dev
# Wed Aug  3 10:22:41 CDT 2022

[Unit]
Description=Podman pod-iot.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=
Requires=container-homeassistant.service container-zwavejs.service
Before=container-homeassistant.service container-zwavejs.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/pod-iot.pid %t/pod-iot.pod-id
ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile %t/pod-iot.pid --pod-id-file %t/pod-iot.pod-id --name=iot -p 8091:8091 -p 8123:8123 --replace
ExecStart=/usr/bin/podman pod start --pod-id-file %t/pod-iot.pod-id
ExecStop=/usr/bin/podman pod stop --ignore --pod-id-file %t/pod-iot.pod-id -t 10
ExecStopPost=/usr/bin/podman pod rm --ignore -f --pod-id-file %t/pod-iot.pod-id
PIDFile=%t/pod-iot.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

$ systemctl --no-pager cat container-homeassistant
# /etc/systemd/system/container-homeassistant.service
# container-homeassistant.service
# autogenerated by Podman 3.4.1-dev
# Wed Aug  3 10:22:41 CDT 2022

[Unit]
Description=Podman container-homeassistant.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
BindsTo=pod-iot.service
After=pod-iot.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --rm --pod-id-file %t/pod-iot.pod-id --sdnotify=conmon --conmon-pidfile /run/container-homeassistant.pid --conmon-pidfile /run/container-homeassistant.pid -d --replace --label io.containers.autoupdate=image --mount type=volume,src=homeassistant,target=/config -e TZ=America/Chicago --name homeassistant ghcr.io/home-assistant/home-assistant:stable
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target default.target

$ systemctl --no-pager cat container-zwavejs
# /etc/systemd/system/container-zwavejs.service
# container-zwavejs.service
# autogenerated by Podman 3.4.1-dev
# Wed Aug  3 10:22:41 CDT 2022

[Unit]
Description=Podman container-zwavejs.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
BindsTo=pod-iot.service
After=pod-iot.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --rm --pod-id-file %t/pod-iot.pod-id --sdnotify=conmon --conmon-pidfile /run/container-zwavejs.pid --conmon-pidfile /run/container-zwavejs.pid -d --replace --label io.containers.autoupdate=image --device=/dev/serial/by-id/usb-Silicon_Labs_CP2102N_USB_to_UART_Bridge_Controller_88e1d0665594eb11943836703d98b6d1-if00-port0:/dev/zwave:rwm --mount type=volume,src=zwavejs2mqtt,target=/usr/src/app/store --name zwavejs zwavejs/zwavejs2mqtt:test
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target default.target

[vrothberg]

Thanks for reaching out, @kdknigga!

#14546 is already discussing the issue, so I am closing the issue here as a duplicate.

[kdknigga]

I don't know how my issue search didn't find that!

Thanks!