Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: podman kube play stays in memory when running systemd service #17345

Closed
tkit1994 opened this issue Feb 3, 2023 · 8 comments · Fixed by #17469
Closed

[Bug]: podman kube play stays in memory when running systemd service #17345

tkit1994 opened this issue Feb 3, 2023 · 8 comments · Fixed by #17469
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@tkit1994
Copy link

tkit1994 commented Feb 3, 2023

Issue Description

  • podman kube play stays in memory when running
    systemctl --user enable --now podman-kube@$(systemd-escape $(realpath --relative-to=$HOME deployment.yaml))
  • if running from tty podman kube play does not exit when using --service-container flag, without the flag it can exit normally.

Steps to reproduce the issue

Steps to reproduce the issue

  1. On archlinux
  2. Upgrade podman from 4.3.1 to 4.4.0
  3. run podman kube play deployment.yaml --service-container --log-driver=journald

Describe the results you received

on 4.4.0 the command shows nothing, and keeps running

podman kube play deployment.yaml --service-container --log-driver=journald

But the service can be accessed normally.

if using systemd service podman kube play consumes around 50Mb memory for each systemd service, it's not a big deal for x86 pcs but that's too much for arm devices like raspberrypis

Describe the results you expected

on 4.3.1 the command exits normally

podman kube play deployment.yaml --service-container --log-driver=journald
Pod:
b63a3640c5efd8682e057c0eca95d1a1a0a4c7bdc0f39c2e0530c86ab04efc3e
Container:
365bd20fba48854f802b5ebb49bae7c7ddfc54591b5d0b9cad792229b6124a51

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 91.73
    systemPercent: 1.94
    userPercent: 6.33
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: tkit-pc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.9-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 5810577408
  memTotal: 12443545600
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 1h 35m 18.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/tkit/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 3
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tkit/.local/share/containers/storage
  graphRootAllocated: 499897077760
  graphRootUsed: 70373974016
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/tkit/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.0
  Built: 1675293931
  BuiltTime: Thu Feb  2 07:25:31 2023
  GitCommit: 3443f453e28169a88848f90a7ce3137fc4a4bebf-dirty
  GoVersion: go1.20
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

deployment.yaml

apiVersion: v1
kind: Pod
metadata:
  name: alist
  labels:
    name: alist
  annotations:
    io.containers.autoupdate: "registry"
spec:
  containers:
    - name: alist
      image: docker.io/xhofe/alist:latest
      resources:
      ports:
        - containerPort: 5244
          hostIP: 127.0.0.1
      volumeMounts:
        - name: data
          mountPath: /opt/alist/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: alist-data

Additional information

➜  alist git:(master) PAGER=cat LANG=c systemctl --user status podman-kube@$(systemd-escape $(realpath --relative-to=$HOME deployment.yaml)) -l
● podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service - A template for running K8s workloads via podman-kube-play
     Loaded: loaded (/usr/lib/systemd/user/[email protected]; enabled; preset: enabled)
     Active: active (running) since Fri 2023-02-03 10:35:36 CST; 9min ago
       Docs: man:podman-kube-play(1)
   Main PID: 43019 (podman)
      Tasks: 26 (limit: 14214)
     Memory: 41.5M
        CPU: 3.398s
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/app-podman\x2dkube.slice/podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service
             ├─43019 /usr/bin/podman kube play --replace --service-container=true code/scripts/podman-playkube/alist/deployment.yaml
             ├─43033 /usr/bin/conmon --api-version 1 -c 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f -u 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f/userdata -p /run/user/1000/containers/overlay-containers/3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f/userdata/pidfile -n ec405b4f17ee-service --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l k8s-file:/home/tkit/.local/share/containers/storage/overlay-containers/3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f
             ├─43098 rootlessport
             ├─43105 rootlessport-child
             ├─43114 /usr/bin/conmon --api-version 1 -c 7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df -u 7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df/userdata -p /run/user/1000/containers/overlay-containers/7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df/userdata/pidfile -n fb88fd3f0e1e-infra --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df
             └─43121 /usr/bin/conmon --api-version 1 -c c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025 -u c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025 -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025/userdata -p /run/user/1000/containers/overlay-containers/c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025/userdata/pidfile -n alist-alist --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l passthrough --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025

Feb 03 10:35:34 tkit-pc podman[43019]: 2023-02-03 10:35:34.857059689 +0800 CST m=+17.597838779 container create c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025 (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, org.opencontainers.image.version=v3.9.2, org.opencontainers.image.source=https://github.com/alist-org/alist, [email protected], org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。, org.opencontainers.image.licenses=AGPL-3.0, org.opencontainers.image.url=https://github.com/alist-org/alist, io.containers.autoupdate=registry, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075, name=alist, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service, org.opencontainers.image.title=alist)
Feb 03 10:35:34 tkit-pc podman[43019]: 2023-02-03 10:35:34.858310544 +0800 CST m=+17.599089560 container restart 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f (image=localhost/podman-pause:4.4.0-1675293931, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service)
Feb 03 10:35:35 tkit-pc podman[43019]: 2023-02-03 10:35:35.187346927 +0800 CST m=+17.928126007 container init 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f (image=localhost/podman-pause:4.4.0-1675293931, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service)
Feb 03 10:35:35 tkit-pc podman[43019]: 2023-02-03 10:35:35.341375571 +0800 CST m=+18.082154583 container start 3841a0f8094dec2f2562c7d5cd6c2c2f0a3255387955034e1b497d286754c11f (image=localhost/podman-pause:4.4.0-1675293931, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service)
Feb 03 10:35:35 tkit-pc podman[43019]: 2023-02-03 10:35:35.880710174 +0800 CST m=+18.621489197 container init 7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df (image=localhost/podman-pause:4.4.0-1675293931, name=fb88fd3f0e1e-infra, pod_id=fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service, io.buildah.version=1.29.0)
Feb 03 10:35:36 tkit-pc podman[43019]: 2023-02-03 10:35:36.001759572 +0800 CST m=+18.742538658 container start 7710cc8fd7079c9210cb8118d95105e250b200ac6425ebe3d79e5c5967eff7df (image=localhost/podman-pause:4.4.0-1675293931, name=fb88fd3f0e1e-infra, pod_id=fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de, io.buildah.version=1.29.0, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service)
Feb 03 10:35:36 tkit-pc podman[43019]: 2023-02-03 10:35:36.34386966 +0800 CST m=+19.084648823 container init c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025 (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。, org.opencontainers.image.url=https://github.com/alist-org/alist, org.opencontainers.image.version=v3.9.2, org.opencontainers.image.licenses=AGPL-3.0, org.opencontainers.image.source=https://github.com/alist-org/alist, org.opencontainers.image.title=alist, [email protected], io.containers.autoupdate=registry, name=alist, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075)
Feb 03 10:35:36 tkit-pc podman[43019]: 2023-02-03 10:35:36.71713682 +0800 CST m=+19.457915797 container start c3edb10107f887f7ecef63c8cbf74c8aeba3dfb8b9013add541726201c9ee025 (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de, org.opencontainers.image.source=https://github.com/alist-org/alist, PODMAN_SYSTEMD_UNIT=podman-kube@code-scripts-podman\x2dplaykube-alist-deployment.yaml.service, name=alist, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, [email protected], org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075, org.opencontainers.image.url=https://github.com/alist-org/alist, org.opencontainers.image.version=v3.9.2, io.containers.autoupdate=registry, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。, org.opencontainers.image.licenses=AGPL-3.0, org.opencontainers.image.title=alist)
Feb 03 10:35:36 tkit-pc podman[43019]: 2023-02-03 10:35:36.717413443 +0800 CST m=+19.458192477 pod start fb88fd3f0e1e48eb86d5f32cc27b4ea910348c16bc497c49bacf509cf3edc1de (image=, name=alist)
Feb 03 10:35:36 tkit-pc systemd[993]: Started A template for running K8s workloads via podman-kube-play.
@tkit1994 tkit1994 added the kind/bug Categorizes issue or PR as related to a bug. label Feb 3, 2023
@tkit1994
Copy link
Author

tkit1994 commented Feb 3, 2023

#16709

@tkit1994
Copy link
Author

tkit1994 commented Feb 3, 2023

After reboot. There are two podman kube play in the cgroup tree.

tkit@rpi3 ~> PAGER=cat LANG=C systemctl status --user [email protected] -l
* [email protected] - A template for running K8s workloads via podman-kube-play
     Loaded: loaded (/usr/lib/systemd/user/[email protected]; enabled; preset: enabled)
     Active: active (running) since Fri 2023-02-03 13:43:40 CST; 3h 42min ago
       Docs: man:podman-kube-play(1)
   Main PID: 559 (podman)
      Tasks: 39 (limit: 769)
     Memory: 39.6M
        CPU: 2min 58.362s
     CGroup: /user.slice/user-1001.slice/[email protected]/app.slice/app-podman\x2dkube.slice/[email protected]
             |- 414 /usr/bin/podman kube play --replace --service-container=true code/podman/k8s/alist/deployment.yaml
             |- 559 /usr/bin/podman kube play --replace --service-container=true code/podman/k8s/alist/deployment.yaml
             |- 642 /usr/bin/conmon --api-version 1 -c 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 -u 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata -p /run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/pidfile -n ec405b4f17ee-service --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l k8s-file:/home/tkit/.local/share/containers/storage/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943
             |- 798 rootlessport
             |- 830 rootlessport-child
             |- 881 /usr/bin/conmon --api-version 1 -c e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c -u e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata -p /run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/pidfile -n 1946df6646a8-infra --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c
             `-1064 /usr/bin/conmon --api-version 1 -c f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f -u f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata -p /run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/pidfile -n alist-alist --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l passthrough --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f

Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.346509636 +0800 CST m=+9.849679518 container restart 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.751386843 +0800 CST m=+10.254556465 container init 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.845971731 +0800 CST m=+10.349141249 container start 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:39 rpi3 podman[559]: 2023-02-03 13:43:39.689905868 +0800 CST m=+12.193075386 container init e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c (image=localhost/podman-pause:4.4.0-1675303101, name=1946df6646a8-infra, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, io.buildah.version=1.29.0)
Feb 03 13:43:39 rpi3 podman[559]: 2023-02-03 13:43:39.737252178 +0800 CST m=+12.240421644 container start e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c (image=localhost/podman-pause:4.4.0-1675303101, name=1946df6646a8-infra, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, io.buildah.version=1.29.0)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.829660122 +0800 CST m=+13.332829692 container init f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, org.opencontainers.image.version=v3.9.2, io.containers.autoupdate=registry, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, org.opencontainers.image.source=https://github.com/alist-org/alist, [email protected], org.opencontainers.image.licenses=AGPL-3.0, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。, org.opencontainers.image.title=alist, org.opencontainers.image.url=https://github.com/alist-org/alist, name=alist, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.885619623 +0800 CST m=+13.388789141 container start f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, org.opencontainers.image.title=alist, org.opencontainers.image.url=https://github.com/alist-org/alist, org.opencontainers.image.source=https://github.com/alist-org/alist, [email protected], name=alist, org.opencontainers.image.licenses=AGPL-3.0, org.opencontainers.image.version=v3.9.2, io.containers.autoupdate=registry, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.886245706 +0800 CST m=+13.389415276 pod start 1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644 (image=, name=alist)
Feb 03 13:43:40 rpi3 systemd[390]: [email protected]: Supervising process 559 which is not our child. We'll most likely not notice when it exits.
Feb 03 13:43:40 rpi3 systemd[390]: Started A template for running K8s workloads via podman-kube-play.

@tkit1994
Copy link
Author

tkit1994 commented Feb 3, 2023

On a raspberry pi 3b with archlinuxarm running the following services, memeory usage nearly doubled after this upgrade!

tkit@rpi3 ~> podman pod ps
POD ID        NAME         STATUS      CREATED      INFRA ID      # OF CONTAINERS
badc133b99e8  prometheus   Running     4 hours ago  70ffe7ed8694  2
0ddc27b8af27  stream-url   Running     4 hours ago  e85873aac1af  2
1946df6646a8  alist        Running     4 hours ago  e03c7002cb13  2
a90241d40de7  freenom      Running     4 hours ago  ca21735500ac  2
1c7bc8c1fd83  vlmcsd       Running     4 hours ago  b656755bf640  2
523d2237406b  vaultwarden  Running     4 hours ago  426eb0358399  2
66bf24dd0d4f  adguard      Running     4 hours ago  c350189e98fd  2

FireShot Capture 001 - Raspberry Pi - Dashboards - Grafana - tkit1994 grafana net

@vrothberg
Copy link
Member

Thanks for reaching out, @tkit1994!

This behavior is due to the changes in #16709. We can optimize to only have the podman process when custom sdnotify policies are used but we ran out of time.

@tkit1994
Copy link
Author

tkit1994 commented Feb 3, 2023

Thanks for you development, @vrothberg.
I hope there will be a memory optimization for this in future releases.

@vrothberg
Copy link
Member

@umohnani8 interested in tackling it?

@vrothberg
Copy link
Member

I have some free cycles today and will fix it.

@vrothberg
Copy link
Member

Opened #17469 to fix the issue.

vrothberg added a commit to vrothberg/libpod that referenced this issue Feb 10, 2023
Commit 4fa307f fixed a number of issues in the sdnotify proxies.
Whenever a container runs with a custom sdnotify policy, the proxies
need to keep running which in turn required Podman to run and wait for
the service container to stop.  Improve on that behavior and set the
service container as the main PID (instead of Podman) when no container
needs sdnotify.

Fixes: containers#17345
Signed-off-by: Valentin Rothberg <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 1, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants