-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: podman kube play stays in memory when running systemd service #17345
Comments
After reboot. There are two tkit@rpi3 ~> PAGER=cat LANG=C systemctl status --user [email protected] -l
* [email protected] - A template for running K8s workloads via podman-kube-play
Loaded: loaded (/usr/lib/systemd/user/[email protected]; enabled; preset: enabled)
Active: active (running) since Fri 2023-02-03 13:43:40 CST; 3h 42min ago
Docs: man:podman-kube-play(1)
Main PID: 559 (podman)
Tasks: 39 (limit: 769)
Memory: 39.6M
CPU: 2min 58.362s
CGroup: /user.slice/user-1001.slice/[email protected]/app.slice/app-podman\x2dkube.slice/[email protected]
|- 414 /usr/bin/podman kube play --replace --service-container=true code/podman/k8s/alist/deployment.yaml
|- 559 /usr/bin/podman kube play --replace --service-container=true code/podman/k8s/alist/deployment.yaml
|- 642 /usr/bin/conmon --api-version 1 -c 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 -u 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata -p /run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/pidfile -n ec405b4f17ee-service --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l k8s-file:/home/tkit/.local/share/containers/storage/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943
|- 798 rootlessport
|- 830 rootlessport-child
|- 881 /usr/bin/conmon --api-version 1 -c e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c -u e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata -p /run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/pidfile -n 1946df6646a8-infra --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c
`-1064 /usr/bin/conmon --api-version 1 -c f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f -u f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f -r /usr/bin/crun -b /home/tkit/.local/share/containers/storage/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata -p /run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/pidfile -n alist-alist --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l passthrough --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/oci-log --conmon-pidfile /run/user/1001/containers/overlay-containers/f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tkit/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/tkit/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f
Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.346509636 +0800 CST m=+9.849679518 container restart 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.751386843 +0800 CST m=+10.254556465 container init 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:37 rpi3 podman[559]: 2023-02-03 13:43:37.845971731 +0800 CST m=+10.349141249 container start 2a6bcba3cb2687de60fa72e0a863769462860e57adb7298606af37b64303d943 (image=localhost/podman-pause:4.4.0-1675303101, name=ec405b4f17ee-service, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service)
Feb 03 13:43:39 rpi3 podman[559]: 2023-02-03 13:43:39.689905868 +0800 CST m=+12.193075386 container init e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c (image=localhost/podman-pause:4.4.0-1675303101, name=1946df6646a8-infra, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, io.buildah.version=1.29.0)
Feb 03 13:43:39 rpi3 podman[559]: 2023-02-03 13:43:39.737252178 +0800 CST m=+12.240421644 container start e03c7002cb13aaf7efdf249696ad6e225863f1d38d056d1f6b9bc307ecdcc35c (image=localhost/podman-pause:4.4.0-1675303101, name=1946df6646a8-infra, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, io.buildah.version=1.29.0)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.829660122 +0800 CST m=+13.332829692 container init f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, org.opencontainers.image.version=v3.9.2, io.containers.autoupdate=registry, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, org.opencontainers.image.source=https://github.com/alist-org/alist, [email protected], org.opencontainers.image.licenses=AGPL-3.0, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。, org.opencontainers.image.title=alist, org.opencontainers.image.url=https://github.com/alist-org/alist, name=alist, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.885619623 +0800 CST m=+13.388789141 container start f9bae35c1ae423fea87b0025535eb8a013345500b304353184ef63abaaf5244f (image=docker.io/xhofe/alist:latest, name=alist-alist, pod_id=1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644, org.opencontainers.image.revision=c92e11dad5cf5cb300c184cc49e31d0f811cc075, PODMAN_SYSTEMD_UNIT=podman-kube@code-podman-k8s-alist-deployment.yaml.service, org.opencontainers.image.created=2023-01-27T07:54:36.317Z, org.opencontainers.image.title=alist, org.opencontainers.image.url=https://github.com/alist-org/alist, org.opencontainers.image.source=https://github.com/alist-org/alist, [email protected], name=alist, org.opencontainers.image.licenses=AGPL-3.0, org.opencontainers.image.version=v3.9.2, io.containers.autoupdate=registry, org.opencontainers.image.description=🗂️A file list program that supports multiple storage, powered by Gin and Solidjs. / 一个支持多存储的文件列表程序,使用 Gin 和 Solidjs。)
Feb 03 13:43:40 rpi3 podman[559]: 2023-02-03 13:43:40.886245706 +0800 CST m=+13.389415276 pod start 1946df6646a832abb40cb6c62c982a6a74afd757af276045dcdb8a482fc5c644 (image=, name=alist)
Feb 03 13:43:40 rpi3 systemd[390]: [email protected]: Supervising process 559 which is not our child. We'll most likely not notice when it exits.
Feb 03 13:43:40 rpi3 systemd[390]: Started A template for running K8s workloads via podman-kube-play. |
On a raspberry pi 3b with archlinuxarm running the following services, memeory usage nearly doubled after this upgrade! tkit@rpi3 ~> podman pod ps
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
badc133b99e8 prometheus Running 4 hours ago 70ffe7ed8694 2
0ddc27b8af27 stream-url Running 4 hours ago e85873aac1af 2
1946df6646a8 alist Running 4 hours ago e03c7002cb13 2
a90241d40de7 freenom Running 4 hours ago ca21735500ac 2
1c7bc8c1fd83 vlmcsd Running 4 hours ago b656755bf640 2
523d2237406b vaultwarden Running 4 hours ago 426eb0358399 2
66bf24dd0d4f adguard Running 4 hours ago c350189e98fd 2 |
Thanks for you development, @vrothberg. |
@umohnani8 interested in tackling it? |
I have some free cycles today and will fix it. |
Opened #17469 to fix the issue. |
Commit 4fa307f fixed a number of issues in the sdnotify proxies. Whenever a container runs with a custom sdnotify policy, the proxies need to keep running which in turn required Podman to run and wait for the service container to stop. Improve on that behavior and set the service container as the main PID (instead of Podman) when no container needs sdnotify. Fixes: containers#17345 Signed-off-by: Valentin Rothberg <[email protected]>
Issue Description
podman kube play
stays in memory when runningsystemctl --user enable --now podman-kube@$(systemd-escape $(realpath --relative-to=$HOME deployment.yaml))
podman kube play
does not exit when using--service-container
flag, without the flag it can exit normally.Steps to reproduce the issue
Steps to reproduce the issue
podman kube play deployment.yaml --service-container --log-driver=journald
Describe the results you received
on 4.4.0 the command shows nothing, and keeps running
But the service can be accessed normally.
if using systemd service
podman kube play
consumes around 50Mb memory for each systemd service, it's not a big deal for x86 pcs but that's too much for arm devices like raspberrypisDescribe the results you expected
on 4.3.1 the command exits normally
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
deployment.yaml
Additional information
The text was updated successfully, but these errors were encountered: