podman healthcheck + sdnotify: Error: container is stopped

[edsantiago]

<+015ms> # # podman run --name IK28z8EJtw --health-cmd=touch /terminate --sdnotify=healthy quay.io/libpod/testimage:20240123 sh -c while test \! -e /terminate; do sleep 0.1; done; echo finished
<+572ms> # finished
         # Error: container is stopped
<+004ms> # [ rc=126 (** EXPECTED 0 **) ]

Reproducer fails within seconds on 1mt:

# while :;do bin/podman run --health-cmd="touch /terminate" --sdnotify=healthy quay.io/libpod/testimage:20240123 sh -c "while test \! -e /terminate; do sleep 0.1; done; echo finished" || break; done
finished
finished
finished
finished
finished
finished
finished
finished
finished
finished
Error: container is stopped

While I'm at it, this is probably a bug too: (above command, with run --rm, fails instantly):

# bin/podman run --rm --health-cmd="touch /terminate" --sdnotify=healthy quay.io/libpod/testimage:20240123 sh -c "while test \! -e /terminate; do sleep 0.1; done; echo finished"
finished
# echo $?
127

Almost certainly related to #22658. @giuseppe PTAL. Only seen on aarch64, but that's consistent with the previous flake that you fixed in your PR.

• fedora-40-aarch64 : sys podman fedora-40-aarch64 root host sqlite
  • PR Use a defined constant instead of a hard-coded magic value #22731
    • 05-16 18:10 in [sys] [260] sdnotify : healthy
    • 05-16 16:49 in [sys] [260] sdnotify : healthy

x	x	x	x	x	x
sys(2)	podman(2)	fedora-40-aarch64(2)	root(2)	host(2)	sqlite(2)

[edsantiago]

OBTW vim /usr/share/containers/storage.conf and comment out the thinpool line, otherwise lots of nasty warnings

[giuseppe]

thanks for the report. The --sdnotify=healthy feature has a race condition in its implementation as we release the lock on the container middle way https://github.com/containers/podman/blob/main/libpod/container_internal.go#L1316-L1323. When the lock is released, the cleanup process deletes the container. I am not sure yet how this can be solved, except trying to not report "container is stopped" as an error.
The issue is even more evident with --rm since the container is gone once the lock is released (and the cleanup process was faster). Maybe the easiest for now is to just disallow --rm and --sdnotify=healthy.

[giuseppe]

opened a tentative PR: #22764

Marked as a Draft as I want to test it better. It doesn't solve the root issue as it would still fail if the healthcheck change takes much longer than a waiting interval