sdnotify test: podman wait: hangs

[edsantiago]

New flake seen in f36 rootless:

[+1429s] not ok 305 sdnotify : play kube - with policies
         # (from function `run_podman' in file test/system/[helpers.bash, line 207](https://github.com/containers/podman/blob/dd53ee59988d49a892fde7a86e41ef636f9814aa/test/system/helpers.bash#L207),
         #  in test file test/system/[260-sdnotify.bats, line 342](https://github.com/containers/podman/blob/dd53ee59988d49a892fde7a86e41ef636f9814aa/test/system/260-sdnotify.bats#L342))
         #   `run_podman container wait $container_a' failed
         # $ podman rm -t 0 --all --force --ignore
         # $ podman ps --all --external --format {{.ID}} {{.Names}}
         # $ podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
         # quay.io/libpod/testimage:20220615 f26aa69bb3f3
         # $ podman pull quay.io/libpod/fedora:31
         # Trying to pull quay.io/libpod/fedora:31...
         # Getting image source signatures
         # Copying blob sha256:c28ace6b0c4ae099f6f81091731bdf41d9771d28bad96ae4a3507fe950560930
         # Copying config sha256:a7a37f74ff864eec55891b64ad360d07020827486e30a92ea81d16459645b26a
         # Writing manifest to image destination
         # Storing signatures
         # a7a37f74ff864eec55891b64ad360d07020827486e30a92ea81d16459645b26a
         # $ podman container wait test_pod-a test_pod-b --condition=running
         # Error: no container with name or ID "test_pod-a" found: no such container
         # [ rc=125 ]
         # $ podman ps -a
         # CONTAINER ID  IMAGE                                        COMMAND     CREATED                 STATUS      PORTS       NAMES
         # 8afb597b4654  localhost/podman-pause:4.3.0-dev-1663157913              Less than a second ago  Created                 9cffc8377ee1-service
         # 6c9d131429af  localhost/podman-pause:4.3.0-dev-1663157913              Less than a second ago  Created                 53ad1b768e5f-infra
         # $ podman container wait test_pod-a test_pod-b --condition=running
         # Error: no container with name or ID "test_pod-a" found: no such container
         # [ rc=125 ]
         # $ podman ps -a
         # CONTAINER ID  IMAGE                                        COMMAND     CREATED                 STATUS      PORTS       NAMES
         # 8afb597b4654  localhost/podman-pause:4.3.0-dev-1663157913              Less than a second ago  Created                 9cffc8377ee1-service
         # 6c9d131429af  localhost/podman-pause:4.3.0-dev-1663157913              Less than a second ago  Created                 53ad1b768e5f-infra
         # 10dccf1282bc  quay.io/libpod/fedora:31                                 Less than a second ago  Created                 test_pod-a
         # $ podman container wait test_pod-a test_pod-b --condition=running
         # Error: no container with name or ID "test_pod-b" found: no such container
         # [ rc=125 ]
         # $ podman ps -a
         # CONTAINER ID  IMAGE                                        COMMAND     CREATED                 STATUS                     PORTS       NAMES
         # 8afb597b4654  localhost/podman-pause:4.3.0-dev-1663157913              1 second ago            Up Less than a second ago              9cffc8377ee1-service
         # 6c9d131429af  localhost/podman-pause:4.3.0-dev-1663157913              1 second ago            Up Less than a second ago              53ad1b768e5f-infra
         # 10dccf1282bc  quay.io/libpod/fedora:31                                 1 second ago            Up Less than a second ago              test_pod-a
         # 10a2c0df9a40  quay.io/libpod/testimage:20220615                        Less than a second ago  Up Less than a second ago              test_pod-b
         # $ podman container wait test_pod-a test_pod-b --condition=running
         # -1
         # -1
         # $ podman container inspect test_pod-a test_pod-b 9cffc8377ee1-service --format {{.Config.SdNotifyMode}}
         # container
         # conmon
         # ignore
         # /var/tmp/go/src/github.com/containers/podman/test/system/260-sdnotify.bats: line 326: /tmp/podman_bats.v90amR/socat.log: No such file or directory
         # $ podman container inspect test_pod-a --format {{.Config.SdNotifySocket}}
         # /var/tmp/-podman-notify-proxy.sock1715659030
         # $ podman logs test_pod-a
         # /run/notify/notify.sock
         # READY
         # $ podman exec test_pod-a /bin/touch /stop
         # $ podman container inspect 9cffc8377ee1-service --format {{.State.ConmonPid}}
         # 100133
         # $ podman container wait test_pod-a
         # timeout: sending signal TERM to command ‘/var/tmp/go/src/github.com/containers/podman/bin/podman’
         # [ rc=124 (** EXPECTED 0 **) ]
         # *** TIMED OUT ***

That is, podman container wait is hanging. Code:

podman/test/system/260-sdnotify.bats

Line 342 in ae20f19

run_podman container wait $container_a

Looking at flake logs, first logged instance was on Monday:

[sys] 304 sdnotify : play kube - with policies

• fedora-36 : sys podman fedora-36 rootless host
  • PR Ensure that the DF endpoint updated volume refcount #15753
    • 09-12 22:10

[edsantiago]

All right kids, this one's getting trendy[1]. Here we go again, still f36 rootless.

[1] for purposes of this discussion, "trendy" is not considered a Good Thing.

[vrothberg]

I'll take a look. Thanks, Ed!

[edsantiago]

Seen as root now too

[sys] 305 sdnotify : play kube - with policies

• fedora-36 : sys podman fedora-36 root host
  • PR System tests: fix three races #15794
    • 09-14 14:10
• fedora-36 : sys podman fedora-36 rootless host
  • PR label.Relabel third option is shared not recurse #15844
    • 09-17 08:32
  • PR fix ci: update systemd generate unit test #15796
    • 09-14 09:34
  • PR System tests: cleanup in --format test #15786
    • 09-14 13:49
  • PR Add support for podman context as alias to podman system connection #15072
    • 09-15 09:34

[vrothberg]

Thanks, Ed! #15820 should fix it.