stop podman.service: unable to connect to podman socket: dial tcp ECONNREFUSED

[edsantiago]

Been happening for a while, but this is the first time I see it happen without the fd flake:

stop podman.service
...
Execing /var/tmp/go/src/github.com/containers/podman/bin/podman-remote --url tcp://127.0.0.1:38573 create --tty --name top_XVlBzgba --entrypoint top quay.io/libpod/alpine_labels:latest
         
Listening on 127.0.0.1:38573 as 3.
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.3.0/libpod/_ping": dial tcp 127.0.0.1:38573: connect: connection refused

This seems to be a one-off flake: the kind that passes when ginkgo retries. But it seems to have a really odd pattern, a correlation with #15943 (the fd flake): nearly every time we see the fd flake, we see one of these stop podman.service flakes. The difference from the above log excerpt is that when the fd flake happens, instead of ECONNREFUSED, the stop test flakes with:

Error: unexpected fd received from systemd: cannot listen on it

The above excerpt and link are the first time I see stop flaking without the fd flake.

Systemd activate [It] stop podman.service

• fedora-36 : int podman fedora-36 root container
  • PR use cached containers.conf #16238
    • 10-21 09:38
• fedora-36 : int podman fedora-36 rootless host
  • PR Ensure that the DF endpoint updated volume refcount #15753
    • 09-12-2022 17:35
• ubuntu-2204 : int podman ubuntu-2204 rootless host
  • PR Bump to v4.3.0 #16221
    • 10-18 17:03
  • PR Makefile: don't install systemd generator binaries on FreeBSD #16219
    • 10-18 11:32
  • PR build(deps): bump github.com/docker/docker from 20.10.18+incompatible to 20.10.19+incompatible #16183
    • 10-14 08:52
  • PR libpod: Factor out the call to PidFdOpen from (*Container).WaitForExit #16182
    • 10-14 09:13
  • PR Allow volume mount dups, iff source and dest dirs #16165
    • 10-15 08:21
  • PR vendor containers/storage@main #16159
    • 10-13 11:39
  • PR Avoid unnecessary calls to Container.Config() and Container.Spec() #16123
    • 10-12 08:32
  • PR fix "podman system prune networks" flake #15994
    • 09-29 09:44
    • 09-29 09:44
    • 09-29 09:44
  • PR Revert "remote: fix manifest add --annotation" #15992
    • 09-29 09:49
    • 09-29 09:01
  • PR Add volume support for FreeBSD #15962
    • 09-27 12:20
  • PR compat API: network inspect do not show isolate option #15907
    • 09-25-2022 22:53
  • PR Fix: display online_cpus in compat REST API #15867
    • 10-06 06:00
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 10-12 08:57
    • 09-28 21:46

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Is this the one that @vrothberg was working on?

[vrothberg]

Is this the one that @vrothberg was working on?

No. I fixed a flake in the system tests but we're facing an e2e test here, see https://api.cirrus-ci.com/v1/artifact/task/6512229666258944/html/int-podman-ubuntu-2204-rootless-host.log.html#t--stop-podman-service--1.

[vrothberg]

@edsantiago are we still seeing this flake?

[edsantiago]

No instances since 10-21. Since we no longer test Ubuntu, I'll just close and assume we won't see it again.