system test flake -- tcp connection test

[cdoern]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I thought this test was failing on my SSH PR due to my changes but now I am seeing it in 2 PRs of very different styles.

The podman system connection tcp test flakes:
https://github.com/containers/podman/pull/16601/checks?check_run_id=10234988773
https://cirrus-ci.com/task/4577298783928320

the output is

Cannot connect to Podman. Please verify your connection to the Linux system using podman system connection list, or try podman machine init and podman machine start to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.4.0/libpod/_ping": dial tcp [::1]:63371: connect: connection refused

or something like this with different ports. Could be something that changed recently in system connection add.

I have only seen it in aarch system tests and fedora 37 root system tests.

[vrothberg]

@cdoern, I suspect that wait_for_port is not enough to make sure the service is running (see https://github.com/containers/podman/blob/main/test/system/272-system-connection.bats#L115-L117).

Waiting for the port to be available does not imply that the system service is ready to serve, so I think we have a race condition in the test. Instead, I suggest to do an exponential back off waiting for podman info to succeed.

WDYT?

[rhatdan]

I think the service supports sd_notify, we could also attempt to implement something with that and have a read on leaked socket. But I guess that would hang.

[vrothberg]

I think the service supports sd_notify, we could also attempt to implement something with that and have a read on leaked socket. But I guess that would hang.

Right. It must run in parallel, so there's always some race waiting for it to be ready.

[vrothberg]

I opened #16929 to address the flake as it is hitting hard the moment.