Netavark: userns: nc: bad address 'nc-server.dns.podman'

[edsantiago]

In podman int tests:

podman Netavark network works across user ns
...
$ podman [options] run --log-driver k8s-file --rm --net 051ae9d9bd --uidmap 0:1:4096 \
       quay.io/libpod/alpine:latest \
       sh -c echo podman | nc -w 1 nc-server.dns.podman 9480
nc: bad address 'nc-server.dns.podman'

Probably a race condition in test setup. Rare, but still triggering.

Podman run networking [It] podman Netavark network works across user ns

• fedora-36 : int podman fedora-36 root container
  • PR Add support to sig-proxy for podman-remote #15131
    • 09-20-2022 17:14
    • 09-20-2022 17:14
    • 09-20-2022 17:14
• fedora-36 : int podman fedora-36 rootless host
  • PR Fix system df issues with -f and -v #16234
    • 10-21 00:02
    • 10-21 00:02
    • 10-21 00:02
• fedora-36 : int remote fedora-36 root host [remote]
  • PR Makefile: Mount . with --security-opt label=disable instead of using --privileged #15479
    • 08-29-2022 04:14
    • 08-29-2022 04:14
    • 08-29-2022 04:14
  • PR Remove duplicate annotations in generated service yaml #15474
    • 08-25-2022 00:55
    • 08-25-2022 00:55
    • 08-25-2022 00:55

[Luap99]

I think this is the same as #14173,
the dns tests never fails alone which means that some condition is causing aardvark to fail and keep it from working again.
We reworked the start-up so it should be better now? Do we know the netavark/aardvark version from the tests?

[edsantiago]

• Click one of the timestamps in the latest failure (the 10-21 one)
  • Press Home key to go to top of log
    • Click the Task link
      • Press End key to go to bottom
        • Click the 'Run package_versions` accordion

Shows netavark-1.1.0-1.fc36-x86_64 (does not show aardvark; I'll open a PR to fix that)

[Edit: instructions above provided as reference because this isn't the last time we'll need to know package versions]

[Luap99]

Thanks, we keep aardvark and netavark in sync so it should be the same. Using netavark/aardvark v1.2 might fix this but honestly I don't know since the underlying cause is unknown.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove stale

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove stale

[Luap99]

@edsantiago Any new instances in your log?

[edsantiago]

@Luap99 I'm still working my way through the last few weeks of flakes. I'll try to answer later today.

[edsantiago]

Never mind, I forgot about cirrus-flake-grep. It shows the last instances on October 20 and 21, so I'm declaring this fixed.

[edsantiago]

Reopening. My disable-retries PR (#17831) shows that this is still happening even on main with updated CI VMs:

podman Netavark network works across user ns
...
# podman [options] network create f48f8e5b27
f48f8e5b27
# podman [options] run --log-driver k8s-file -d --name nc-server --net f48f8e5b27 quay.io/libpod/alpine:latest nc -l -p 9480
117a5425064b2bd6b8417e53c797b8a2271c8fcc2f5f3cfcef9cc89433b60e55
# podman [options] run --log-driver k8s-file --rm --net f48f8e5b27 --uidmap 0:1:4096 quay.io/libpod/alpine:latest sh -c echo podman | nc -w 1 nc-server.dns.podman 9480
nc: bad address 'nc-server.dns.podman'

We're probably not seeing it because it passes on retry.

[edsantiago]

Another one, remote f37 root

[edsantiago]

And another one, f37 root not-remote

[Luap99]

Sure this was never really fixed. There is absolutely no guarantee that aardvark-dns is ready before the container is started.

[edsantiago]

There is absolutely no guarantee that aardvark-dns is ready before the container is started.

This disturbs me. For tests, it's easy to add sleep(5) and sweep that under the rug, but I can envision a number of real-world scenarios in which users will want to do DNS queries early on. (In fact I do see a /remove stale from an outside party, suggesting there is at least one real-world person experiencing this issue).

Do we have plans to fix this?

[Luap99]

No plans so far. The big race condition was fixed. Now it is much harder to hit this issue that is why the flake went under your radar.

[edsantiago]

FWIW, here's my motivation for cleaning this up in #18342. It's failing quite often:

• debian-12 : int remote debian-12 root host sqlite [remote]
  • 04-24 18:10 in Podman run networking [It] podman Netavark network works across user ns
  • 04-24 12:44 in Podman run networking [It] podman Netavark network works across user ns
  • 04-20 21:34 in Podman run networking [It] podman Netavark network works across user ns
• fedora-37 : int podman fedora-37 root container sqlite
  • 04-20 21:22 in Podman run networking [It] podman Netavark network works across user ns
  • 04-20 10:58 in Podman run networking [It] podman Netavark network works across user ns
• fedora-37 : int remote fedora-37 root host sqlite [remote]
  • 04-24 18:09 in Podman run networking [It] podman Netavark network works across user ns
  • 04-19 17:37 in Podman run networking [It] podman network works across user ns