e2e: network scoped DNS server test: nslookup write: ECONNREFUSED

[edsantiago]

New flake seen twice now in my no-flake-retries PR:

podman verify network scoped DNS server and also verify updating network dns server
...
# podman-remote [options] network create IntTest4c3cee2e11 --dns 1.1.1.1
...
# podman-remote [options] run -d --name con1 --network IntTest4c3cee2e11 busybox top
74b815892191c854cef88dba95dc09edd362a8d65a09b326db55d65e77b9dc61
# podman-remote [options] exec con1 nslookup google.com 10.89.1.1
nslookup: write to '10.89.1.1': Connection refused    <<<< expected: "Non-authoritative answer"
;; connection timed out; no servers could be reached

Maybe just another manifestation of containers/aardvark-dns#338 ?

• debian-12 : int remote debian-12 root host sqlite [remote]
  • 06-14 09:18 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
• rawhide : int podman rawhide root host sqlite
  • 06-07 22:02 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server

[Luap99]

Could be related, but maybe not. Journal shows this shortly after:

Jun 14 13:04:24 cirrus-task-4832038906232832 aardvark-dns[225152]: 2816 dns request got empty response
Jun 14 13:04:24 cirrus-task-4832038906232832 aardvark-dns[225152]: 2816 dns request got empty response
Jun 14 13:04:24 cirrus-task-4832038906232832 con1[225264]:    11     0 root     S     1316  0.0   1  0.0 nslookup google.com 10.89.2.1

However that may be expected and is caused by another test after that which passes.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago still an issue?

[edsantiago]

Infrequent, but yes, still present as of this weekend:

• debian-12 : int podman debian-12 root host sqlite
  • 07-10-2023 18:34 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
• fedora-38 : int podman fedora-38 root host sqlite
  • 08-26 22:22 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
• fedora-38 : int remote fedora-38 root host boltdb [remote]
  • 07-06-2023 11:03 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
• rawhide : int podman rawhide root host sqlite
  • 07-31-2023 07:55 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server

[edsantiago]

Seen again last night. And a handful of times since my last update. All the old logs are gone, but my new summarize code shows the breakdown table, so this is still useful.

• fedora-38 : int podman fedora-38 root host sqlite
  • 08-26-2023 22:22 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
• fedora-39 : int podman fedora-39 root container sqlite
  • 01-15 17:33 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
  • 12-06-2023 11:29 in Podman run networking podman run check dns
  • 11-28-2023 12:26 in Podman run networking podman run check dns
• rawhide : int podman rawhide root host sqlite
  • 07-31-2023 07:55 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server

x	x	x	x	x	x
int(5)	podman(5)	fedora-39(3)	root(5)	container(3)	sqlite(5)
		rawhide(1)		host(2)
		fedora-38(1)

[edsantiago]

Still infrequent, but two hits this month

• debian-13 : int podman debian-13 root host sqlite
  • 08-12 15:11 in Podman run networking podman verify network scoped DNS server and also verify updating network dns server
  • 08-05 18:33 in Podman run networking podman run check dns
• fedora-38 : int podman fedora-38 root host boltdb
  • 02-15-2024 14:06 in Podman run networking podman run check dns
• fedora-39 : int podman fedora-39 root host sqlite
  • 02-19-2024 12:36 in Podman run networking podman run check dns
• fedora-40 : int remote fedora-40 root host sqlite [remote]
  • 05-23-2024 08:18 in Podman run networking podman run check dns
• rawhide : int podman rawhide root host sqlite
  • 02-29-2024 11:48 in Podman run networking podman run check dns
  • 02-21-2024 12:05 in Podman run networking podman run check dns

x	x	x	x	x	x
int(7)	podman(6)	rawhide(2)	root(7)	host(7)	sqlite(6)
	remote(1)	debian-13(2)			boltdb(1)
		fedora-39(1)
		fedora-40(1)
		fedora-38(1)

[edsantiago]

Here's an interesting little hiccup: this really looks like the same flake, but it's in system tests (Bats) (debian root), and it's the first time I've ever seen it there, and it's in my parallel-testing PR:

<+053ms> # # podman exec 22cb717ee5c44ff7b8362655a2f4946ab08311d4c5cefc6a65c92fb8a6a5191e nslookup google.com
<+153ms> # ;; connection timed out; no servers could be reached
         #
         # nslookup: write to '172.15.164.1': Connection refused
<+006ms> # [ rc=1 (** EXPECTED 0 **) ]

I'm not sure what that means, if anything, but it's eyebrowraising enough for me to comment on.

[Luap99]

Yes this is expected, we need at least aardvark-dns 1.12 to fix some reliability problems when starting. Before that we never waited for aardvark-dns to bind the port on start. (This might take years on debian)
But unless containers/aardvark-dns#338 is fixed we might always have some dns flakes I think, but for this specific system test alone it should already work