Aardvark/Netavark flakes #14173

edsantiago · 2022-05-09T21:18:24Z

Aardvark Test 2: Two containers, same subnet
...

Expected
               <[]string | len:0, cap:0>: nil
           to have length 2
         
           /var/tmp/go/src/github.com[/containers/podman/test/e2e/run_aardvark_test.go:105](https://github.com/containers/podman/blob/bb8f53a727aeebb49873b2f4c9c12d6f1d0ce1d3/test/e2e/run_aardvark_test.go#L105)

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

fedora-35 : Netavark int podman root
fedora-36 : int podman fedora-36 root container
- PR test: fix "podman search format json" #14152
  - 05-08 12:26
fedora-36 : int podman fedora-36 root host
- PR e2e tests: use HaveField() for better error checking #14040
  - 04-28 08:29
  - 04-28 08:29

The text was updated successfully, but these errors were encountered:

edsantiago · 2022-05-16T16:55:18Z

This is flaking a lot lately. Can a network person PTAL.

vrothberg · 2022-05-18T10:23:16Z

@flouthoc @Luap99 in case you have/find time.

flouthoc · 2022-05-20T10:13:59Z

I am unable to reproduce this flake on host not entirely sure but I think i know why this happens on CI one reason it below but there can be more reasons for it.

One root cause could be that netavark spawns aardvark-dns using Command::spawn() and never waits for it especially in case of systemd-run. Usually it does not makes any difference since by the time podman actually runs command on container , aardvark-dns should be ready but on CI env on certain runs we are surpassing this window.

I think adding a backoff on netavark end should help here. Something like this: #14173

Above case is specially for the test [It] podman run check dnsname plugin with Netavark where this can happen more often.

flouthoc · 2022-05-20T10:28:40Z

PS: Also while running tests on host I am only running test/e2e/run_networking_test.go test/e2e/run_aardvark_test.go with commons assuming that tests are isolated with each other they share nothing.

edsantiago · 2022-06-13T21:19:32Z

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

fedora-36 : int podman fedora-36 root host
fedora-36 : int remote fedora-36 root host [remote]
- PR Bump github.com/containernetworking/cni from 1.1.0 to 1.1.1 #14455

This is one the redesign PR for aardvark-dns and netavark. Design proposes that double-forking will happen at aardvark-end instead of netavark. Redesign proposal * Aardvark will invoke server on the child process by double-forking. * Parent waits for child to show up and verify against a dummy DNS query to check if server is running. * Exit parent on success and deatch child. * Calling process will wait for aardvark's parent process to return. * On successful return from parent it will be assumed that aardvark is running properly One new design is implemented and merged and netavark starts using this it should close * containers/podman#14173 * containers/podman#14171 Signed-off-by: Aditya R <[email protected]>

flouthoc · 2022-06-14T09:52:05Z

There is a bigger design change under discussion which should help in this as well as some other issue, please track it here. However it will take some time to get merged: containers/aardvark-dns#148

This is one the redesign PR for aardvark-dns and netavark. Design proposes that double-forking will happen at aardvark-end instead of netavark. Redesign proposal * Aardvark will invoke server on the child process by double-forking. * Parent waits for child to show up and verify against a dummy DNS query to check if server is running. * Exit parent on success and deatch child. * Calling process will wait for aardvark's parent process to return. * On successful return from parent it will be assumed that aardvark is running properly One new design is implemented and merged and netavark starts using this it should close * containers/podman#14173 * containers/podman#14171 Signed-off-by: Aditya R <[email protected]>

This is one the redesign PR for aardvark-dns and netavark. Design proposes that forking will happen at aardvark-end instead of netavark and aardvark will verify if servers are up before parent goes away. Redesign proposal * Aardvark will invoke server on the child process by forking. * Parent waits for child to show up and verify against a dummy DNS query to check if server is running. * Exit parent on success and deatch child. * Calling process will wait for aardvark's parent process to return. * On successful return from parent it will be assumed that aardvark is running properly One new design is implemented and merged and netavark starts using this it should close * containers/podman#14173 * containers/podman#14171 Signed-off-by: Aditya R <[email protected]>

Luap99 · 2022-07-04T14:11:48Z

The netavark/aardvark changes are fine but I don't think it is required to fix this flake in CI. The test already "tries" to implement a back off logic:

podman/test/e2e/common_test.go

Lines 1045 to 1062 in 6dffa45

    
           func digShort(container, lookupName string, matchNames []string, p *PodmanTestIntegration) { 
        
           	digInterval := time.Millisecond * 250 
        
           	for i := 0; i < 6; i++ { 
        
           		time.Sleep(digInterval * time.Duration(i)) 
        
           		dig := p.Podman([]string{"exec", container, "dig", "+short", lookupName}) 
        
           		dig.WaitWithDefaultTimeout() 
        
           		if dig.ExitCode() == 0 { 
        
           			output := dig.OutputToString() 
        
           			Expect(output).To(MatchRegexp(IPRegex)) 
        
           			for _, name := range matchNames { 
        
           				Expect(output).To(Equal(name)) 
        
           			} 
        
           			// success 
        
           			return 
        
           		} 
        
           	} 
        
           	Fail("dns is not responding") 
        
           }

This does not work because dig always returns exit code 0 even when there is no match.

The retry logic in digshort() did not work because dig always exits with 0 even when the domain name is not found. To make it work we have to check the standard output. We work on fixing the underlying issue in aardvark/netavark but this will take more time. Fixes containers#14173 Fixes containers#14171 Signed-off-by: Paul Holzinger <[email protected]>

flouthoc · 2022-07-04T15:23:11Z

Netavark/aardvark change is for more scenarios like #14356 (comment) i am not able to collect all the reference but it was reported few times that on the first container run dns resolve failed intermittently. But we can close these issues early if dig output match is fixed.

edsantiago · 2022-07-07T12:25:42Z

I'm still seeing flakes on this test, and I'm somewhat sure that it's after the "fix" merged. Error message seems different:

dns is not responding

Shall I file a new issue?

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

fedora-36 : int podman fedora-36 root host
- PR use c/common code for resize and CopyDetachable #14841

Luap99 · 2022-07-07T12:29:19Z

Yeah something is wrong, I have to add more debugging information but at this point it does not look like a race. There should be more than enough timout.

This is one the redesign PR for aardvark-dns and netavark. Design proposes that forking will happen at aardvark-end instead of netavark and aardvark will verify if servers are up before parent goes away. Redesign proposal * Aardvark will invoke server on the child process by forking. * Parent waits for child to show up and verify against a dummy DNS query to check if server is running. * Exit parent on success and deatch child. * Calling process will wait for aardvark's parent process to return. * On successful return from parent it will be assumed that aardvark is running properly One new design is implemented and merged and netavark starts using this it should close * containers/podman#14173 * containers/podman#14171 Signed-off-by: Aditya R <[email protected]>

edsantiago · 2022-07-14T16:36:56Z

New set of symptoms in remote f36 root:

Execing /usr/sbin/ip netns add XVlBzgbaiCMR

Cannot create namespace file "/var/run/netns/XVlBzgbaiCMR": File exists

The surprising thing about this is that a lot of tests fail, and all of them have the same presumably-random string. I would expect each test to have a unique string. Whoever groks the random-netns-name code, that might be a good starting place to look.

edsantiago · 2022-07-14T16:37:49Z

Renaming & repurposing this issue as a general place to log *vark flakes

Luap99 · 2022-07-14T16:41:51Z

New set of symptoms in remote f36 root:
Execing /usr/sbin/ip netns add XVlBzgbaiCMR

Cannot create namespace file "/var/run/netns/XVlBzgbaiCMR": File exists
The surprising thing about this is that a lot of tests fail, and all of them have the same presumably-random string. I would expect each test to have a unique string. Whoever groks the random-netns-name code, that might be a good starting place to look.

This should not effect netavark/aardvark at all, using a custom net namespace will never call into the network backend.

edsantiago · 2022-07-18T22:27:39Z

Another one, this time podman (non-remote) f36 root, same "file exists" symptom

github-actions · 2022-08-18T00:07:27Z

A friendly reminder that this issue had no activity for 30 days.

edsantiago · 2023-01-09T18:35:22Z

Last seen November 10. I will assume this is fixed.

edsantiago added the flakes Flakes from Continuous Integration label May 9, 2022

flouthoc mentioned this issue May 20, 2022

dns: wait for aardvark server just after spawning aardvark-dns. containers/netavark#300

Closed

flouthoc mentioned this issue Jun 14, 2022

main: run aardvark as a daemon via forking and maintain its partial daemon like nature. containers/aardvark-dns#148

Merged

flouthoc mentioned this issue Jun 15, 2022

netavark, dns: don't double-fork aardvark instead wait for the aardvark process to return. containers/netavark#308

Merged

Luap99 mentioned this issue Jul 4, 2022

fix flake in aardvark tests #14822

Merged

openshift-ci bot closed this as completed in #14822 Jul 4, 2022

Luap99 reopened this Jul 7, 2022

edsantiago changed the title ~~Aardvark Test 2: Two containers, same subnet~~ Aardvark/Netavark flakes Jul 14, 2022

edsantiago mentioned this issue Jul 14, 2022

Netavark network with multiple aliases #14172

Closed

github-actions bot added the stale-issue label Aug 18, 2022

edsantiago mentioned this issue Sep 29, 2022

CI Flake: podman system prune networks: expected string length 1 #15990

Closed

Luap99 mentioned this issue Oct 24, 2022

Netavark: userns: nc: bad address 'nc-server.dns.podman' #16272

Closed

edsantiago closed this as completed Jan 9, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 5, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aardvark/Netavark flakes #14173

Aardvark/Netavark flakes #14173

edsantiago commented May 9, 2022

edsantiago commented May 16, 2022

vrothberg commented May 18, 2022

flouthoc commented May 20, 2022 •

edited

Loading

flouthoc commented May 20, 2022

edsantiago commented Jun 13, 2022

flouthoc commented Jun 14, 2022

Luap99 commented Jul 4, 2022

flouthoc commented Jul 4, 2022

edsantiago commented Jul 7, 2022

Luap99 commented Jul 7, 2022

edsantiago commented Jul 14, 2022

edsantiago commented Jul 14, 2022

Luap99 commented Jul 14, 2022

edsantiago commented Jul 18, 2022

github-actions bot commented Aug 18, 2022

edsantiago commented Jan 9, 2023

Aardvark/Netavark flakes #14173

Aardvark/Netavark flakes #14173

Comments

edsantiago commented May 9, 2022

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

edsantiago commented May 16, 2022

vrothberg commented May 18, 2022

flouthoc commented May 20, 2022 • edited Loading

flouthoc commented May 20, 2022

edsantiago commented Jun 13, 2022

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

flouthoc commented Jun 14, 2022

Luap99 commented Jul 4, 2022

flouthoc commented Jul 4, 2022

edsantiago commented Jul 7, 2022

Podman run networking [It] Aardvark Test 2: Two containers, same subnet

Luap99 commented Jul 7, 2022

edsantiago commented Jul 14, 2022

edsantiago commented Jul 14, 2022

Luap99 commented Jul 14, 2022

edsantiago commented Jul 18, 2022

github-actions bot commented Aug 18, 2022

edsantiago commented Jan 9, 2023

flouthoc commented May 20, 2022 •

edited

Loading