Epic: Fix Flaky tests

[thaJeztah]

Fix Flaky tests

There's a number of tests that are intermittently failing. Given that a whole
test run can take up to three hours on ARM, these tests are giving us serious
headaches.

There's a list of tests that are known to be flaky.
If you discover additional tests that are possibly flaky; open a new issue,
and use Flaky test: in the title. Also ping a maintainer to have the
"flaky test" label added and have it added to the list below.

Test duration

There's also a related issue for reducing the total duration of our tests;
Optimize slow bottleneck tests
that we could use some help on.

If you want to work solving one of these tests, leave a comment, and we add
your name to the list below!

• DockerTrustSuite.SetUpTest Flaky test: DockerTrustSuite.SetUpTest (more apparent on ARM?) #19393 (@thaJeztah) PR Fix DockerTrustSuite SetUpTest #19471
• TestRunSlowStdoutConsumer Flaky test: TestRunSlowStdoutConsumer #19472
• TestDockerNetworkHostModeUngracefulDaemonRestart (on experimental) Flaky test: TestDockerNetworkHostModeUngracefulDaemonRestart #19368 PR Flaky test: Test Docker daemon ungracefully restart #19653 (@wenchma) not fully resolved (gccgo failing) PR Fix flaky test, TestDockerNetworkHostModeUngracefulDaemonRestart #20246
• Panic in 'TestEventsImageUntagDelete' (on experimental) Flaky test (?): Panic in 'TestEventsImageUntagDelete' (on experimental) #19207 PR Fix channel closing race in event tests. #19519 (@calavera)
• TestGetContainerStatsRmRunning Flaky test: TestGetContainerStatsRmRunning #18679 (@WeiZhang555) PR Fix flaky test TestGetContainerStatsRmRunning #19570
• TestStatsAllNewContainersAdded Flaky test: TestStatsAllNewContainersAdded #18672
• TestExecTTY Flaky test: TestExecTTY #18544 / TestExec* are racy (on userns) tests: userns tests TestExec* are racy #17473
• DockerSuite.TestLinksHostsFilesInject (on userns) Flaky test: DockerSuite.TestLinksHostsFilesInject #17226
• DockerSuite.TestEventsOOMDisableFalse Flaky test: DockerSuite.TestEventsOOMDisableFalse #17215 (@clnperez)
• DockerHubPullSuite.TestPullAllTagsFromCentralRegistry Flaky test: DockerHubPullSuite.TestPullAllTagsFromCentralRegistry #19904
• DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts Flaky test: DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts #17214 (@aaronlehmann) PR Improve resilience of TestPullFromCentralRegistryImplicitRefParts #20411
• DockerSuite.TestLinksUpdateOnRestart on userns Flaky Test : TestLinksUpdateOnRestart on userns #20595

[thaJeztah]

ping @vdemeester, our master of tests! 🐸

[WeiZhang555]

Already pushed a fix for #18679, PR: #19570

[wenchma]

@thaJeztah I submitted a PR: #19653
fix TestDockerNetworkHostModeUngracefulDaemonRestart (on experimental) #19368

[clnperez]

@thaJeztah, I just saw TestRunSeccompAllowAptKey fail, and I feel like I've seen other tests fail with similar (or identical) output:

20:40:48 gpg: requesting key F6B0FC61 from hkp server p80.pool.sks-keyservers.net
20:40:48 gpgkeys: key E871F18B51E0147C77796AC81196BA81F6B0FC61 can't be retrieved
20:40:48 gpg: no valid OpenPGP data found.
20:40:48 gpg: Total number processed: 0
20:40:48 : exit status 2

Does that look familiar to you?

[thaJeztah]

@clnperez I think that's because the key server is down. Unfortunately we've seen that happening regularly, so we may need to think of using other key servers, or whatever.

[vdemeester]

Gonna close this one as the original list is fixed and we track the other individually for now 👼