CI: Ubuntu 2004 is slow

[edsantiago]

I've updated my test-timings script to have it interpret system test results. Ubuntu is slow, especially 2004:

(All of these are remote, from #9556. It takes just a few seconds to get results for any PR, so just ask if you want others)

	fedora 33	ubuntu 2004	ubuntu 2010
20 podman run - basic tests	3.00	5.00	5.00
33 podman run --log-driver	4.00	9.00	5.00
48 podman ps --filter	2.00	7.00	4.00
51 podman stop --all	5.00	9.00	6.00
53 podman stop - can trap signal	5.00	9.00	6.00
65 podman cp file from host to container	19.00	76.00	26.00
66 podman cp file from container to host	15.00	95.00	25.00
67 podman cp dir from host to container	17.00	142.00	35.00
68 podman cp dir from container to host	14.00	107.00	28.00
69 podman cp file from host to container volume	1.00	34.00	6.00
70 podman cp file from host to container mount	2.00	29.00	6.00
71 podman cp * - wildcard copy multiple files from container to host	1.00	18.00	5.00
72 podman cp - will not recognize symlink pointing into host space	1.00	14.00	5.00
73 podman cp - will not expand globs in host space (#3829)	1.00	13.00	4.00
74 podman cp - will not expand wildcard	1.00	13.00	5.00
75 podman cp into container: weird symlink expansion	3.00	28.00	13.00
76 podman cp into a subdirectory matching GraphRoot	3.00	22.00	8.00
77 podman cp from stdin to container	13.00	41.00	24.00
78 podman cp from container to stdout	12.00	38.00	21.00
79 podman build - basic test	5.00	35.00	15.00
82 podman build - cache (#3920)	4.00	24.00	11.00
83 podman build - URLs	2.00	15.00	6.00
84 podman build - workdir, cmd, env, label	17.00	129.00	58.00
85 podman build - stdin test	4.00	17.00	14.00
86 podman build - does not gobble stdin that does not belong to it	2.00	7.00	7.00
87 podman build --layers test	6.00	13.00	16.00
88 build with copy-from referencing the base image	12.00	18.00	21.00
99 podman load - by image ID	2.00	5.00	6.00
100 podman load - by image name	3.00	6.00	6.00
102 podman load - multi-image archive	4.00	8.00	7.00
103 podman load - multi-image archive with redirect	4.00	8.00	7.00
105 podman kill - rejects invalid args	2.00	6.00	6.00
120 podman volume: exec/noexec	2.00	5.00	5.00
121 podman volume, implicit creation with run	4.00	7.00	6.00
123 podman volume prune	3.00	7.00	6.00
125 podman pod - communicating between pods	3.00	8.00	7.00
126 podman pod - communicating via /dev/shm	2.00	6.00	6.00
135 podman system df - with active containers and volumes	3.00	5.00	5.00
153 podman shell completion test	65.00	201.00	142.00

Someone with access to Ubuntu should probably do some manual runs of system tests, verify the slowness, then possibly instrument the tests in question to track down the cause.

[edsantiago]

Timing results from #9697:

type	distro	user	local	remote	container
int	fedora-33	root	32:47	39:55	32:00
int	ubuntu-2004	root	41:52	57:30
int	ubuntu-2010	root	42:24	48:51
int	fedora-33	rootless	34:38
sys	fedora-33	root	15:42	13:15
sys	ubuntu-2004	root	19:13	34:31
sys	ubuntu-2010	root	19:57	13:01
sys	fedora-33	rootless	15:55

From #9686:

type	distro	user	local	remote	container
int	fedora-33	root	29:26	46:22	29:25
int	ubuntu-2004	root	39:31	55:57
int	ubuntu-2010	root	37:03	49:57
int	fedora-33	rootless	32:13
sys	fedora-33	root	16:24	10:09
sys	ubuntu-2004	root	19:40	30:20
sys	ubuntu-2010	root	20:04	16:04
sys	fedora-33	rootless	15:40

From #9669 :

type	distro	user	local	remote	container
int	fedora-33	root	32:00	40:36	30:33
int	ubuntu-2004	root	40:37	01:00:22
int	ubuntu-2010	root	36:33	45:27
int	fedora-33	rootless	34:14
sys	fedora-33	root	16:00	11:33
sys	ubuntu-2004	root	18:55	28:14
sys	ubuntu-2010	root	19:19	14:21
sys	fedora-33	rootless	15:17

From #9652 :

type	distro	user	local	remote	container
int	fedora-33	root	32:30	46:21	34:22
int	ubuntu-2004	root	42:08	01:01:43
int	ubuntu-2010	root	40:51	48:33
int	fedora-33	rootless	34:48
sys	fedora-33	root	15:24	11:38
sys	ubuntu-2004	root	19:18	33:02
sys	ubuntu-2010	root	19:11	15:14
sys	fedora-33	rootless	15:32

DAE see a pattern with ubuntu 2004?

[edsantiago]

Thank you @baude! From #9970:

type	distro	user	local	remote	container
int	fedora-33	root	31:56	34:48	30:29
int	ubuntu-2004	root	29:07	32:12
int	ubuntu-2010	root	28:39	31:00
int	fedora-33	rootless	33:42
sys	fedora-33	root	17:43	10:53
sys	ubuntu-2004	root	17:40	10:56
sys	ubuntu-2010	root	17:50	10:37
sys	fedora-33	rootless	20:54

[rhatdan]

Does this mean we can close this issue?

[edsantiago]

I've run a few more timings. Yes, it looks like this is fixed. Thanks again @baude.