remote e2e: speed up populating and loading cache

[vrothberg]

The remote e2e tests run significantly slower than the local ones.
In some runs I have seen a factor of two. Among other things, I
suspect the continuous loading of the cache to be a bottleneck as
(un)taring is CPU-intensive.

Speed up the aforementioned bottleneck by using the local podman binary
to populate and load the cache to workaround these more expensive remote
operations.

Signed-off-by: Valentin Rothberg [email protected]

Does this PR introduce a user-facing change?

None

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vrothberg]

Now, I am curious how much we'd gain.

[vrothberg]

Now, I am curious how much we'd gain.

Nothing ...

[vrothberg]

@baude, did I misread the code and the e2e tests already use the local podman for that?

[edsantiago]

cirrus-pr-timing:

type	distro	user	local	remote	container
int	fedora-36	root	30:01	48:17	29:23
int	ubuntu-2204	root	30:27	52:12
int	fedora-36	rootless	32:28
int	ubuntu-2204	rootless	30:51
sys	fedora-36	root	01:08	01:13
sys	fedora-36-aarch64	root	01:26	01:19
sys	ubuntu-2204	root	01:16	01:15
sys	fedora-36	rootless	01:17	01:25
sys	ubuntu-2204	rootless	01:18
bud	fedora-36	root	01:16	01:07

EDIT: I obviously have a bug in my script. Nothing runs in one minute.

[vrothberg]

@edsantiago I think your scripts are accurate. Cirrus states 1min https://github.com/containers/podman/pull/15266/checks?check_run_id=7765843970

[edsantiago]

Oh - duh, it's because of the test-skipper. Whew. Yeah, results are good. Only int tests ran, nothing else.

[vrothberg]

@baude, did I misread the code and the e2e tests already use the local podman for that?

Indeed the remote tests already use the local one: https://github.com/containers/podman/blob/main/test/e2e/libpod_suite_remote_test.go#L154

[baude]

@vrothberg yes, they do. i verified yesterday. i dug quite a bit into this though and some really odd findings.