Experiment: cross-project testing

[martinpitt]

Let's presume we want to do cross-project testing and validate a bridge change against podman with the Testing Farm. See https://packit.dev/posts/testing-farm-triggering.

This PR breaks the pybridge, so it should break the podman runs if they really use the RPMs from this PR.

Triggered in cockpit-project/cockpit-podman#1345 . This worked as expected: The fedora-37 run passed, as that uses the C bridge. fedora-38 went kaboom.

That is useful for testing two project PRs which depend on each other. We can use this to validate a bridge change against podman/machines with an empty dummy PR and manual trigger. However, it is not sufficient for automating this testing, e.g. for running c-podman tests in podman PRs.

• packit: Build COPR for main commits cockpit-podman#1365

[martinpitt]

I discussed this a bit with the packit team, and in the end https://packit.dev/posts/testing-farm-triggering is not what I had in mind for automated reverse dependency testing:

• These need to be triggered on the "reverse dependency" side, they can't be triggered from the "source" side. So you'd always need some open PR in all reverse dependency projects and don't get the feedback where you need it.
• This cannot be set up in packit.yaml, and having to introduce workflows which send magic comments automatically is both annoying and noisy.

I think packit is the wrong level for this. I'll instead try to use TMT for that, and add the reverse dependency projects as additional plans. Preferably they could be constrained somewhat, like only running podman/machines when something in pkg/lib or the python bridge changes. https://packit.dev/posts/manual-triggering#triggering-test-jobs-based-on-labels-and-identifiers may help, although it doesn't fit perfectly.

This first approach did not work, it didn't try to run podman at all.

The second approach of adding a new plans/podman.fmf justs ran a single podman plan, instead of all three. Not surprising as I was including them as a plan, not something at the project level. Unfortunately tmt does not have a way to "import everything" from a remote repo, it has to duplicate the plans. teemtee/tmt#1770

However, this is still not right, as it tests the upstream main tests (72+git) against the version 71 which is in Fedora, see the tmt log. This causes a lot of faied tests even though they are supposed to work on F37. So either we run them against Fedora dist-git (although that can be behind as well for updates-proposed), or we somehow build cockpit-podman in a copr for cockpit PRs.

[martinpitt]

We talked this through in a meeting with @kkaarreell (tmt), @lachmanfrantisek and @lbarcziova (packit), and @thrix (TF). There is currently no good solution for downstream, due to not being able to map a particular rpm version to a dist-git ref:. But for upstream-to-upstream there is an idea:

• Configure reverse dependencies to continuously build their main branches into a COPR. The copr job has a supported_triggers: commit
• This isn't just useful for CI, but also allows users to install and the latest available main.
• Configure packit.yaml to add that COPR via tf_extra_params, see Providing additional Testing Farm artifacts examples. This ends up becoming a TF test request parameter in environments → artifacts

This is actually the more interesting part for me, as it provides a much tighter feedback loop ("shift left"). At some point we still need to sort out dependency gating in Fedora, though.

I'll set up that new experiment, thank you all!

This could run into https://issues.redhat.com/browse/TFT-2102 , so make sure to test this with multiple builds in the COPR.

[martinpitt]

Perfect, this works!

The revdeps run installs c-ws from the PR packit COPR and c-podman from the main-builds COPR, and the podman tests fail as expected on the KABOOM. See statuses on commit 037b72b

[martinpitt]

This is good enough for gating e.g. podman → cockpit-podman, or selinux-policy → cockpit, where we want to run this on every PR. So I'll send a PR to land this now, to have a reference for a blog post.

For cockpit → c-{podman,machines} we could optimize this further: the unit test could check if there are any changes to src/ (bridge or ws), and only then trigger revdeps manually. I'm not 100% sure if packit will see comments created by the GitHub default workflow token, as they can't trigger other workflows. In the worst case we could use the cockpituous token. Let's find out!

Update: See martinpitt#16 , this indeed doesn't work. 😢

[martinpitt]

I squeezed what I wanted out of this PR.