Cirrus: Implement containerized system testing

[cevich]

Fixes #8714 (comment)

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cevich
To complete the pull request process, please assign umohnani8 after the PR has been reviewed.
You can assign the PR to them by writing /assign @umohnani8 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rhatdan]

LGTM

[edsantiago]

This is going to be a tricky one. The .cirrus.yml changes LGTM, with one probably-trivial question. The hard part is going to be updating the container image to un-comment-out the mount_program.*fuse-overlayfs line in /etc/containers/storage.conf and then forcing --cgroup-manager=cgroupfs (to avoid "Error: cannot open sd-bus: No such file or directory: OCI runtime command not found error"). Then figuring out how to deal with "write unixgram @aefbb->/run/systemd/journal/socket: sendmsg: no such file or directory". And then, of course, running tests and seeing what fails. This is going to need to wait until next year.

[edsantiago]

Trivial low-priority question: is this *branch deliberate? In looking at the Cirrus graph, I notice that this is the only one of the right-hand boxes (system tests) that has it set; all the others -- rootless, local, remote -- skip on CI:DOCS but not on BRANCH. I don't see a strong reason to go either way, I'm just asking because it's a curious inconsistency.

[edsantiago]

Is it just me, or is the graph beginning to look like a... um... cetacean?

[cevich]

......This is going to need to wait until next year.

Or I COULD just close this PR 😁

TBH, this doesn't sound too awful. The containerized tests are executed through a re-run of setup_environment.sh and runner.sh inside the container. The hardest part will be me writing all the comments and not spelling something wrong 😉

is this *branch deliberate

Yes, I stuck it there because the containerized integration tests have it. All it does is bypass containerized testing when a PR merges or a branch is (otherwise) pushed. Long ago this was "okay-ed", since these tests are low-risk/low-value at the branch-level (assuming they passed PR testing). TBH, we probably could skip a lot more testing on branches, that would improve the cron-based fail-rate as well as make Matt's life easier come release-time.

Is it just me, or is the graph beginning to look like a... um... cetacean?

Are you're ocular orbs spherical today? Clearly it's a Elephant standing in line at the McDonalds drive-through, during a McRib BOGO sale*

* By one get one only valid when meal-deal is super-sized, and includes a super-extra-large-gulp milkshake

[cevich]

Might as well get the ball rolling while waiting for other test results...

[cevich]

...hmmm, well that's no good: gopath_cache is failing to restore /var/tmp/go, but the keys are the correct values from the corresponding build jobs 🤔

[cevich]

level=error msg="unable to write pod event: \"write unixgram @002ca->/run/systemd/journal/socket: sendmsg: no such file or directory\"

@edsantiago this seems to be the main item left failing. Could this be fixed by a bind-mount of that socket from the host, conditional on containerized execution of the system-tests??

[cevich]

now Failed to add conmon to systemd sandbox cgroup: dial unix /run/systemd/private: connect: no such file or directory

Hmmm...welp, I dislike sharing something called 'private', but let's try the same medicine there...

[cevich]

...hmmm, that doesn't appear to work. With JOURNALD_ARGS="-v /dev/log:/dev/log -v /var/run/systemd/journal/socket:/var/run/systemd/journal/socket -v /run/systemd/private:/run/systemd/private" now we get:

level=warning msg="Failed to add conmon to systemd sandbox cgroup: Process with ID 65992 does not exist."
[+0205s] # Error: cannot open sd-bus: No such file or directory: OCI not found

[rhatdan]

I think you could change the events-backend from journald to file. I believe the issue is attempts to connect to systemd socket from inside of a container.

[vrothberg]

Is there a way to execute system tests without waiting for others to complete before?

[edsantiago]

No. @cevich started on a mechanism, #8754, which I believe adds undesirable complexity to the CI yaml for too little return.

[cevich]

I think you could change the events-backend from journald to file.

Thanks @rhatdan trying it with none to see if there is any affect...

(I'm not sure where file goes, might be something we want to grab as an artifact for debugging).

[cevich]

Is there a way to execute system tests without waiting for others to complete before?

The logic is way to complex since it also needs to not break [CI:DOCS]. For future ref. @vrothberg: Temporarily adding skip: $CI == $CI to all/most of the tasks is the easiest way. Like in the 'DO NOT MERGE' commit I made here.

[vrothberg]

Thanks!

[cevich]

@edsantiago would it be a PITA to modify the system-tests such that:

When $container is defined non-empty, always use podman --events-backend=file for every command?

I'm pretty sure the setting of $container is podman (maybe docker) built-in behavior, so it should be a reliable indicator.

[cevich]

(f4c89a6 is my naive attempt)

[edsantiago]

That's not going to work with podman-remote. Don't even bother.

[cevich]

🤷‍♂️ CI is already "bothering"...but it told me it doesn't mind 😄

[cevich]

@edsantiago hmmm, maybe was worth. Even with that option I still see errors like:

/var/tmp/go/src/github.com/containers/podman/bin/podman --events-backend=file --events-backend=file run --name running-u8m0JU3ooEKMsupVpGj6vjbdi7zgjI -d quay.io/libpod/testimage:20200929 top
[+0202s] # time="2020-12-18T14:21:25Z" level=warning msg="Failed to add conmon to systemd sandbox cgroup: dial unix /run/systemd/private: connect: no such file or directory"
[+0202s] # Error: cannot open sd-bus: No such file or directory: OCI not found

So it seems we're going to need a much more complicated solution 😓

[cevich]

Giving up on this, it's too complicated to be properly implemented anytime soon.