[main] Cirrus: Update VM Images for 4.0 release

[cevich]

This is to ensure VM images for CI, which contain the
intended dependency versions to support the podman
4.0 release.

Ref: containers/automation_images#114

[cevich]

I'm trying to update our VM images, but for some reason checkpoint test (among others) aren't happy just on F34. @adrianreber any idea what's breaking in the checkpoint tests? Do we need package updates in F34 maybe?

[cevich]

@adrianreber un-ping. I think the problems may have been caused by premature/unintentional introduction of netavark & aardvark-dns packages on F35 for all tests.

[cevich]

@adrianreber re-ping 😢 F34 criu is breaking with the latest updates. @edsantiago IIRC you touched these tests recently (disabled/enabled them), no? Do any of the errors in those logs ring any bells?

Since this is only affecting F34, I'm thinking it must be due to some package available in F35 (possible from updates-testing) that's necessary.

[edsantiago]

Hm. The recent criu errors were due to rawhide kernel new-stuff. The errors I see here are all:

Error: cannot define the pod as the cgroup parent at the same time as joining the infra container's cgroupNS: invalid argument

...and they're happening in container create, before even thinking of checkpoint. This is not a criu problem.

Link to annotated log, which is infinitely better than the cirrus page.

[cevich]

The recent criu errors were due to rawhide kernel new-stuff

Oh okay, so this is something different then.

before even thinking of checkpoint

There are some instances which do fail after criu:

Error: pod XYZ does not share the <something> namespace

But that sounds like it could be related to the error you pointed out. The packages on both new VM flavors are basically the same crun-1.4.2-1 and criu-3.16.1-2. Hmmm 😕

Edit: F35 annotated log ref.

[edsantiago]

The packages on both new VM flavors are basically the same crun-1.4.2-1 and criu-3.16.1-2.

This is f34, so runc not crun, right? I see one runc error in the log, and we've been having runc-related failures in RHEL recently, so I think there's something in runc that broke recently. Can you compare if this runc == previous runc?

[cevich]

This is f34, so runc not crun, right?

Oh right, I completely forgot, thanks!

I see one runc error in the log, and we've been having runc-related failures in RHEL recently, so I think there's something in runc that broke recently. Can you compare if this runc == previous runc?

Grabbing a run from main ("old" VM images), I see runc-1.0.2-2.fc34-x86_64. The new F34 image has runc-1.1.0-1.fc34-x86_64.

[edsantiago]

Well, that's really interesting because some of the discussion on the RHEL 8.6 gating-test failures is zeroing in on the move from runc 1.0 to 1.1.

Are you able to rebuild the VM using runc 1.0, and just sweep 1.1 under the carpet, la la la?

[adrianreber]

The broken checkpoint/restore tests seem all to be about checkpointing out of and restoring into another pod. That seems broken with runc 1.1.

There is also something wrong with setting up a pod that shares a cgroup namespace.

         # podman [options] --network-backend cni --storage-driver vfs pod create --share cgroup,ipc,net,uts,pid
         Error: cannot define the pod as the cgroup parent at the same time as joining the infra container's cgroupNS: invalid argument

Seems like this triggers a couple of different errors.

[adrianreber]

Overall none of these checkpoint/restore errors seems to be a CRIU problem, but all of them seem to be triggered by changes in Podman and runc as far as I can tell. Strange that CI did not catch it earlier.

[adrianreber]

The cgroup error is related to #12930. But CI was using a runc version which did not support certain CRIU features and so the tests which break here were never run. Similar for the network namespace related errors.

#13214 has fixes for the checkpoint/restore related errors in this PR.

[cevich]

Thanks for all the analysis and PR-work @adrianreber I sincerely appreciate it. I can't tell from your comments, but hopefully these efforts will help ensure stability downstream for runc users.

[cevich]

Rebased on #13214 and force-pushed.

[cevich]

@adrianreber your PR does seem to help, though curiously, not in the "remote" case. Annotated results:
https://storage.googleapis.com/cirrus-ci-6707778565701632-fcae48/artifacts/containers/podman/6090737132503040/html/int-remote-fedora-34-root-host.log.html

[cevich]

Are you able to rebuild the VM using runc 1.0, and just sweep 1.1 under the carpet, la la la?

Yes of course @edsantiago if that's what's needed, but I'm not sure I'd call it a solution - we'd not be testing in an environment representative of what users are using. The "proper" way to handle it for both tests and users, is to (somehow) roll back the version released in Fedora (but which I believe takes weeks 😖). Of course, I am assuming that the issue also affects the F34-stock podman 3.4 - may/not be true.

[cevich]

Rebased/force-pushed.

[cevich]

This should be ready to go (The v4.0 PR w/ same images already merged).

[edsantiago]

/lgtm

[rhatdan]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[edsantiago]

New task map, for completeness's sake