Cirrus: Update VM Images; Both Fedora and Ubuntu "prior" flavors run with CGroupsV1 & runc

[cevich]

TODO: Open issues for disabled tests

• Disable blkio.weight test on Ubuntu (Error: opening file io.bfq.weight for writing: Permission denied #8737)
• Disable pod stats tests in containerized Fedora w/ CGroupsV1 (Error: unable to load cgroup at /machine.slice/.../: cgroup deleted #8732)
• Disable remote pod stats on net=host post (Ignore containers.conf sysctls when sharing namespaces #8685)
• Disable rootless pod stats tests w/ CgroupV1 (Error: pod stats is not supported in rootless mode without cgroups v2 #8734)
• Disable F32 rootless system testing (Cirrus: Fix Disabled F32 rootless system testing #8729)

[cevich]

Status: Ubuntu 20.10 image's runc version is too damn old.

[rhatdan]

@cevich Can we switch to using crun?

[cevich]

@rhatdan Yes, I believe we can...or at least make an attempt. I was actually looking into this last night, it seems that systemd in Ubuntu enables cgroupsv2 the same way as Fedora (just not by default).

@mheon @baude @giuseppe Q: Assuming it can be done, are all'y'all okay with the only testing of runc, occurring on "prior" Ubuntu (20.04 LTS) VMs? This implies we would drop testing with runc in the future, when the 20.04 VMs go out of style.

[rhatdan]

Well maybe at some point someone will update ubuntu version of runc. We could also switch one of the fedora tests to use runc.

[cevich]

We could also switch one of the fedora tests to use runc.

This is true...and based on past experience, much easier to obtain necessary package updates.

[cevich]

Oof! This is the best I can do, I really need help sorting through why/fixing the 11x F32 (CGv1) rootless tests are failing.

[edsantiago]

Thank you! #8726 is halfway through CI, maybe an hour left. I'm monitoring it closely to restart flakes quickly; will ping you once it's merged.

[cevich]

maybe an hour left.

Gee wiz, I've been saying the exact same thing on this PR...for about a month 😄

If only it were that simple...keep in mind THIS PR will then need to be rebased, and rerun all the tests again...with fingers crossed nothing got missed. I dunno if the team wants to wait that long or not.

[edsantiago]

#8726 is merged!

[edsantiago]

Tricky. I'm on it.

[edsantiago]

Fix (tested on f32 cgroupsv1 VM; also on my f32 cgroupsv2 laptop):

diff --git a/test/system/600-completion.bats b/test/system/600-completion.bats
index 1e43cdc41..d1244c7d7 100644
--- a/test/system/600-completion.bats
+++ b/test/system/600-completion.bats
@@ -8,6 +8,17 @@

 load helpers

+# Returns true if we are able to podman-pause
+function _can_pause() {
+    # Even though we're just trying completion, not an actual unpause,
+    # podman barfs with:
+    #    Error: unpause is not supported for cgroupv1 rootless containers
+    if is_rootless && is_cgroupsv1; then
+        return 1
+    fi
+    return 0
+}
+
 function check_shell_completion() {
     local count=0

@@ -70,8 +81,13 @@ function check_shell_completion() {
                     ;;

                 *CONTAINER*)
+                    # podman unpause fails early on rootless cgroupsv1
+                    if [[ $cmd = "unpause" ]] && ! _can_pause; then
+                        continue 2
+                    fi
+
                     run_completion "$@" $cmd "${extra_args[@]}" ""
-                    is "$output" ".*-$random_container_name${nl}" "Found expected container in suggestions"
+                    is "$output" ".*-$random_container_name${nl}" "Found expected container in suggestions for '$cmd'"

                     match=true
                     # resume
@@ -212,7 +228,9 @@ function _check_completion_end() {
     run_podman create --name created-$random_container_name $IMAGE
     run_podman run --name running-$random_container_name -d $IMAGE top
     run_podman run --name pause-$random_container_name -d $IMAGE top
-    run_podman pause pause-$random_container_name
+    if _can_pause; then
+        run_podman pause pause-$random_container_name
+    fi
     run_podman run --name exited-$random_container_name -d $IMAGE echo exited

     # create pods for each state

downloadable diff

@Luap99 PTAL

[cevich]

@edsantiago tip, if you're using the WebUI: you can drag-n-drop files directly into the comment box. It will upload and automatically splat in the necessary markdown.

[edsantiago]

I'm a crusty old UNIX user from the 1980s. I've heard of drag & drop, but it requires newfangled window managers or something like that which I've never bothered upgrading to. Now get off my lawn!

[cevich]

Now get off my lawn!

Nownow ---> I <---- also grew up in the 80's, but with M$ DOS...shudder. Hell, I only graduated (downgraded) from a flip-phone just a few years ago. If I can learn to clicky-clicky...so can you 😅

The truly magical part is the uploading...It seems to go into some special storage location associated with the repository, but which is otherwise inaccessible (in the WebUI). Like this:

somefile.txt

[cevich]

@edsantiago we got really lucky here. Something just fell over on the Cirrus-CI side at the end of the run. The "Success" task (a noop) normally starts within seconds, it took a very slow 1:25 to get going. Anyway, thanks for your help clearing up the system-testing issue. I recommend we don't push our luck any further before the hollow-days break. No need to subject people to Ubuntu 19, or hold up Dan's PR any longer.

[edsantiago]

/lgtm
/hold

(Disclosure: I did the vast majority of review yesterday; right now I just reviewed the diffs between this and the prior PR revision). This is a large enough PR that I'd like one more pair of eyes before merging, hence the hold.

Nice work @cevich . Thank you for your patience with my intransigence.

[rhatdan]

/hold cancel
LGTM

[rhatdan]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment