cirrus,ci: default to overlay if using vfs

[flouthoc]

In debian environment we are hitting an edge-case where older buildah version is not compatible with newer podman version because both of them are using different storage driver.

I.e

• Podmand defaults to native overlay.
• Older buildah version defaults to vfs.

See discussions below for more details

• fix(deps): update common, image, and storage deps #18510 (comment)
• DO NOT MERGE: buildah vendor treadmill #13808

Does this PR introduce a user-facing change?

cirrus,ci: default to `overlay` for debian env

[flouthoc]

@Luap99 @cevich @edsantiago @nalind PTAL

[edsantiago]

This is frighteningly complicated and duplicatey. Can it be done like this instead?

diff --git a/contrib/cirrus/setup_environment.sh b/contrib/cirrus/setup_environment.sh
index f3068787c..b3da90147 100755
--- a/contrib/cirrus/setup_environment.sh
+++ b/contrib/cirrus/setup_environment.sh
@@ -100,6 +100,17 @@ esac
 # shellcheck disable=SC2154
 printf "[engine]\ndatabase_backend=\"$CI_DESIRED_DATABASE\"\n" > /etc/containers/containers.conf.d/92-db.conf
 
+# Blah blah include a FIXME comment to reevaluate after (e.g.) 2023-12-31
+buildah_storage=$(buildah info --format '{{.store.GraphDriverName}}')
+if [[ "$buildah_storage" != "overlay" ]]; then
+    conf=/etc/containers/storage.conf
+    if [[ -e $conf ]]; then
+        die "FATAL! INTERNAL ERROR! Cannot override $conf"
+    fi
+    msg "Overriding $conf, setting overlay (was: $buildah_storage)"
+    printf "...driver=overlay\n" >$conf
+fi
+
 if ((CONTAINER==0)); then  # Not yet running inside a container
     # Discovered reemergence of BFQ scheduler bug in kernel 5.8.12-200
     # which causes a kernel panic when system is under heavy I/O load.

[flouthoc]

@edsantiago Thanks looks much better, I have a doubt about if [[ -e $conf ]]; then check, shouldn't we just create it instead of breaking here ?

Also I previously tried setting similar check in setup_environment.sh before but I was getting weird build errors, maybe I was trying it in the wrong place. I hope it should pass this time.

[edsantiago]

/etc/containers/storage.conf should never exist on a CI system. If it does, that means someone has done something special to change defaults -- and if we overwrite it, we will be clobbering that and changing expectations. I believe we should panic early and loudly if this happens.

PS f38 flake was unlinkat/EBUSY. I restarted.

[edsantiago]

Why are all the other things (runroot, graphroot) needed?

And, please reinstate the check for existing storage.conf, thanks.

[flouthoc]

| Why are all the other things (runroot, graphroot) needed?

@edsantiago When default file is selected and runroot is empty we hit error https://github.com/containers/storage/blob/main/types/options.go#L159 here, this needs to be fixed on c/storage I'll open a different PR on c/storage for this.

[cevich]

Since this only applies to debian (and likely, only for a short while), consider moving this block under the debian-host setup case - around line 142-147.

Also, I'm guessing this only impacts a very specific and known range of buildah versions. Perhaps consider adding a TODO comment indicating this change may be removed once buildah version XYZ is present in CI VM images.

[flouthoc]

@cevich done added a note.

[edsantiago]

LGTM but let's wait for CI on #13808

[edsantiago]

CI still failing in what looks like exactly the same way, in buildah treadmill, even with your PR cherrypicked. I'll try to rerun in terminal and figure out what happened.

[edsantiago]

When run as root, this always returns overlay. I'm sorry, I didn't know that. I guess it's OK to revert to checking for "debian", unless someone can think of a better test.

[flouthoc]

@edsantiago I reverted to old commit, could you cherry-pick this one ?

[edsantiago]

gah, no, that one is really not acceptable. I am testing a simpler fix in #13808 right now.

[edsantiago]

just stay tuned. Once #13808 passes I will comment here. (Probably very, very late your time, so just check back in tomorrow)

[edsantiago]

Good news, bad news.

Good news: debian rootless passed.

Bad news: new failure seen in remote debian root and also remote f38 aarch64 root:

<+009ms> # # podman-remote --url unix:/tmp/podman_tmp_9HtW pod rm c3cc31049b088d86ef22ad5b279ce5c1d01b88887116bc9f6486b9b0a28f2080
<+104ms> # Error: 2 errors occurred:
         # 	* removing container 0c7966924a988d4e00f2585da31627a0db0621e6fbd6531cea4d02779e3bcfe2 from pod c3cc31049b088d86ef22ad5b279ce5c1d01b88887116bc9f6486b9b0a28f2080: cleaning up storage: removing container 0c7966924a988d4e00f2585da31627a0db0621e6fbd6531cea4d02779e3bcfe2 root filesystem: unmounting "/var/lib/containers/storage/overlay/4e8aaf7a108d0bf91f873379b4558162620f9e4ffaca942d06ef4d40778456bc/merged": invalid argument
         # 	* removing container 55ade4c16aa0fffb758fc893be4d24c8fdaabcde14eec89df7885b42c4b03881 from pod c3cc31049b088d86ef22ad5b279ce5c1d01b88887116bc9f6486b9b0a28f2080: cleaning up storage: removing container 55ade4c16aa0fffb758fc893be4d24c8fdaabcde14eec89df7885b42c4b03881 root filesystem: unmounting "/var/lib/containers/storage/overlay/1ef8c38e52bd1d345801a0a4a7b2e0631b85965881be472dcb6db0afd6c3a65b/merged": invalid argument

After that, everything is hosed: nothing ever works again, all tests fail.

It looks a heck of a lot like unlinkat/EBUSY, except this one is unmounting/EINVAL. I'm debating right now if I should re-run them. UPDATE: yep, they were flakes. Flakes I've never seen before, hence, probably a problem with the new vendoring and probably something we'll be seeing soon in podman.

[edsantiago]

@flouthoc here's a patch you can use. Please apply it exactly precisely 100% as it is there, no edits. That will guarantee that it will pass (modulo flakes) in the treadmill.

[Luap99]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, flouthoc, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Luap99,edsantiago,flouthoc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[edsantiago]

/lgtm

[Luap99]

/hold cancel