Cirrus: Update CI VM images to F37beta #15760

cevich · 2022-09-12T21:34:39Z

Fixes: #15349
Depends on: #15871 #15890
Image build PR: containers/automation_images#195

Does this PR introduce a user-facing change?

None

cevich · 2022-09-12T21:42:24Z

@umohnani8 this PR also stubs-in a spot for you to setup and run your kube tests. I added it to run after the system-tests, but am not married to the order if you want it to run sooner. However, since this PR brings in a new Fedora version as well, it might get hairy depending on how CI is feeling today 😄 If it goes south, you're welcome to split out only the kube-additions into a separate PR using image-suffix c5167094042984448 from containers/automation_images#192 In other words, those images are the good 'ole F35 & F36. The important diffs for this are (edited):

As always, ping me if you need help.

umohnani8 · 2022-09-13T14:42:24Z

Thanks @cevich!

Is this still WIP or can we get this merged in?

cevich · 2022-09-13T15:13:30Z

Is this still WIP or can we get this merged in?

That's the 10-billion dollar question. Historically, bringing in a new OS version into CI is fraught with "challenges". Sometimes though, it goes smoothly. So this PR is me rolling the dice and crossing my fingers.

Initially, the validate failures don't give me confidence 😢

umohnani8 · 2022-09-13T15:23:23Z

Looks like the package name for io/ioutil changed with the Go version bump, maybe updating the package name in the files it is complaining about will fix the issue.

cevich · 2022-09-13T15:29:39Z

maybe updating the package name in the files it is complaining about will fix the issue.

You're talking to a guy who dreams in bash 😀

I'm going to take a wild-guess that we're going to hit that same error in every single containers-repo 😢

lsm5 · 2022-09-13T16:21:45Z

from rerun with terminal

# rpm -q golang
golang-1.19.1-1.fc37.x86_64

lsm5 · 2022-09-13T16:39:11Z

I'm now hitting the can't load fmt issue open at golangci/golangci-lint#3107 both locally and on cirrus . Seems like there's a workaround at go-critic/go-critic#1126 (comment) but didn't work for me.

cevich · 2022-09-14T14:33:04Z

Crud, so @containers/podman-maintainers we're potentially going to see this pop up everywhere then until there's a fix and we bump all our golangci-lint version variables in every repo. 😞

umohnani8 · 2022-09-15T14:39:38Z

@cevich can we separate the fedora version bump from the addition of minikube so we don't block on that? I can open another PR with just the minikube changes

cevich · 2022-09-15T15:33:06Z

I can open another PR with just the minikube changes

Yes absolutely, please do this, and poke me if you run into trouble. My comment above links to the minikube-enablement bits you need. Pay careful attention to the image ID called out there, since everything built after that is using F36/F37.

cevich · 2022-09-15T15:37:07Z

Force-push: Removed the mini-kube addition.

cevich · 2022-10-19T19:23:17Z

Update: I re-ran a F36 system test task, all the others appear related to #16107

rhatdan · 2022-10-28T09:59:49Z

@cevich Any update on this?

cevich · 2022-10-31T17:02:08Z

c5178639502278656 has systemd 251.7-611

edsantiago · 2022-11-02T17:05:06Z

rootless f36 is thinking that it's netavark:

[+0008s] # Arch:amd64 [...]  Cgroups:v2+systemd Net:netavark

I think we need a CI_DESIRED_NETWORKING mechanism like the RUNTIME one added in #14912.

I have an f36 VM but cannot find any way to make it CNI. I've tried rm -rf /var/lib/containers/*, I obviously created /etc/cni/net.d/87-blahblah, tried rebooting, nothing makes it go to CNI.

Luap99 · 2022-11-02T17:11:45Z

The only way to force cni/netavark is via containers.conf network_backend option or --network-backend option for each podman command.

edsantiago · 2022-11-02T17:13:44Z

OK, we need #15413 then.

Signed-off-by: Chris Evich <[email protected]>

Building Ubuntu VM images is temporarily broken due to dependency problems on (missing) netavaro for the (required) podman package. Signed-off-by: Chris Evich <[email protected]>

Comments added to code for clarity. Signed-off-by: Chris Evich <[email protected]>

cevich · 2022-11-02T19:56:43Z

Status: Finally Green!

I'm rolling up one final fresh set of images so I can remove a workaround Jason did for the new windows tests. I'll slap that in here when it's ready and then (:crossed_fingers:) this will be good to go (finally :sweat_smile:)

edsantiago

One request (important, but I won't block over it) and one clarification/confusion question

edsantiago · 2022-11-02T20:01:10Z

contrib/cirrus/lib.sh

@@ -227,6 +236,9 @@ use_netavark() {
    export NETWORK_BACKEND=netavark  # needed for install_test_configs()
    msg "Removing any/all CNI configuration"
    rm -rvf /etc/cni/net.d/*
+    # N/B: The netavark/aardvark-dns packages are still installed and


Do you mean CNI packages?

Oh woops, yes I did mean CNI. I'll fix it, thanks.

edsantiago · 2022-11-02T20:03:01Z

.cirrus.yml

-    #PRIOR_FEDORA_CACHE_IMAGE_NAME: "prior-fedora-${IMAGE_SUFFIX}"
-    UBUNTU_CACHE_IMAGE_NAME: "ubuntu-${IMAGE_SUFFIX}"
+    PRIOR_FEDORA_CACHE_IMAGE_NAME: "prior-fedora-${IMAGE_SUFFIX}"
+    #UBUNTU_CACHE_IMAGE_NAME: "ubuntu-${IMAGE_SUFFIX}"


There's a hecka lot of commented-out stuff here. Some day someone will need to add them back, and will miss one even if they do a git log -1 -p of this commit to make sure.

Could you add # FIXME: reenable once <BUGID> is resolved to each commented-out section?

Yep I can/should add a summary comment. No problem. I thought I made this a separate commit so it can just be reverted, no?

I'm hoping to get the debian stuff rolling before too long, so hopefully that can replace Ubuntu entirely.

revert assumes an unchanging .cirrus.yml, and that will be void within (I hope) one day of this PR merging.

edsantiago · 2022-11-02T20:55:43Z

CI is green, I'm OK with merging this since it conflicts with my network-check PR. I will fix the comments there.

/lgtm

openshift-ci · 2022-11-02T20:59:41Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: cevich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cevich · 2022-11-03T14:18:35Z

Ugh...but...but....there was one more thing I needed to add...sigh...#16396

edsantiago · 2022-11-03T15:39:39Z

Timing:

type	distro	user	local	remote	container
int	fedora-36	root	29:41	30:06	29:21
int	fedora-37	root	26:19	31:16	26:01
int	fedora-36	rootless	27:00
int	fedora-37	rootless	25:23
sys	fedora-36	root	24:47	16:36
sys	fedora-37	root	25:14	16:26
sys	fedora-37-aarch64	root	29:45	20:08
sys	fedora-36	rootless	23:39
sys	fedora-37	rootless	22:37	13:50
bud	fedora-37	root	25:13	25:16

openshift-ci bot added release-note-none do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Sep 12, 2022

cevich force-pushed the enable_kube_tests branch 3 times, most recently from 6b05ccd to 7205c5d Compare September 13, 2022 14:11

cevich force-pushed the enable_kube_tests branch from 7205c5d to cdc0ba6 Compare September 14, 2022 20:23

cevich force-pushed the enable_kube_tests branch from cdc0ba6 to 0144936 Compare September 15, 2022 15:36

lsm5 linked an issue Sep 19, 2022 that may be closed by this pull request

Cirrus: enable F37 testing #15349

Closed

cevich force-pushed the enable_kube_tests branch 3 times, most recently from 936022c to 3344501 Compare September 20, 2022 19:42

github-actions bot added the kind/api-change Change to remote API; merits scrutiny label Sep 20, 2022

cevich force-pushed the enable_kube_tests branch 2 times, most recently from 3f22fb9 to ab7898b Compare September 21, 2022 14:34

github-actions bot removed the kind/api-change Change to remote API; merits scrutiny label Sep 21, 2022

cevich force-pushed the enable_kube_tests branch 3 times, most recently from 22f984f to aa8a296 Compare September 23, 2022 14:12

cevich mentioned this pull request Sep 26, 2022

Future CI: F37 fails 'podman load compressed tar file' #15944

Closed

cevich force-pushed the enable_kube_tests branch from 1c6e10c to 5e51ee2 Compare October 19, 2022 17:37

cevich mentioned this pull request Oct 20, 2022

Cirrus: enable F37 testing #15349

Closed

cevich force-pushed the enable_kube_tests branch from 5e51ee2 to 95599a1 Compare October 31, 2022 16:58

cevich changed the title ~~[WIP] Cirrus: Update CI VM images to F37beta~~ Cirrus: Update CI VM images to F37beta Oct 31, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 31, 2022

cevich added 2 commits November 2, 2022 13:38

Cirrus: Update to F37beta

ecd1927

Signed-off-by: Chris Evich <[email protected]>

Cirrus: temp. disable all Ubuntu testing

fd9de87

Building Ubuntu VM images is temporarily broken due to dependency problems on (missing) netavaro for the (required) podman package. Signed-off-by: Chris Evich <[email protected]>

cevich force-pushed the enable_kube_tests branch from 95599a1 to 0d2acb1 Compare November 2, 2022 17:39

Cirrus: Guarantee CNI testing w/o nv/av present

8530724

Comments added to code for clarity. Signed-off-by: Chris Evich <[email protected]>

cevich force-pushed the enable_kube_tests branch from 0d2acb1 to 8530724 Compare November 2, 2022 17:42

edsantiago reviewed Nov 2, 2022

View reviewed changes

edsantiago mentioned this pull request Nov 2, 2022

CI: set and verify DESIRED_NETWORK (netavark, cni) #16389

Merged

openshift-ci bot assigned edsantiago Nov 2, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 2, 2022

edsantiago added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2022

openshift-merge-robot merged commit 774e950 into containers:main Nov 2, 2022

edsantiago mentioned this pull request Nov 3, 2022

New f37 flake: podman image prune: APK error or timeout #16391

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cirrus: Update CI VM images to F37beta #15760

Cirrus: Update CI VM images to F37beta #15760

cevich commented Sep 12, 2022 •

edited

Loading

cevich commented Sep 12, 2022 •

edited

Loading

umohnani8 commented Sep 13, 2022

cevich commented Sep 13, 2022

umohnani8 commented Sep 13, 2022

cevich commented Sep 13, 2022

lsm5 commented Sep 13, 2022

lsm5 commented Sep 13, 2022

cevich commented Sep 14, 2022 •

edited

Loading

umohnani8 commented Sep 15, 2022

cevich commented Sep 15, 2022

cevich commented Sep 15, 2022 •

edited

Loading

cevich commented Oct 19, 2022

rhatdan commented Oct 28, 2022

cevich commented Oct 31, 2022

edsantiago commented Nov 2, 2022

Luap99 commented Nov 2, 2022

edsantiago commented Nov 2, 2022

cevich commented Nov 2, 2022

edsantiago left a comment

edsantiago Nov 2, 2022

cevich Nov 2, 2022

edsantiago Nov 2, 2022

cevich Nov 2, 2022

edsantiago Nov 2, 2022

edsantiago commented Nov 2, 2022

openshift-ci bot commented Nov 2, 2022

cevich commented Nov 3, 2022

edsantiago commented Nov 3, 2022

Cirrus: Update CI VM images to F37beta #15760

Cirrus: Update CI VM images to F37beta #15760

Conversation

cevich commented Sep 12, 2022 • edited Loading

Does this PR introduce a user-facing change?

cevich commented Sep 12, 2022 • edited Loading

umohnani8 commented Sep 13, 2022

cevich commented Sep 13, 2022

umohnani8 commented Sep 13, 2022

cevich commented Sep 13, 2022

lsm5 commented Sep 13, 2022

lsm5 commented Sep 13, 2022

cevich commented Sep 14, 2022 • edited Loading

umohnani8 commented Sep 15, 2022

cevich commented Sep 15, 2022

cevich commented Sep 15, 2022 • edited Loading

cevich commented Oct 19, 2022

rhatdan commented Oct 28, 2022

cevich commented Oct 31, 2022

edsantiago commented Nov 2, 2022

Luap99 commented Nov 2, 2022

edsantiago commented Nov 2, 2022

cevich commented Nov 2, 2022

edsantiago left a comment

Choose a reason for hiding this comment

edsantiago Nov 2, 2022

Choose a reason for hiding this comment

cevich Nov 2, 2022

Choose a reason for hiding this comment

edsantiago Nov 2, 2022

Choose a reason for hiding this comment

cevich Nov 2, 2022

Choose a reason for hiding this comment

edsantiago Nov 2, 2022

Choose a reason for hiding this comment

edsantiago commented Nov 2, 2022

openshift-ci bot commented Nov 2, 2022

cevich commented Nov 3, 2022

edsantiago commented Nov 3, 2022

cevich commented Sep 12, 2022 •

edited

Loading

cevich commented Sep 12, 2022 •

edited

Loading

cevich commented Sep 14, 2022 •

edited

Loading

cevich commented Sep 15, 2022 •

edited

Loading