CI - various fixes #7070

edsantiago · 2020-07-23T17:44:22Z

Primary purpose: upgrade crun to 0.14 on f31, in hopes of
eliminating the 'cgroups.freeze' flake that is plaguing CI.

While I'm at it:

remove a no-longer-needed dnf upgrade that was running in CI
itself (not image building, in each actual CI run). The purpose
was to upgrade conmon, but that was added a long time ago and
the required conmon is now in stable. The effect of this
dnf upgrade today was simply to cause flakes when fedora
repos were offline.
remove a no-longer-needed check for varlink.
networking.sh : add a timeout! 'openssl s_client' will happily
hang forever if a host is unreachable, which means we waste
two hours waiting for Cirrus to time out.
timestamp.awk : include date (not just time) in START/END msgs.
There are times when I'm looking at a CI log and it is ultra
important to know if it is from yesterday or today.
add progress messages in some places where I've previously
struggled to understand context in logs; and improve some
unlikely error messages to include script name.

...then, after all that, wrote a new README about how to to
all this. Hope it helps someone.

Signed-off-by: Ed Santiago [email protected]

mheon · 2020-07-23T18:22:42Z

LGTM

rhatdan · 2020-07-23T19:33:12Z

/lgtm
/hold

rhatdan · 2020-07-23T19:33:19Z

/approve

openshift-ci-robot · 2020-07-23T19:33:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

edsantiago · 2020-07-23T20:47:52Z

I don't think this is working. Cirrus logs show no progress. I think it's going to time out real soon.

edsantiago · 2020-07-23T21:14:52Z

Yep, timeout in networking.sh. Added a fix for that so future devs don't have to waste two hours like this.

edsantiago · 2020-07-23T22:50:09Z

Step 1 complete. Step 2 just beginning now. At the time of my push, all tests except tide were green.

Notable changes since first revision: addition of timeout in networking.sh; removal of ad-hoc dnf step; addition of new README file which I hope will help the next non-Chris person needing to do this.

edsantiago · 2020-07-24T01:46:30Z

@cevich PTAL. This has lots of little changes, mostly independent of each other, just a hodgepodge of problems I've found over the last few months. Plus a new README because this process is super duper confusing. I know you've documented some of it but my README is intended purely as a simple recipe which any monkey -- i.e. me -- can follow step by step. It is not meant to cover all the complicated actions that take place in Cirrus.

vrothberg

I love the new doc, thanks @edsantiago!

Let's wait for @cevich's head nod before merging.

cevich · 2020-07-24T14:43:21Z

Yeah, thanks a bunch Ed, I'll be the first one to admit I can't do it all, well, and readable. I greatly appreciate the assistance (and documentation to boot!). Peeking now...

cevich

Nice work Ed, and again, very much appreciated. Only a few items I found for you to consider, and maybe one line to drop. My motivation behind recommending contrib/cirrus/README.md is also to get this wonderful work more up-front in developers faces. It should not be much of an obscure, back-end process. Rolling these VM images is a somewhat frequent occurrence as the workflow (and sometimes images) are used by all containers/* repos.

cevich · 2020-07-24T14:46:05Z

.cirrus.yml

@@ -39,7 +39,7 @@ env:
    UBUNTU_NAME: "ubuntu-20"
    PRIOR_UBUNTU_NAME: "ubuntu-19"

-    _BUILT_IMAGE_SUFFIX: "podman-5869602141896704"
+    _BUILT_IMAGE_SUFFIX: "podman-6439450735542272"


👏 you made it through the "magic VM building workflow", yay 👏

(I found this out because the number in the name is a $CIRRUS_BUILD_ID...which maps back to this PR)

cevich · 2020-07-24T14:47:03Z

contrib/cirrus/networking.sh

@@ -10,7 +10,7 @@ while read host port
 do
    if [[ "$port" -eq "443" ]]
    then
-        item_test "SSL/TLS to $host:$port" "$(echo -n '' | openssl s_client -quiet -no_ign_eof -connect $host:$port &> /dev/null; echo $?)" -eq "0"
+        item_test "SSL/TLS to $host:$port" "$(echo -n '' | timeout 60 openssl s_client -quiet -no_ign_eof -connect $host:$port &> /dev/null; echo $?)" -eq "0"


contrib/cirrus/packer/README.how-to-update-cirrus-vms

cevich · 2020-07-24T15:02:05Z

contrib/cirrus/packer/fedora_packaging.sh

+# That way this will be dead code in future CI:IMG PRs but will
+# serve as an example for anyone in a similar future situation.
+#   $BIGTO ooe.sh $SUDO dnf --enablerepo=updates-testing -y upgrade crun
+


Nice comment, I hope that will be helpful to someone.

contrib/cirrus/rootless_test.sh

cevich · 2020-07-24T15:04:14Z

contrib/cirrus/setup_environment.sh

@@ -57,9 +57,6 @@ case "${OS_RELEASE_ID}" in

        workaround_bfq_bug

-	# HACK: Need Conmon 2.0.17, currently in updates-testing on F31.
-	dnf update -y --enablerepo=updates-testing conmon
-


Oh good, yes, thank you for confirming this can be removed.

cevich · 2020-07-24T15:04:40Z

contrib/cirrus/timestamp.awk

@@ -7,7 +7,7 @@
 BEGIN {
    STARTTIME=systime()
    printf "[%s] START", strftime("%T")
-    printf " - All [+xxxx] lines that follow are relative to right now.\n"
+    printf " - All [+xxxx] lines that follow are relative to %s.\n", strftime("%FT%T")


ahh yes...much more helpful-ing this way. Thanks.

Primary purpose: upgrade crun to 0.14 on f31, in hopes of eliminating the 'cgroups.freeze' flake that is plaguing CI. While I'm at it: - remove a no-longer-needed dnf upgrade that was running in CI itself (not image building, in each actual CI run). The purpose was to upgrade conmon, but that was added a long time ago and the required conmon is now in stable. The effect of this dnf upgrade today was simply to cause flakes when fedora repos were offline. - remove a no-longer-needed check for varlink. - networking.sh : add a timeout! 'openssl s_client' will happily hang forever if a host is unreachable, which means we waste two hours waiting for Cirrus to time out. - timestamp.awk : include date (not just time) in START/END msgs. There are times when I'm looking at a CI log and it is ultra important to know if it is from yesterday or today. - add progress messages in some places where I've previously struggled to understand context in logs; and improve some unlikely error messages to include script name. ...then, after all that, wrote a new README about how to to all this. Hope it helps someone. Signed-off-by: Ed Santiago <[email protected]>

vrothberg

LGTM
@mheon @rhatdan PTAL

mheon · 2020-07-27T15:12:36Z

LGTM - but would like a final signoff from @cevich before merge

rhatdan · 2020-07-27T18:53:04Z

/lgtm
/hold cancel

openshift-ci-robot requested review from umohnani8 and vrothberg July 23, 2020 17:44

edsantiago force-pushed the ci_fixes branch from 338cfce to c0ca548 Compare July 23, 2020 18:50

openshift-ci-robot assigned rhatdan Jul 23, 2020

openshift-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Jul 23, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 23, 2020

edsantiago force-pushed the ci_fixes branch from c0ca548 to ae9c827 Compare July 23, 2020 21:12

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jul 23, 2020

edsantiago force-pushed the ci_fixes branch from ae9c827 to ee6073c Compare July 23, 2020 22:48

edsantiago changed the title ~~[CI:IMG] CI - various fixes~~ CI - various fixes Jul 24, 2020

edsantiago force-pushed the ci_fixes branch from ee6073c to 697d261 Compare July 24, 2020 00:21

vrothberg reviewed Jul 24, 2020

View reviewed changes

cevich reviewed Jul 24, 2020

View reviewed changes

edsantiago force-pushed the ci_fixes branch from 697d261 to 9fad55c Compare July 25, 2020 18:33

edsantiago mentioned this pull request Jul 25, 2020

CI flake: Panic in Spec Teardown: send on closed channel #6518

Closed

vrothberg reviewed Jul 27, 2020

View reviewed changes

edsantiago mentioned this pull request Jul 27, 2020

When chowning we should not follow symbolic link #7094

Merged

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 27, 2020

openshift-merge-robot merged commit 956caf3 into containers:master Jul 27, 2020

edsantiago deleted the ci_fixes branch July 27, 2020 19:14

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI - various fixes #7070

CI - various fixes #7070

edsantiago commented Jul 23, 2020 •

edited

Loading

mheon commented Jul 23, 2020

rhatdan commented Jul 23, 2020

rhatdan commented Jul 23, 2020

openshift-ci-robot commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 24, 2020

vrothberg left a comment

cevich commented Jul 24, 2020

cevich left a comment

cevich Jul 24, 2020

cevich Jul 24, 2020

cevich Jul 24, 2020

cevich Jul 24, 2020

cevich Jul 24, 2020

vrothberg left a comment

mheon commented Jul 27, 2020

rhatdan commented Jul 27, 2020

CI - various fixes #7070

CI - various fixes #7070

Conversation

edsantiago commented Jul 23, 2020 • edited Loading

mheon commented Jul 23, 2020

rhatdan commented Jul 23, 2020

rhatdan commented Jul 23, 2020

openshift-ci-robot commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 23, 2020

edsantiago commented Jul 24, 2020

vrothberg left a comment

Choose a reason for hiding this comment

cevich commented Jul 24, 2020

cevich left a comment

Choose a reason for hiding this comment

cevich Jul 24, 2020

Choose a reason for hiding this comment

cevich Jul 24, 2020

Choose a reason for hiding this comment

cevich Jul 24, 2020

Choose a reason for hiding this comment

cevich Jul 24, 2020

Choose a reason for hiding this comment

cevich Jul 24, 2020

Choose a reason for hiding this comment

vrothberg left a comment

Choose a reason for hiding this comment

mheon commented Jul 27, 2020

rhatdan commented Jul 27, 2020

edsantiago commented Jul 23, 2020 •

edited

Loading