Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental workaround for cdn03.quay.io flake #17505

Merged
merged 1 commit into from
Feb 16, 2023

Conversation

edsantiago
Copy link
Member

I can't find instances of the "lookup cdn03.quay.io" flake
in RHEL gating tests. RHEL does not use systemd-resolved.
Maybe pure coincidence. Maybe not.

Signed-off-by: Ed Santiago [email protected]

None

@openshift-ci openshift-ci bot added release-note-none approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 14, 2023
@edsantiago edsantiago force-pushed the fix_dns branch 6 times, most recently from 69fa82a to ad40da7 Compare February 15, 2023 03:11
Comment on lines 224 to 227
systemctl start NetworkManager

# Sigh. The first time, NetworkManager doesn't seem
# to create a resolv.conf. Keep retrying until it does.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to restart it, systemd will ignore start when a unit it is already running and AFAIK NetworkManager should already be running unless we explicitly disabled it use.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, could be. On a 1minutetip VM it was not started by default. Anyhow, no flakes in two runs, I'm going to keep hammering away at CI with this to see how it goes. Thanks for the suggestion.

@edsantiago edsantiago force-pushed the fix_dns branch 5 times, most recently from fe44d73 to ddb2d8f Compare February 15, 2023 20:49
@edsantiago
Copy link
Member Author

Five (six?) runs, and no sign of the flake; but here's the flake, right now, in a different PR: https://cirrus-ci.com/task/5480576929497088

I can't find instances of the "lookup cdn03.quay.io" flake
in RHEL gating tests. RHEL does not use systemd-resolved.
Maybe pure coincidence. Maybe not.

Let's try disabling systemd-resolved and see how things go.

Signed-off-by: Ed Santiago <[email protected]>
@edsantiago edsantiago changed the title DO NOT MERGE: Experimental workaround for cdn03.quay.io flake Experimental workaround for cdn03.quay.io flake Feb 15, 2023
@edsantiago
Copy link
Member Author

I lost count of the retries. None failed with the cdn03 flake, despite other concurrent PRs flaking that way.

@containers/podman-maintainers it makes me queasy to ask this, but please review and merge if you see fit. I'd like to reevaluate in a few weeks, see if the cdn03 flake persists, and then figure out where to go from there.

Copy link
Member

@vrothberg vrothberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I added a reminder to my calendar.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 16, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [edsantiago,vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Luap99
Copy link
Member

Luap99 commented Feb 16, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 16, 2023
@openshift-merge-robot openshift-merge-robot merged commit 8a6b373 into containers:main Feb 16, 2023
@edsantiago edsantiago deleted the fix_dns branch February 16, 2023 14:40
@vrothberg
Copy link
Member

LGTM

I added a reminder to my calendar.

Reminder kicked in. @edsantiago, all good with this change?

@edsantiago
Copy link
Member Author

Oh, interesting. Here's the cdn03 flake list for the last 25 days:

...and of those, some are in "Build" step (which ISTR happens before my disable-systemd code?) and the rest are in v4.4 branches. I would say this has been a success. Now what???

@vrothberg
Copy link
Member

Now what???

Can we disable it when building the VM images?

@vrothberg
Copy link
Member

Probably worth reporting the bug as well.

@cevich
Copy link
Member

cevich commented Mar 31, 2023

/cherrypick v4.4

@openshift-cherrypick-robot
Copy link
Collaborator

@cevich: only containers org members may request cherry picks. You can still do the cherry-pick manually.

In response to this:

/cherrypick v4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 3, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note-none
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants