-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DO NOT MERGE - WIP] Cirrus: use Ubuntu 22.04 LTS #14397
Conversation
cd67454
to
4949830
Compare
And now the "real fun" begins (log). 😁 |
@mtrmac ping, any idea why the |
(The linked log seems to be a straightforward repo unavailability; looking at https://api.cirrus-ci.com/v1/task/5492536713150464/logs/main.log instead.)
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes LGTM
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cevich, lsm5 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
int podman fedora-36 root container is hitting a |
The RNG/name-collision problem was seen in Fedora-land last year I believe. Miloslav and I (mostly Miloslav) spent quite a bit of time poking at it. I would point out that we do have a software RNG service (rngd) enabled in the Fedora images, but not explicitly enabled in Ubuntu (IIRC). If there isn't one by default, it's possible this needs to be installed/enabled for Ubuntu. |
Note that the collision happens on the very first test that is not skipped, and all subsequent tests consistently fail as well. This is not like the random collisions of earlier, when there would be a random failure once in a month. At this point I’m guessing there’s nothing random about that; the lock file creation is consistently failing for a reason that should be fixable. (But I also didn’t do any work to diagnose this further.) |
4949830
to
aea263e
Compare
I remember things differently (and maybe wrongly), a |
Hmmm, so adding e.g. I wonder if we're loosing |
Yes, that’s what the “alternative hypothesis” part of #14397 (comment) suggests. Do y’all need me to prepare a patch to that effect? |
na, looking into it now. Let me get back to you. |
@lsm5 as per https://paste.centos.org/view/f7b8a0f8 I'm onboard with mtrmac. Damn lock file simply isn't being created for some reason specific to Ubuntu 😕 I peeked at the other failures here and TBH I'd just ignore them until the lockfile thing is figured out. Hopefully by then there won't be too many more new problems (and maybe less if we're luck) 😁 |
aea263e
to
43745a2
Compare
@lsm5 looking better now. The next-worse problem seems to be the |
rerunning them now. Could it be because registry was down or something? |
|
IIRC this is a locally run registry container. But yeah, it could be a flake, worth checking open-issues if it reproduces. |
Yeah, that's concerning. |
43745a2
to
d957460
Compare
force-push: Rebased on main. |
I think there's something broken in networking. I did a # hack/podman-registry start
PODMAN_REGISTRY_IMAGE="quay.io/libpod/registry:2.6"
PODMAN_REGISTRY_PORT="5055"
PODMAN_REGISTRY_USER="user0cDX"
PODMAN_REGISTRY_PASS="keyT0ZSieGrqCy3"
# telnet localhost 5055
Trying 127.0.0.1...
telnet: Unable to connect to remote host: No route to host
There is a bad interface on the system: # ip a
...
3: cni-podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 12:d0:4c:e4:e6:c3 brd ff:ff:ff:ff:ff:ff
inet 10.88.0.1/16 brd 10.88.255.255 scope global cni-podman0
valid_lft forever preferred_lft forever
inet6 fe80::10d0:4cff:fee4:e6c3/64 scope link
valid_lft forever preferred_lft forever Deleting it (via @Luap99 here's the iptables output you requested: # iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
19 9639 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
1 32 NETAVARK-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1654 107K CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
133 8891 NETAVARK-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
488 30887 NETAVARK-HOSTPORT-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0
3042 192K CNI-HOSTPORT-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd requiring masquerade */
0 0 CNI-a8255fdfc238d8dd191b91e2 all -- * * 10.88.0.176 0.0.0.0/0 /* name: "podman" id: "0efdabc90252e4ecb9d07ac8dfb8dd0289c6fbb8b4420e1efcf7a3688913690a" */
0 0 CNI-e060fd9781bbc14435e13cb1 all -- * * 10.88.2.67 0.0.0.0/0 /* name: "podman" id: "8d20cf97764f17ec4fe5bc64ab301a043803ea175260465085d0defa9685c20e" */
0 0 CNI-d2034e09d0207d90cb0b3c73 all -- * * 10.88.73.146 0.0.0.0/0 /* name: "podman" id: "0af3ee4d0991539a49ffbd4bfd1e7b512a5a3e981ed63880c486a60f0230107c" */
0 0 CNI-e9768ba8e3d20a3ac43ebc4d all -- * * 10.88.73.147 0.0.0.0/0 /* name: "podman" id: "82f1c45e16095ed236ada94be140f8dcd82ea0d6e92fadc82412bb732e08b369" */
13 828 NETAVARK-1D8721804F16F all -- * * 10.88.0.0/16 0.0.0.0/0
0 0 CNI-0b1cafeb9628ee0b55490e13 all -- * * 10.88.0.16 0.0.0.0/0 /* name: "podman" id: "edfe69e088678a6d08c47f634caf413dc6b874b33e644d2b20cf6559d43a3a58" */
0 0 CNI-8b675d9317655824ae30fa13 all -- * * 10.88.35.217 0.0.0.0/0 /* name: "podman" id: "1969dd24f05c18fb26aa82daa25045dff36879132efb2d495cf2992d80c70b4f" */
0 0 CNI-21bee5fab41f61342381d410 all -- * * 10.88.35.232 0.0.0.0/0 /* name: "podman" id: "14f96381d6958166bea5ef1d81c70570b07ff4505c30894f19afc0d38b82592d" */
0 0 CNI-4e1cb35a555b72d297edfd63 all -- * * 10.88.35.233 0.0.0.0/0 /* name: "podman" id: "469afa180258626ce228bd42c83fb53b10d34ec5e91b72d9bce3542072e3463d" */
Chain CNI-0b1cafeb9628ee0b55490e13 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "edfe69e088678a6d08c47f634caf413dc6b874b33e644d2b20cf6559d43a3a58" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "edfe69e088678a6d08c47f634caf413dc6b874b33e644d2b20cf6559d43a3a58" */
Chain CNI-21bee5fab41f61342381d410 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "14f96381d6958166bea5ef1d81c70570b07ff4505c30894f19afc0d38b82592d" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "14f96381d6958166bea5ef1d81c70570b07ff4505c30894f19afc0d38b82592d" */
Chain CNI-4e1cb35a555b72d297edfd63 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "469afa180258626ce228bd42c83fb53b10d34ec5e91b72d9bce3542072e3463d" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "469afa180258626ce228bd42c83fb53b10d34ec5e91b72d9bce3542072e3463d" */
Chain CNI-8b675d9317655824ae30fa13 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "1969dd24f05c18fb26aa82daa25045dff36879132efb2d495cf2992d80c70b4f" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "1969dd24f05c18fb26aa82daa25045dff36879132efb2d495cf2992d80c70b4f" */
Chain CNI-HOSTPORT-DNAT (2 references)
pkts bytes target prot opt in out source destination
Chain CNI-HOSTPORT-MASQ (1 references)
pkts bytes target prot opt in out source destination
119 7140 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000
Chain CNI-HOSTPORT-SETMARK (0 references)
pkts bytes target prot opt in out source destination
296 17760 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd masquerade mark */ MARK or 0x2000
Chain CNI-a8255fdfc238d8dd191b91e2 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "0efdabc90252e4ecb9d07ac8dfb8dd0289c6fbb8b4420e1efcf7a3688913690a" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "0efdabc90252e4ecb9d07ac8dfb8dd0289c6fbb8b4420e1efcf7a3688913690a" */
Chain CNI-d2034e09d0207d90cb0b3c73 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "0af3ee4d0991539a49ffbd4bfd1e7b512a5a3e981ed63880c486a60f0230107c" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "0af3ee4d0991539a49ffbd4bfd1e7b512a5a3e981ed63880c486a60f0230107c" */
Chain CNI-e060fd9781bbc14435e13cb1 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "8d20cf97764f17ec4fe5bc64ab301a043803ea175260465085d0defa9685c20e" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "8d20cf97764f17ec4fe5bc64ab301a043803ea175260465085d0defa9685c20e" */
Chain CNI-e9768ba8e3d20a3ac43ebc4d (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "82f1c45e16095ed236ada94be140f8dcd82ea0d6e92fadc82412bb732e08b369" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "82f1c45e16095ed236ada94be140f8dcd82ea0d6e92fadc82412bb732e08b369" */
Chain NETAVARK-1D8721804F16F (1 references)
pkts bytes target prot opt in out source destination
1 60 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16
12 768 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4
Chain NETAVARK-DN-1D8721804F16F (1 references)
pkts bytes target prot opt in out source destination
0 0 NETAVARK-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:5055
3 180 NETAVARK-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:5055
3 180 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5055 to:10.88.0.3:5000
Chain NETAVARK-HOSTPORT-DNAT (2 references)
pkts bytes target prot opt in out source destination
3 180 NETAVARK-DN-1D8721804F16F tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5055 /* dnat name: podman id: 5a0f24868116ef2bd39df01e9065b1a8775584491afd2cd34cf72bcaa0134864 */
Chain NETAVARK-HOSTPORT-MASQ (1 references)
pkts bytes target prot opt in out source destination
180 10800 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* netavark portfw masq mark */ mark match 0x2000/0x2000
Chain NETAVARK-HOSTPORT-SETMARK (2 references)
pkts bytes target prot opt in out source destination
3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x2000 |
That clearly shows a mix use of cni/netavark which is not supported! |
Thanks you, that's it.
podman/test/e2e/manifest_test.go Lines 297 to 300 in cf74739
(side note: the double use of Anyhow, SOLUTION: get rid of the |
Submitted #14845 to address the registry startup issue. I chose not to heed my "however all the other tests do it" declaration because that way lies madness. |
manifest_test:authenticated_push() is the final test left to fix before merging containers#14397. The reason it's failing _seems_ to be that podman is running with a mix of netavark and CNI, and that _seems_ to be because this test invokes hack/podman-registry which invokes plain podman without whatever options used in e2e. Starting a registry directly from the test is insane: there is no reusable code for doing that (see login_logout_test.go and push_test.go. Yeesh.) Solution: set $PODMAN, by inspecting the podmanTest object which includes both a path and a list of options. podman-registry will invoke that. (It will also override --root and --runroot. This is the desired behavior). Also: add cleanup. If auth-push test fails, stop the registry. Also: add a sanity check to podman-registry script, have it wait for the registry port to activate. Die if it doesn't. That could've saved us a nice bit of debugging time. Signed-off-by: Ed Santiago <[email protected]>
@lsm5: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Lol, thanks for checking into this Ed and Paul. FYI: I'm pretty sure during setup we |
That doesn't matter. The short version of the problem is: e2e tests make excessive use of podman command line args ( But it's moot, because my fixer PR is merged, so, @lsm5, please rebase & repush, this should pass now. |
Well screw up any future e2e runs, yes maybe. Though |
No, I mean, screw up the e2e test run right smack in the middle of the run. It's kind of complicated, but here's the skeleton:
Does that make sense? |
Oh I see, gotcha, okay thanks. |
I'm going to close this PR in favor of: #14719 because:
|
Aw, phooey, @cevich, can I ask you to reconsider? All this needs is a rebase & push, and life will (should) be happy. We can get Ubuntu 22 right now, all we need is this PR to merge. The criu issue does not affect this PR. |
I think all we're loosing is the PR comments (which may actually be valuable). My #14719 has all the changes here + the fix for bats (slightly newer images) + my attention - @lsm5 asked to be let off the hook in a "help!" comment above. Otherwise I'm not strongly opposed to re-opening this and re-running the tests, just lazy and don't want to be overwhelmed with too many image-update PRs in-flight. |
Signed-off-by: Lokesh Mandvekar [email protected]
Does this PR introduce a user-facing change?
depends on containers/automation_images#134