Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

network flakes: other #17288

Open
edsantiago opened this issue Jan 30, 2023 · 6 comments
Open

network flakes: other #17288

edsantiago opened this issue Jan 30, 2023 · 6 comments
Labels
flakes Flakes from Continuous Integration

Comments

@edsantiago
Copy link
Member

This is a placeholder issue for podman CI network flakes that are not #16973 (cdn03.quay.io) or #17193 ("happened during read").

RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading

git clone --recursive --branch=$DEST_BRANCH https://x-access-token:${CIRRUS_REPO_CLONE_TOKEN}@github.com/${CIRRUS_REPO_FULL_NAME}.git $CIRRUS_WORKING_DIR
Cloning into '/var/tmp/go/src/github.com/containers/podman'...
error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
error: 8192 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

golangci-lint info found version

Attempt 1 of 7 (retry on non-zero exit):
    + ooe.sh dnf install -y python3-devel 
    exit(0)
VERSION=1.50.1 ./hack/install_golangci.sh
Installing golangci-lint v1.50.1 into ./bin/golangci-lint
golangci/golangci-lint info checking GitHub for tag 'v1.50.1'
golangci/golangci-lint info found version:  <!DOCTYPE html> <html lang= for  <!DOCTYPE html> <html lang=/linux/amd64
make: *** [Makefile:925: .install.golangci-lint] Error 3

Also here

curl repo.tbz 404

time $ARTCURL/build_aarch64/repo/repo.tbz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0    70    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404

curl HTTP/2 stream not closed cleanly

This is by far the most common one.

time $ARTCURL/build_aarch64/repo/repo.tbz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  195k    0  195k    0     0   322k      0 --:--:-- --:--:-- --:--:--  321k
100 82.0M    0 82.0M    0     0  51.1M      0 --:--:--  0:00:01 --:--:-- 51.1M
100  103M    0  103M    0     0  36.5M      0 --:--:--  0:00:02 --:--:-- 36.5M
100  103M    0  103M    0     0  27.0M      0 --:--:--  0:00:03 --:--:-- 27.0M
100  103M    0  103M    0     0  21.4M      0 --:--:--  0:00:04 --:--:-- 21.4M
100  103M    0  103M    0     0  17.7M      0 --:--:--  0:00:05 --:--:-- 19.8M
100  103M    0  103M    0     0  15.1M      0 --:--:--  0:00:06 --:--:-- 4253k
100  103M    0  103M    0     0  13.2M      0 --:--:--  0:00:07 --:--:--     0
[lots and lots and lots of "0" lines]
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

some apk error

(Not sure if this one is really network)

# podman-remote build ...
...
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/8.3.0/cc1: I/O error
ERROR: gcc-8.3.0-r0: IO ERROR

ping error

(just three of these, and we probably won't see more, thanks to buildah PR 4531)

# podman-remote build ...
PING 4.2.2.2 (4.2.2.2): 56 data bytes

--- 4.2.2.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
Error: building at STEP "RUN ping -c 1 4.2.2.2": while running runtime: exit status 1

I realize it is hopeless to lump these all together. I'm doing so anyway because they're getting more and more frequent, and I'm starting to suspect that the quay.io flake, the "happened during" flake, and these flakes, might be all connected and perhaps be a network problem in our CI.
@edsantiago edsantiago added the flakes Flakes from Continuous Integration label Jan 30, 2023
@Luap99
Copy link
Member

Luap99 commented Jan 31, 2023

might be all connected and perhaps be a network problem in our CI.

The aarch VMs run in AWS and the x86 ones in gcloud, right? It might be interesting to filter the network errors based on that to see if this a cloud provider issue?
cc @cevich

@cevich
Copy link
Member

cevich commented Jan 31, 2023

The aarch VMs run in AWS and the x86 ones in gcloud, right?

Correct, and I'm pretty sure the underlying networking setup is almost entirely different.

@cevich
Copy link
Member

cevich commented Jan 31, 2023

Possibly related (also aarch64 on AWS): containers/netavark#433

@edsantiago edsantiago mentioned this issue Feb 14, 2023
Open
@edsantiago
Copy link
Member Author

This is the only quay.io failure I can find in RHEL gating-test logs:

not ok 538 pull image into a full storage
...
$ buildah --root=/tmp/buildah-test pull --signature-policy /usr/share/buildah/test/system/./policy.json alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
parsing image configuration: Get "https://cdn03.quay.io/sha256/96/961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
     ?X-Amz-Algorithm=AWS4-HMAC-SHA256
     &X-Amz-Credential=AKIAI5LUAQGPZRPNKSJA%2F20230206%2Fus-east-1%2Fs3%2Faws4_request
     &X-Amz-Date=20230206T124402Z
     &X-Amz-Expires=600
     &X-Amz-SignedHeaders=host
     &X-Amz-Signature=ce946b86fe8265c8db2dc7695e0758d104a584b385f122433a02cd670bdc1fb6
     &cf_sign=cqeGovWnO5c8L7E3AnZxOTbW%2FTX01Zl%2BiePEnxn7BA7DseHg8wde58MdN3joVi43OlROpIN6RjWGAzn3t0PxI1GN6iTdINMSK7Ngh12EDjvgGLqjr%2FTyDZQ%2BU1Ppb%2BvRuvh%2BwbysezFTFasli%2BUInIVOXFFa9uGUb%2BdAv4FTp6Q3UI8EyHNzycTbcvUgESsX9MOAhu4gUyd3vWqpdNQWRzP1XPBENSfZyHjezr5MMH3A2DzdnTVgu0v8U7iQD06KAdL7ZBH06PXr0ALyOg9Xs7UqAzUpHctyTS9oyuHGbz9nPw3fPGY29Mn1r7PUNE6%2F6wIGNlfJQVsiYLTur73QjQ%3D%3D
     &cf_expiry=1675688042
     &region=us-east-1": remote error: tls: handshake failure

@edsantiago edsantiago mentioned this issue Feb 14, 2023
Closed
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@edsantiago
Copy link
Member Author

Here's another handshake failure, APIv2 tests, f37 rootless:

>Trying to pull quay.io/libpod/alpine_labels:latest...
>Error: creating build container: parsing image configuration: Get "https://cdn03.quay.io/sha256/5e/5e9e9275e4d60569c72dde5d1e31cee0795df81a386bdc325873b8ed22875bc6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI5LUAQGPZRPNKSJA%2F20230405%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230405T020717Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f0e1a2a4f91c67ac341ffa7524d165cf3b428c86131076e46b3d5c4e9d32762d&cf_sign=G2seBRiZcD6dwtMsbSRVTf2yqq9j6OVuYek86y73u%2BlES%2B8yfn9J5IOF1bKL2pdhX4uAaj5eImoKzqu6qDihi%2Br5LGf6og9caikIVozgjOkasB1Gjz45XJOXu1E5ufNi7r%2BFQ8IWevdkOHh%2F80JkYKe%2BewaDDl%2B1n5LVh3MbE6auaEvqK%2FLtvB6uSv%2BjoWfkk2GSSXJ%2BY5NirC7g5%2BlQQuVd%2Fy3WJuVgaHXq30y%2BF305YvKzAefTX80v2%2BfbQULUQQ67OuKChCG%2F274e9CQGlSPcJqyPUxS2CvyNz87ipnYppydr43Jpq4sqI0miS%2BuUCm7tX3mVlxwlX6bS%2F%2FxpXQ%3D%3D&cf_expiry=1680661037&region=us-east-1": remote error: tls: handshake failure
Bailing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flakes Flakes from Continuous Integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@edsantiago @cevich @rhatdan @Luap99