Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pprof test is flaking again #12167

Closed
edsantiago opened this issue Nov 3, 2021 · 10 comments · Fixed by #12518, #12569 or #12588
Closed

pprof test is flaking again #12167

edsantiago opened this issue Nov 3, 2021 · 10 comments · Fixed by #12518, #12569 or #12588
Assignees
@jwhonce
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless

Comments

@edsantiago
Copy link
Member

Currently our number one flake. @jwhonce PTAL.

podman system service verify pprof endpoints [It] are available

podman system service verify pprof endpoints [It] are not available

Common factor seems to be rootless.
@edsantiago edsantiago added flakes Flakes from Continuous Integration rootless labels Nov 3, 2021
@edsantiago
Copy link
Member Author

@jwhonce could you make time to look at this please?

podman system service verify pprof endpoints [It] are not available

podman system service verify pprof endpoints [It] are available

@edsantiago
Copy link
Member Author

@jwhonce pretty pretty please? This is our number-one flake right now

@edsantiago
Copy link
Member Author

podman system service verify pprof endpoints [It] are not available

@jwhonce ping

@edsantiago
Copy link
Member Author

Can someone else, anyone, please take a look at this flake? It's hitting us hard.

podman system service verify pprof endpoints [It] are not available

@vrothberg
Copy link
Member

I'll take a look ✔️

vrothberg added a commit to vrothberg/libpod that referenced this issue Dec 6, 2021
@vrothberg
e2e: fix pprof flakes
6d7e6d7 
Give the services some more time after SIGINT to shutdown.
`Eventually(session, 5)` will poll the session for the specific
condition at a frequency of 10ms for at most 5 seconds.

Fixes: containers#12167
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Dec 6, 2021
Merged
@edsantiago
Copy link
Member Author

Oh noooooo... now it's flaking in the Interrupt, not the Eventually

@edsantiago edsantiago reopened this Dec 9, 2021
@vrothberg
Copy link
Member

Come on! 😆

vrothberg added a commit to vrothberg/libpod that referenced this issue Dec 10, 2021
@vrothberg
pprof CI flakes: enforce 5 seconds grace period
99bc004 
This gives the service 5 seconds to digest the signal and 5 more seconds
to shutdown.  Create a new variable to make bumping the timeout easier
in case we see re-flake in the future.

Fixes: containers#12167
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Dec 10, 2021
Merged
@Luap99
Copy link
Member

Luap99 commented Dec 13, 2021

Still flaking (I confirmed that my branch has the latest "fix"): https://storage.googleapis.com/cirrus-ci-6707778565701632-fcae48/artifacts/containers/podman/5457134111424512/html/int-podman-ubuntu-2110-rootless-host.log.html

@Luap99 Luap99 reopened this Dec 13, 2021
@vrothberg
Copy link
Member

😭 ... OK, let's bump it to 10 seconds?

vrothberg added a commit to vrothberg/libpod that referenced this issue Dec 14, 2021
@vrothberg
pprof flakes: bump timeout to 20 seconds
1fff76c 
This is the third and hopefully the last attempt to address the flakes
in the pprof tests.  We first bumped the timeouts to 2 seconds, then to
5, and since I am running out of ideas let's bump it now to 20 seconds.

Since the timeouts poll, the tests will terminate much earlier but 20
seconds should now really be enough even under highly loaded CI VMs.

Fixes: containers#12167
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Dec 14, 2021
Merged
@vrothberg
Copy link
Member

Bumped to 20 in #12588

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jwhonce jwhonce

Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless
Projects
None yet
Milestone
No milestone
4 participants
@edsantiago @jwhonce @vrothberg @Luap99