Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: increase timeouts for tests that are flaky on slow CI #2450

Merged
merged 3 commits into from
Jul 3, 2024

Conversation

Frando
Copy link
Member

@Frando Frando commented Jul 3, 2024

Description

We have some tests that use timeouts to not wait infinitely for an event that might not be coming. These tests are flaky if the timeout is too low, especially on windows and likely if the machines are overworked. This PR increases these timeouts:

Breaking Changes

Notes & open questions

Change checklist

  • Self-review.

Copy link
Contributor

@dignifiedquire dignifiedquire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe log the time it took, so we can see a little better if this was really the issue and what variance is in these

@Frando
Copy link
Member Author

Frando commented Jul 3, 2024

maybe log the time it took, so we can see a little better if this was really the issue and what variance is in these

Sure, we can add that, but will we ever read the logs of successful CI runs for these? Because logging the time doesn't help on the failed runs, right? We'd need something like "CI metrics collection" to actually make use of these numbers. For general variance, I'd say total test runtime is already good enough likely.

@flub
Copy link
Contributor

flub commented Jul 3, 2024

maybe log the time it took, so we can see a little better if this was really the issue and what variance is in these

Sure, we can add that, but will we ever read the logs of successful CI runs for these? Because logging the time doesn't help on the failed runs, right? We'd need something like "CI metrics collection" to actually make use of these numbers. For general variance, I'd say total test runtime is already good enough likely.

Yeah, I agree this is not terribly effective. I don't care about it really.

@dignifiedquire
Copy link
Contributor

Because logging the time doesn't help on the failed runs, right?

well I would look at it retrospectively, if this is close to the timeout often, or not

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 3, 2024
@Frando Frando enabled auto-merge July 3, 2024 10:19
@Frando Frando added this pull request to the merge queue Jul 3, 2024
Merged via the queue into main with commit cc30743 Jul 3, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
3 participants