-
-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shuffle tests #15581
Shuffle tests #15581
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be nice to add the description of the new command line parameter in the Comments of the TestCommandLineHandler class. The comment of the class is used by the help option
I can predict that at least the failing tests of this PR will fail randomly: #13349 :p |
The error is related with the change, windows passed because it did not run the tests.... There is another issue there. I think the windows issue is not related to the PR |
When running the tests in the small image it fails to initialize the Random seed from the clock. Maybe for the tests in the small image we can provide a seed calculated from outside. |
Looks like |
If we shuffle test it would be good to generate a way to reproduce a specific order. Like this, if there is a failure that depend on the execution order of tests, we could have a way to reproduce it. Else we might never be able to get in the same situation ever again. For example by printing the seed and the command line to execute to run tests with this seed? |
well, that is already in the PR, what do you think is missing? |
Add test shuffling during run to guarantee that tests run in any order.
The shuffle seed is configurable in the test suite and as a command line argument when running tests from the command line