Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shuffle tests #15581

Merged
merged 10 commits into from
Dec 22, 2023
Merged

Shuffle tests #15581

merged 10 commits into from
Dec 22, 2023

Conversation

guillep
Copy link
Member

@guillep guillep commented Nov 27, 2023

Add test shuffling during run to guarantee that tests run in any order.
The shuffle seed is configurable in the test suite and as a command line argument when running tests from the command line

@guillep guillep requested a review from tesonep November 27, 2023 12:35
Copy link
Collaborator

@tesonep tesonep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be nice to add the description of the new command line parameter in the Comments of the TestCommandLineHandler class. The comment of the class is used by the help option

@jecisc
Copy link
Member

jecisc commented Nov 27, 2023

I can predict that at least the failing tests of this PR will fail randomly: #13349

:p

@tesonep tesonep added Status: Need more work The issue is nearly ready. Waiting some last bits. and removed Status: Tests passed please review! labels Nov 28, 2023
@tesonep
Copy link
Collaborator

tesonep commented Nov 28, 2023

The error is related with the change, windows passed because it did not run the tests.... There is another issue there. I think the windows issue is not related to the PR

@tesonep
Copy link
Collaborator

tesonep commented Nov 28, 2023

When running the tests in the small image it fails to initialize the Random seed from the clock. Maybe for the tests in the small image we can provide a seed calculated from outside.

@guillep
Copy link
Member Author

guillep commented Nov 28, 2023

When running the tests in the small image it fails to initialize the Random seed from the clock. Maybe for the tests in the small image we can provide a seed calculated from outside.

Looks like DoubleWordArray has a wrong format in the minimal image?

@guillep guillep marked this pull request as draft December 4, 2023 14:28
@guillep
Copy link
Member Author

guillep commented Dec 5, 2023

imagen

Shuffling tests is here!

@guillep guillep marked this pull request as ready for review December 5, 2023 07:28
@jecisc
Copy link
Member

jecisc commented Dec 5, 2023

If we shuffle test it would be good to generate a way to reproduce a specific order. Like this, if there is a failure that depend on the execution order of tests, we could have a way to reproduce it. Else we might never be able to get in the same situation ever again.

For example by printing the seed and the command line to execute to run tests with this seed?

@guillep
Copy link
Member Author

guillep commented Dec 5, 2023

well, that is already in the PR, what do you think is missing?

@guillep guillep merged commit 5e1afdd into pharo-project:Pharo12 Dec 22, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Need more work The issue is nearly ready. Waiting some last bits.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants