Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests involve fragile RNG that fails in some contexts #42

Closed
ExpandingMan opened this issue Oct 1, 2021 · 2 comments
Closed

tests involve fragile RNG that fails in some contexts #42

ExpandingMan opened this issue Oct 1, 2021 · 2 comments
Labels
bug Something isn't working tests

Comments

@ExpandingMan
Copy link
Contributor

I get the following failures on Pkg.test("ReTest")

MiscSeed: Test Failed at /home/expandingman/.julia/dev/ReTest/test/runtests.jl:878
  Expression: MiscSeed.RAND1 === MiscSeed.RAND2 === rands[2]
   Evaluated: 0x3214 === 0x3214 === 0x837e
Stacktrace:
 [1] macro expansion
   @ ~/.julia/dev/ReTest/test/runtests.jl:878 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/ReTest/test/setup.jl:71 [inlined]
 [3] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [4] top-level scope
   @ ~/.julia/dev/ReTest/test/setup.jl:71
                  Pass
Main.MiscSeed |      0
MiscSeed: Test Failed at /home/expandingman/.julia/dev/ReTest/test/runtests.jl:880
  Expression: MiscSeed.RAND1 === MiscSeed.RAND2 === rands[2]
   Evaluated: 0x24ae === 0x24ae === 0x837e
Stacktrace:
 [1] macro expansion
   @ ~/.julia/dev/ReTest/test/runtests.jl:880 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/ReTest/test/setup.jl:71 [inlined]
 [3] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [4] top-level scope
   @ ~/.julia/dev/ReTest/test/setup.jl:71

However, this doesn't seem to happen in CI/CD, see #41 which has exactly the same tests as I ran to get this but succeeds.

It looks like something needs to be seeded somewhere, but as this seems to be checking something rather tricky I'm not sure where would be appropriate to set the seed.

@rfourquet
Copy link
Collaborator

rfourquet commented Nov 9, 2021

The problem is that we test against different versions of the global RNG, and the hardcoded values correspond to an obsolete version of the RNG on what was nightly, which does not show as failing on CI because of the useless continue-on-error option, which shows all-green although the test actually failed.

@rfourquet rfourquet added the tests label Nov 9, 2021
@rfourquet rfourquet added the bug Something isn't working label Nov 9, 2021
@ExpandingMan
Copy link
Contributor Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working tests
Projects
None yet
Development

No branches or pull requests

2 participants