Create an RNG test suite

[johnmyleswhite]

We have come across several bugs in our RNG's lately: randg() was broken when the input parameter was 1.0, while randexp() is broken without any input parameters. To verify the quality of RNG's, we need to set up a probabilistic test suite that will be very compute-intensive.

At an abstract level, the suite will just contain null hypothesis tests. For example,

• Is the mean value of 10,000,000 samples within a reasonable region around the theoretical mean?
• Are successive samples plausibly IID?

In practice, this can be broken into two parts:

• One part will use the Crush tests from TestU01 to assess the quality of low-level random number generators. This runs a large series of tests and should work for any RNG for which we have a high-precision quantile function. This involves an external library.
• The other part will test much more basic properties of the RNG outputs using pure Julia code. We can confirm that the mean of a large number samples from an RNG is near the expected mean or that the samples are all in the support of the distribution. These tests are easy to build and will allow us to insure that future changes to the RNG's do produce samples that are clearly biased. This can written already using pure Julia code.

This test suite cannot become part of the main tests, because occasionally failures will happen. We can minimize the chance of failures by using very large sample sizes, but that will require that the tests run for a long time.

[pao]

This pairs well with #915, I think. It's not a hard dependency, but it would allow this to be a part of the standard (though not precommit) tests, since RNG goodness can become a metric tracked with the evolution of the codebase rather than a pass/fail.

[johnmyleswhite]

Agreed. At this point, I think the RNG's have two continuous metrics we can track over time:

• Number of RNG tests passed
• Speed of RNG sample generation

[johnmyleswhite]

Sketched out the type of thing we might do: https://github.com/johnmyleswhite/rng_tests.jl

The Gamma and Exponential results seem broken. Not yet sure whether I'm doing the math wrong or whether there's a bug in their RNG's.

[ViralBShah]

Also see this:
https://github.com/andreasnoackjensen/rngtest.jl

We should have a completely separate repository for all RNG testing that use external libraries. This can be hosted under juliaLang. The basic statistical tests that can be written as pure julia code for unit/sanity testing should certainly be in the julia tests.

We also have the potential new issue of the collision test with randn().

[andreasnoack]

Some comments:

Julia's random normals passed all tests in Crush, so the Collision problem that Doornik wrote about didn't apply for my specific run. When I am back at work tomorrow I'll run the BigCrush with a new seed.

The SmallCrush has a version that works for a file of random numbers. We could use that to test the different distribution generated within Julia. For example for the cases where the error is in the julia code and a pure C test does not apply. I guess this is the easiest and safest as the tests are standard and well tested. Also, I think it would be a good choice to use the Rmath cdfs to convert the numbers to uniformly distributed variables. They should be thoroughly tested.

[ViralBShah]

I think that https://github.com/andreasnoackjensen/RNGTest.jl/ sufficiently addresses this.