Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporarily change test_randint_randomness not to run #1672

Closed
stress-tess opened this issue Aug 10, 2022 · 2 comments · Fixed by #1673
Closed

Temporarily change test_randint_randomness not to run #1672

stress-tess opened this issue Aug 10, 2022 · 2 comments · Fixed by #1673
Assignees

Comments

@stress-tess
Copy link
Member

test_randint_randomness has been failing in CI runs a bit too frequently. We should lower the threshold for now and decide how we want to proceed (either relying solely on randtest after #1665 or if we are okay with the lower confidence test)

@stress-tess stress-tess self-assigned this Aug 10, 2022
@stress-tess
Copy link
Member Author

stress-tess commented Aug 10, 2022

Upon thinking about this a bit more, I think we should actually not run the test for now.

I did some quick back of the envelope math (which should def be double checked because i'm very bad at stats):
if provides a 95% confidence. So I think that if randint is truly random, then every trial has a 95% chance of passing and 5% chance of failing. Since the test should be 20 (i think independent) trials, this is a Bernoulli experiment with $n = 20$ and $p = 0.95$

So the probability of getting $k$ successes is
$$P(X= k) = { n \choose k}p^k q^{n-k}$$

Since the overall test only fails if we have >4 failures, to calculate the odds of overall success we can sum the probabilities of passing 16, 17, 18, 19, and 20 trials (since these are all independent events)
$$\sum_{i=16}^{20}{20 \choose i}0.95^{i} 0.05^{20-i} = 0.997426$$

so the odds of failure is $1-$ this and should be $0.0025739$. The overall test should only fail $0.25$% of the time... So I'm thinking there is something wrong with randint given the frequency of failures

@Ethan-DeBandi99
Copy link
Contributor

Upon thinking about this a bit more, I think we should actually not run the test for now.

I did some quick back of the envelope math (which should def be double checked because i'm very bad at stats): if provides a 95% confidence. So I think that if randint is truly random, then every trial has a 95% chance of passing and 5% chance of failing. Since the test should be 20 (i think independent) trials, this is a Bernoulli experiment with n=20 and p=0.95

So the probability of getting k successes is P(X=k)=(nk)pkqn−k

Since the overall test only fails if we have >4 failures, to calculate the odds of overall success we can sum the probabilities of passing 16, 17, 18, 19, and 20 trials (since these are all independent events) ∑i=1620(20i)0.95i0.0520−i=0.997426

so the odds of failure is 1− this and should be 0.0025739. The overall test should only fail 0.25 of the time... So I'm thinking there is something wrong with randint given the frequency of failures

@pierce314159 - based on this, I am inclined to agree. We should probably pull the test for now.

@stress-tess stress-tess changed the title Lower threshold of test_randint_randomness Temporarily change test_randint_randomness not to run Aug 10, 2022
stress-tess pushed a commit to stress-tess/arkouda that referenced this issue Aug 10, 2022
…t to run

This PR (Closes Bears-R-Us#1672):
- Changes the name of `test_randint_randomness` to `randint_randomness` which will cause the test to not run
Ethan-DeBandi99 pushed a commit that referenced this issue Aug 10, 2022
…1673)

This PR (Closes #1672):
- Changes the name of `test_randint_randomness` to `randint_randomness` which will cause the test to not run

Co-authored-by: Pierce Hayes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants