Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

forAll sample size incorrectly set to constant 1 #136

Closed
cihleman3 opened this issue Oct 1, 2021 · 5 comments · Fixed by #137
Closed

forAll sample size incorrectly set to constant 1 #136

cihleman3 opened this issue Oct 1, 2021 · 5 comments · Fixed by #137

Comments

@cihleman3
Copy link

PropF.forAll { (s: String) => does not increase the size of the strings it is testing: it only uses strings of length 0 or 1.
(The same holds for lists and presumably for alle "size-sensitive" data, i.e., not numbers.)
The apparent reason is that the loop inside of check implementing the quantifier uses a constant size of 1:
val size = minSize.toDouble + sizeStep.

@arosien
Copy link
Contributor

arosien commented Oct 1, 2021

I think you mean that the sample size is incorrectly set to be constant (size 1!). Can you update the description? Saying "does not increase" implies the sample size increases, which is not technically correct; the sample size bounds are set (incorrectly!) and do not vary.

@arosien
Copy link
Contributor

arosien commented Oct 1, 2021

Re: constant size:

val sizeStep = (maxSize - minSize) / minSuccessfulTests
...
val size = minSize.toDouble + sizeStep

The default parameters are:

  • maxSize = 100
  • minSize = 0
  • minSuccessfulTests = 100
    therefore size is set to 1.

@cihleman3 cihleman3 changed the title forAll does not increase the size of the examples it is testing. forAll sample size incorrectly set to constant 1 Oct 1, 2021
@cihleman3
Copy link
Author

Ok, is it clearer now?

@arosien
Copy link
Contributor

arosien commented Oct 1, 2021

Looking at the impl which computes this stepSize, it's not clear to me what it is for. Couldn't the size parameters be unmodified from their defaults, which would avoid the bug?

@mpilquist
Copy link
Member

Wow, thanks for finding this one. I'm amazed we haven't noticed this behavior until now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants