You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Swarm testing is way to improve the diversity of generated test cases. Instead of potentially including all features in every test case, a large “swarm” of randomly generated configurations is used, each of which omits some features. ... First, some features actively prevent the system from executing interesting behaviors; e.g., pop calls may prevent an overflow bug from executing. Second, test features compete for space in each test, limiting the depth to which logic driven by features can be explored. Experimental results show that swarm testing increases coverage and can improve fault detection dramatically.
This is at minimum worth evaluating for Hypothesis - stateful testing would be a close match to the paper, while the benefits of swarm testing for data generation in general are less well explored but potentially substantial for e.g. Unicode text. The latter would require exposing data structure to the Conjecture engine as proposed in e.g. #1621.
I've thought a bit about how to do shrinker friendly swarm testing under the current model. It's actually not too bad, though I haven't actually tried it so there's probably some annoying bits that I'm missing. The main points are:
You need swarm flags to "shrink open" so that once the shrinker has run to completion, all flags are enabled. e.g. you could do this by generating a set of banned flags.
You need to use rejection sampling rather than anything more clever.
Basically if you can then do characters().filter(character_class_is_enabled), then what happens during shrinking is follows:
We delete all of the initial iterations of the loop. Now, as if by magic, we just happen to have picked only values that are enabled.
The flags now shrink open, so we've left swarm mode and everything is now enabled.
We can now shrink the values as normal characters.
[To ensure that choosing swarm flags works well with the shrinker:] ensure that every time we check whether a flag is enabled, if it's already been set we call data.write to record the flag in the data stream, so that if we delete the first use the subsequent uses turn into an initialisation.
The text was updated successfully, but these errors were encountered:
To paraphrase Swarm Testing (Groce et al, 2012),
This is at minimum worth evaluating for Hypothesis - stateful testing would be a close match to the paper, while the benefits of swarm testing for data generation in general are less well explored but potentially substantial for e.g. Unicode text. The latter would require exposing data structure to the Conjecture engine as proposed in e.g. #1621.
Quoting @DRMacIver from #1401, which prompted this issue:
The text was updated successfully, but these errors were encountered: