Tests suggestion: meta level to exercise permutations of tunables

[gdevenyi]

I noticed at #9648 (comment)

That this bug was only exposed when a tunable was flipped.

That got me thinking that perhaps there are bug lurking in all the possible combinations of tunables, or as in the case of the referenced bug, where tunables are changed during runtime (where allowable). This would likely be an immense amount of tooling, but perhaps the test suite should have the ability to exercise the tunables before/during runs to shake out any bugs. I realize this is likely to be an exponential increase in the number of tests but it might be worth doing once in a while to see what shakes out.

[PrivatePuffin]

How this is currently done in the compression/rsend suite, that multiple loops are ran, each with their own set of random options but a limited number of loops is ran.

Lets say one runs 4 loops, it's actually 4*num_distro's Which is a reasonable amount of tests.

Add to this the number of tests run a day and you would basically end up with your exponential number of permutations, without directly having to run them all each test :)

[behlendorf]

We do a little bit of this kind of testing today with ztest which modifies some default values. If ztest were rock solid today I could imagine it randomly toggling some of these settings to stress test some less used code paths.