Add config to abort on first fail when performing fuzzing

[xenoliss]

Component

Other (please describe)

Describe the feature you would like

When writing fuzzing tests I often find myself waiting for the all fuzzing to end before being able to see the "bug". It sort of make the debug => fix => run test => debug => fix => run test process quite painfull when working on a failing test case with fuzzing.

Would be super cool to be able to specify somehow that I'd like the fuzzer to abort fuzzing when a test fails. Could be in the command line as an additional parameter or in the [fuzz] section of the foundry.toml file.

Additional context

No response

[mds1]

It sort of make the debug => fix => run test => debug => fix => run test process quite painfull when working on a failing test case with fuzzing

You can run forge test --match-test test_MyTest to only run the failing case which makes the process easier. Can also just Ctrl + C to stop the process once there's a failing run. Going to close this since this feels unnecessary but let me know if I misunderstood 🙂

[xenoliss]

Hey, what I meant is, not related to test selection. Say I'm using --mt to only run a specific fuzz test, the problem is, if run number 34 fails, I still have to wait for the entire fuzz.runs campaign to be ran before seing the failing test (which can take several minutes for complexe test suite with ffi etc.)

[mds1]

If you run a single test and run number 34 of that test fails, the test does fail immediately and no other "normal" fuzz runs are ran. I say "normal" because at that point shrinking occurs, where the test is run more times to find the smallest fuzz input value(s) that still trigger the failure. Perhaps the feature request here is to turn off shrinking to see failures faster? You could probably argue the shrinked values would make debugging faster so it's worth the wait, so I'm curious if a feature to turn off shrinking is useful

[grandizzy]

I am experiencing same issues when running large invariant test suite (bunch of invariants + runs + big depth), would be great to have the option to turn off shrinking. Thank you

[mds1]

@grandizzy created #4867 to track