Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Selective fuzz runs with NatSpec comments "@custom:fuzz-runs NUMBER" #4085

Closed
PaulRBerg opened this issue Jan 13, 2023 · 3 comments
Closed
Labels
A-testing Area: testing C-forge Command: forge T-feature Type: feature

Comments

@PaulRBerg
Copy link
Contributor

PaulRBerg commented Jan 13, 2023

Component

Forge

Describe the feature you would like

It would be useful to be able to set a custom number of fuzz runs for a particular test. See, for instance, the discussion about fuzzing arrays I had with @mds1 here. Array fuzzing is difficult, and the user often needs to adjust the array values to match the business logic of the protocol, e.g. sorting the array in ascending order.

Array operations are expensive and time intensive, so setting the fuzz.runs to a high value in the Foundry config is guaranteed to lead to slow test run times if the user has several tests that fuzz arrays.

A quick and dirty solution is to add a Foundry profile specifically for the fuzzed array tests - in this profile, the fuzz runs would be lower than usual, and potentially the path to test would be different (though until #3841 is implemented, this isn't practical).

A better solution would be for Foundry to provide a way to tweak the number of fuzz runs on a per-test basis, and there might be an elegant way to enable this with custom NatSpec tags. The Solidity SMTChecker currently uses the following custom tag to aid the compiler during formal verification:

/// @custom:smtchecker abstract-function-nondet

We could inspire from this and imagine a similar tag for Foundry to selectively change the fuzz runs, like this:

/// @custom:fuzz-runs 500
function test_Something(uint256 x) external {
    // ...
}

Adding the NatSpec comment above would instruct Foundry to fuzz the x argument 500 times, ignoring the fuzz.runs value from the Foundry profile only for this particular test.

@PaulRBerg PaulRBerg added the T-feature Type: feature label Jan 13, 2023
@PaulRBerg PaulRBerg changed the title Selective fuzzing with NatSpec comments "@custom:fuzz-runs NUMBER" Selective fuzz runs with NatSpec comments "@custom:fuzz-runs NUMBER" Jan 13, 2023
@mds1
Copy link
Collaborator

mds1 commented Jan 13, 2023

@PaulRBerg
Copy link
Contributor Author

Given that #4744 was merged, should we close this issue?

@mds1
Copy link
Collaborator

mds1 commented May 15, 2023

Ah yea, thanks! Closing

@mds1 mds1 closed this as completed May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testing Area: testing C-forge Command: forge T-feature Type: feature
Projects
None yet
Development

No branches or pull requests

3 participants