Insanely annoying "persisted failure reverts" in Foundry test

[CH-coding-fire]

The behavior is, when "forge test --match-test invariant_mustHaveMoreValueThanTotalSupply123", then it tells me [FAIL. Reason: revert: ERC20: insufficient allowance] This is OK

If I do the same command as above, it tells me: [FAIL. Reason: invariant_mustHaveMoreValueThanTotalSupply123 persisted failure revert] This is CONFUSING as FK

I am stuck on this for HOURS. Because I was not aware of the above, that is the repeated failure will cause "persisted failure revert", I googled and ChatGPT but found NOTHING, why is that????? How should a good programmer approach this problem? Also from common sense, why don't it just tell me again [FAIL. Reason: revert: ERC20: insufficient allowance], like I think most programming tools do so there is consistency

[EngrPips]

Hello @CH-coding-fire, Please show your invariant and handler file.

[EngrPips]

Awesome, Keep crushing it.

[mustby]

Thanks guys! This saved me a lot of time...

[EngrPips]

Anytime buddy.

[yawarasuuna]

ty. forge clean saving the day again, guess I'll have to remember to run it every time after an error lol

[EngrPips]

ty. forge clean saving the day again. Guess I'll have to remember to run it every time after an error, lol

Foundry sometimes plays pranks on us, lol.

[nzes1]

Happy 2025 @CH-coding-fire ,

This issue stems from Foundry's Replaying Failures feature. According to the Foundry Book, this feature is designed to incrementally replay the last test run failures by persisting them to disk. The intention is to ensure any previously encountered failures are prioritized in subsequent test runs, helping identify and fix regressions more efficiently.

However, this behavior can sometimes feel "annoying" during development, especially if you're not expecting it.

Here's a breakdown of how it works:

How Replaying Failures Works

1. Fuzz Tests

   • All counterexamples (failures) are saved to ~/.foundry/cache/fuzz/failures.
   • On subsequent runs, these failures are replayed before new test campaigns are started.
   • Entries in this file are appended with new failures from subsequent test runs.

2. Invariant Tests

   • Failures from invariant tests are saved in individual files under:
     ~/.foundry/cache/invariant/failures/{TEST_SUITE_NAME}/{INVARIANT_NAME}.
   • Unlike fuzz tests, these files are replaced when a new counterexample is found.

How to Avoid Persisted Failures

Currently, the only way to avoid replaying failures is to manually run the following command before running your test(s):

forge clean

This clears the cache and prevents Foundry from replaying previously recorded failures.

Possible Improvements

It might be helpful for the Foundry team to introduce a configuration option in foundry.toml to enable or disable this feature dynamically. This would make it easier for users who wish to skip the replaying of failures without manually running forge clean.

Final Thoughts

While this feature is invaluable for catching regressions, it can indeed feel inconvenient in certain workflows. A toggle or more control over this behavior in foundry.toml would likely improve the developer experience significantly.

[EngrPips]

Phenomenal heads-up.