-
Notifications
You must be signed in to change notification settings - Fork 756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add FuzzTest and use it to fuzz types #7246
Conversation
FuzzTest is a state-of-the-art fuzzing framework. It supports writing property-based fuzz targets very similar to googletest unit tests, where the framework provides the arguments to the test function drawn from a given domain. These fuzz tests are run continuously for one second each when running the normal googletest unit tests, but FuzzTest also supports building in "fuzzing mode" where the tests can be run continuously with coverage-guided mutations until they find a bug. Add FuzzTest as a third_party dependency that is not built by default. To build FuzzTest and the fuzz tests that use it, set the CMake variable `BUILD_FUZZTEST=ON`. To build in fuzzing mode, additionally set the CMake variable `FUZZTEST_FUZZING_MODE=ON`. One of FuzzTest's key features is its support for domain combinators, which combine simple domains into more complex domains. For example, the domain `VariantOf(InRange(0, 10), Arbitrary<std::string>())` produces a std::variant that either holds an integer between 0 and 10 or an arbitrary string. The set of available combinators is powerful enough to build domains for arbitrarily structured types. Use domain combinators to define a domain of WebAssembly type definitions. The implementation of this domain follows the same general structure as the existing heap type fuzzer: it chooses the size of rec groups, then it chooses the supertypes and hierarchies for all the definitions, then it generates the particular definitions. The difference is that all random choices are made by the FuzzTest framework rather than our own code. Whenever the domains of future choices will depend on the outcome of the current choice, we use the `FlatMap` combinator to make a choice from the current domain, then pass it to a continuation that finishes constructing the final domain of types. This leads to strange continuation-passing code, but allows us to recursively construct the domain of type definitions. The current implementation of the type definition domain is not ideal: the tree of choices used to produce a particular set of type definitions is deeper and narrower than it could be. Since a mutation of one choice in the tree requires regenerating and changing the subtree of choices rooted at the changed choice, having a narrower tree than necessary means that small mutations are not as diverse as they could be and having a deeper tree means that many mutations are larger than they could be. The quality of the domain construction will be improved in the future.
if(NOT APPLE AND NOT "${CMAKE_CXX_FLAGS}" MATCHES "-fsanitize") | ||
# This flag only applies to shared libraries so don't use add_link_flag | ||
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-undefined") | ||
endif() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I ran into this in a local experiment with libFuzzer too...
[](HeapType ht) { return HeapType(ht.getBasic(Shared)); }, | ||
[](HeapType ht) { | ||
return ht.isShared() | ||
? std::optional{std::tuple{HeapType(ht.getBasic(Unshared))}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need std::tuple
here? I see it in the example but that uses a tuple of integers for a complex number.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, from the docs: "Note that std::tuple is necessary even if the mapping function has a single parameter."
That said, none of the rest of the domain is reversible since FlatMap currently cannot be reversible, so making this particular map a ReversibleMap is not very useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very interesting way to generate fuzz content...
Any findings from it?
No, fortunately/unfortunately our type system is already well-fuzzed, so this didn't find any new bugs. I was able to find bugs I introduced, of course. |
FuzzTest is a state-of-the-art fuzzing framework. It supports writing
property-based fuzz targets very similar to googletest unit tests, where
the framework provides the arguments to the test function drawn from a
given domain. These fuzz tests are run continuously for one second each
when running the normal googletest unit tests, but FuzzTest also
supports building in "fuzzing mode" where the tests can be run
continuously with coverage-guided mutations until they find a bug.
Add FuzzTest as a third_party dependency that is not built by default.
To build FuzzTest and the fuzz tests that use it, set the CMake variable
BUILD_FUZZTEST=ON
. To build in fuzzing mode, additionally set theCMake variable
FUZZTEST_FUZZING_MODE=ON
.One of FuzzTest's key features is its support for domain combinators,
which combine simple domains into more complex domains. For example, the
domain
VariantOf(InRange(0, 10), Arbitrary<std::string>())
produces astd::variant that either holds an integer between 0 and 10 or an
arbitrary string. The set of available combinators is powerful enough to
build domains for arbitrarily structured types.
Use domain combinators to define a domain of WebAssembly type
definitions. The implementation of this domain follows the same general
structure as the existing heap type fuzzer: it chooses the size of rec
groups, then it chooses the supertypes and hierarchies for all the
definitions, then it generates the particular definitions. The
difference is that all random choices are made by the FuzzTest framework
rather than our own code. Whenever the domains of future choices will
depend on the outcome of the current choice, we use the
FlatMap
combinator to make a choice from the current domain, then pass it to a
continuation that finishes constructing the final domain of types. This
leads to strange continuation-passing code, but allows us to recursively
construct the domain of type definitions.
The current implementation of the type definition domain is not ideal:
the tree of choices used to produce a particular set of type definitions
is deeper and narrower than it could be. Since a mutation of one choice
in the tree requires regenerating and changing the subtree of choices
rooted at the changed choice, having a narrower tree than necessary
means that small mutations are not as diverse as they could be and
having a deeper tree means that many mutations are larger than they
could be. The quality of the domain construction will be improved in the
future.