Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FuzzTest and use it to fuzz types #7246

Merged
merged 3 commits into from
Jan 29, 2025
Merged

Add FuzzTest and use it to fuzz types #7246

merged 3 commits into from
Jan 29, 2025

Conversation

tlively
Copy link
Member

@tlively tlively commented Jan 29, 2025

FuzzTest is a state-of-the-art fuzzing framework. It supports writing
property-based fuzz targets very similar to googletest unit tests, where
the framework provides the arguments to the test function drawn from a
given domain. These fuzz tests are run continuously for one second each
when running the normal googletest unit tests, but FuzzTest also
supports building in "fuzzing mode" where the tests can be run
continuously with coverage-guided mutations until they find a bug.

Add FuzzTest as a third_party dependency that is not built by default.
To build FuzzTest and the fuzz tests that use it, set the CMake variable
BUILD_FUZZTEST=ON. To build in fuzzing mode, additionally set the
CMake variable FUZZTEST_FUZZING_MODE=ON.

One of FuzzTest's key features is its support for domain combinators,
which combine simple domains into more complex domains. For example, the
domain VariantOf(InRange(0, 10), Arbitrary<std::string>()) produces a
std::variant that either holds an integer between 0 and 10 or an
arbitrary string. The set of available combinators is powerful enough to
build domains for arbitrarily structured types.

Use domain combinators to define a domain of WebAssembly type
definitions. The implementation of this domain follows the same general
structure as the existing heap type fuzzer: it chooses the size of rec
groups, then it chooses the supertypes and hierarchies for all the
definitions, then it generates the particular definitions. The
difference is that all random choices are made by the FuzzTest framework
rather than our own code. Whenever the domains of future choices will
depend on the outcome of the current choice, we use the FlatMap
combinator to make a choice from the current domain, then pass it to a
continuation that finishes constructing the final domain of types. This
leads to strange continuation-passing code, but allows us to recursively
construct the domain of type definitions.

The current implementation of the type definition domain is not ideal:
the tree of choices used to produce a particular set of type definitions
is deeper and narrower than it could be. Since a mutation of one choice
in the tree requires regenerating and changing the subtree of choices
rooted at the changed choice, having a narrower tree than necessary
means that small mutations are not as diverse as they could be and
having a deeper tree means that many mutations are larger than they
could be. The quality of the domain construction will be improved in the
future.

FuzzTest is a state-of-the-art fuzzing framework. It supports writing
property-based fuzz targets very similar to googletest unit tests, where
the framework provides the arguments to the test function drawn from a
given domain. These fuzz tests are run continuously for one second each
when running the normal googletest unit tests, but FuzzTest also
supports building in "fuzzing mode" where the tests can be run
continuously with coverage-guided mutations until they find a bug.

Add FuzzTest as a third_party dependency that is not built by default.
To build FuzzTest and the fuzz tests that use it, set the CMake variable
`BUILD_FUZZTEST=ON`. To build in fuzzing mode, additionally set the
CMake variable `FUZZTEST_FUZZING_MODE=ON`.

One of FuzzTest's key features is its support for domain combinators,
which combine simple domains into more complex domains. For example, the
domain `VariantOf(InRange(0, 10), Arbitrary<std::string>())` produces a
std::variant that either holds an integer between 0 and 10 or an
arbitrary string. The set of available combinators is powerful enough to
build domains for arbitrarily structured types.

Use domain combinators to define a domain of WebAssembly type
definitions. The implementation of this domain follows the same general
structure as the existing heap type fuzzer: it chooses the size of rec
groups, then it chooses the supertypes and hierarchies for all the
definitions, then it generates the particular definitions. The
difference is that all random choices are made by the FuzzTest framework
rather than our own code. Whenever the domains of future choices will
depend on the outcome of the current choice, we use the `FlatMap`
combinator to make a choice from the current domain, then pass it to a
continuation that finishes constructing the final domain of types. This
leads to strange continuation-passing code, but allows us to recursively
construct the domain of type definitions.

The current implementation of the type definition domain is not ideal:
the tree of choices used to produce a particular set of type definitions
is deeper and narrower than it could be. Since a mutation of one choice
in the tree requires regenerating and changing the subtree of choices
rooted at the changed choice, having a narrower tree than necessary
means that small mutations are not as diverse as they could be and
having a deeper tree means that many mutations are larger than they
could be. The quality of the domain construction will be improved in the
future.
@tlively tlively requested review from kripken and dschuff January 29, 2025 00:48
if(NOT APPLE AND NOT "${CMAKE_CXX_FLAGS}" MATCHES "-fsanitize")
# This flag only applies to shared libraries so don't use add_link_flag
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-undefined")
endif()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I ran into this in a local experiment with libFuzzer too...

[](HeapType ht) { return HeapType(ht.getBasic(Shared)); },
[](HeapType ht) {
return ht.isShared()
? std::optional{std::tuple{HeapType(ht.getBasic(Unshared))}}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need std::tuple here? I see it in the example but that uses a tuple of integers for a complex number.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, from the docs: "Note that std::tuple is necessary even if the mapping function has a single parameter."

That said, none of the rest of the domain is reversible since FlatMap currently cannot be reversible, so making this particular map a ReversibleMap is not very useful.

Copy link
Member

@kripken kripken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very interesting way to generate fuzz content...

Any findings from it?

@tlively
Copy link
Member Author

tlively commented Jan 29, 2025

No, fortunately/unfortunately our type system is already well-fuzzed, so this didn't find any new bugs. I was able to find bugs I introduced, of course.

@tlively tlively enabled auto-merge (squash) January 29, 2025 20:20
@tlively tlively disabled auto-merge January 29, 2025 21:47
@tlively tlively merged commit 30b7241 into main Jan 29, 2025
14 of 15 checks passed
@tlively tlively deleted the fuzztest branch January 29, 2025 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants