Meta issue for fuzzing #972
Labels
Comments
bors-servo
pushed a commit
that referenced
this issue
Nov 30, 2017
Property testing with quickcheck This PR represents an attempt to address issue #970. It also represents a portion of the meta issue for fuzzing #972. The code base reflected here uses quickcheck to generate C headers that include a variety of types including basic types, structs, unions, function prototypes and function pointers. The headers generated by quickcheck are passed to the `csmith-fuzzing/predicate.py` script. Examples of headers generated by this iteration of the tooling can be viewed [here](https://gist.github.com/snewt/03ce934f35c5b085807d2d5cf11d1d5c). At the top of each header are two simple struct definitions, `whitelistable` and `blacklistable`. Those types are present in the vector that represents otherwise primitive types used to generate. They represent a naive approach to exposing custom types without having to intuit generated type names like `struct_21_8` though _any actual whitelisting logic isn't implemented here_. Test success is measured by the success of the `csmith-fuzzing/predicate.py` script. This means that for a test to pass the following must be true: - bindgen doesn't panic - the resulting bindings compile - the resulting bindings layout tests pass #### Usage ```bash cd tests/property_test cargo test ``` Some things I'm unsure of: #### Where should this feature live? At the moment it lives in `tests/property_test` but isn't run when `cargo test` is invoked from bindgen's cargo manifest directory. #### What's an acceptable ammount of time for these tests to take? At this point, the source is genereated in ~1 second but the files are large enough that it takes the `predicate.py` script ~30 seconds to run through each one. In order for the tests to run in under a minute only 2 are generated by quickcheck by default. This can be changed in the `test_bindgen` function of the `tests/property_test/tests/fuzzed-c-headers.rs` file. #### How do we expose the generated code for easy inspection? For now the `run_predicate_script` function in the `tests/property_test/tests/fuzzed-c-headers.rs` file contains a commented block that will copy generated source in the `tests/property_test/tests` directory. Should it be easier? #### Special casing There is some logic in the fuzzer that disallows 0 sized arrays because tests will regulary fail due to issues documented in #684 and #1153. Should this be special casing? #### Does the fuzzer warrant its own crate? After any iterations the reviewers are interested in required to make this a functional testing tool, should/could the fuzzing library be made into its own crate? I didn't move in that direction yet because having it all in one place seemed like the best way to figure out what works an doesn't but I'm interested in whether it might be useful as a standalone library. #### What does it look like to expose more useful functionality? I'm looking forward to feedback on how to make this a more useful tool and one that provides the right configurability. Thanks! r? @fitzgen
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a meta issue tracking the various things we can do to fuzz bindgen.
quickcheck: Add property based testing with quickcheck #970cargo fuzz: Investigate fuzzing withcargo fuzz#971The text was updated successfully, but these errors were encountered: