Tribble

Have a jvm application with an open socket? Need to test your application against un-expected input? Tribble can help generating and testing un-expected input and find out what happens.

Tribble is an easy to use fuzz testing tool for java applications using coverage to guide the process. Heavily based on the wonderful GoFuzz and AFL.

It uses Jacoco to get coverage stats and has a maven plug for running.

• Usage
  • Maven
  • SBT
  • Command line
  • Example
  • General
• Why
  • What is fuzz testing?
  • Why the name Tribble?
• Settings
• Design Goals
• Where Next?
• Contributing
• Contributors
• Thanks

Usage

Maven

• Add the JCenter maven repository to your pom if needed. You will need both repository and pluginRepository entries

<repositories>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray</name>
        <url>http://jcenter.bintray.com</url>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray-plugins</name>
        <url>http://jcenter.bintray.com</url>
    </pluginRepository>
</pluginRepositories>

• Include tribble-core as a dependency.

<dependency>
    <groupId>org.catapult.sa</groupId>
    <artifactId>tribble-core_2.12</artifactId>
    <version>0.8.0</version>
    <classifier>shadow</classifier>
    <scope>test</scope>
</dependency>

• Add the plugin to the build section of your pom.

<plugin>
    <groupId>org.catapult.sa</groupId>
    <artifactId>tribble-maven-plugin_2.12</artifactId>
    <version>0.8.0</version>
    <configuration>
        <target>org.catapult.sa.testcase.TestCase</target>
    </configuration>
</plugin>

• Create a class that implements FuzzTest which will run a single test case with the provided data. This class should return a FuzzResult of OK if the test ran well or throws an exception the run is considered to be a failure. This class should be in the src/test tree
• Configure the target class name in the plugin.
• Create a folder called corpus and populate it with an initial set of inputs that will exercise different functions in your application. The more the merrier.
• Run mvn tribble:fuzztest to start a run.

SBT

• add JCenter to dependency resolvers

resolvers += Resolver.jcenterRepo

• add dependency

"org.catapult.sa" %% "tribble-core" % "0.8.0" % "test" classifier "shadow"

• Create a class that implements FuzzTest which will run a single test case with the provided data. This class should return a FuzzResult of OK if the test ran well or throws an exception the run is considered to be a failure. This class should be in the src/test tree
• Configure the target class name in the plugin.
• Create a folder called corpus and populate it with an initial set of inputs that will exercise different functions in your application. The more the merrier.
• Run sbt test:runMain org.catapult.sa.tribble.App -targetClass <Path to your target class> to start a run. See the section with command line parameters for more options.

Command Line

• Include this library in your project.
• Create a class that implements FuzzTest which will run a single test case with the provided data. If this class returns false or throws an exception the run is considered to be a failure.
• Create a folder called corpus and populate it with an initial set of inputs that will exercise different functions in your application. The more the merrier.
• Run org.catapult.sa.tribble.App passing --targetClass to set where your implementation of FuzzTest lives and the required class path.

Example Test Class

import org.catapult.sa.tribble.{FuzzResult, FuzzTest}

class TestCase extends FuzzTest {

  def test(data : Array[Byte]): FuzzResult = {
    Fish.wibble(data)
    FuzzResult.OK
  }
}

General Usage

Stats will be printed to stderr at regular intervals. A folder called failed will be created on startup and populated with both inputs and stack traces that they cause. The file names will be the md5 hash of the input. Inputs which generate new code paths will be saved to the corpus directory with a file name that is an md5 hash of the code coverage stats.

Try and avoid printing to stdout and stderr in your test cases this can slow things down quite a lot. There is a one second time out on each test run.

When creating corpus entries you can append .hex to the file name and have the file hex encoded rather than raw bytes. This can make it a lot easier to include non printing characters. Generated corpus entries and failed inputs will use this if it finds unprintable characters in an input array.

If you need a particular version of scala we provide builds against 2.12. Change the bit on the end of the group name If you need something else edit the gradle.properties file and then run ./gradlew install

Why?

Because we needed one, you probably do too. If you don't think you do you don't have enough tests yet.

What is fuzz testing?

Fuzz testing (or smoke testing) is the process of sending "random" data (we'll come back to this in a bit) into a system and seeing what happens. The process of doing this has been around for a long time. However most of the time the input is generated from known input and has no way of knowing what happened inside the system under test. Systems like AFL and GoFuzz are able to instrument the system and know when they have found new code paths. This enables them to further modify the input data to work their way through the application.

Using systems like this is more effective and efficient that just sending random data into the system. Unfortunately such systems either don't work with Java/the JVM or have to start the application externally. With the JVM this introduces a significant slowdown as the optimisations and compilations from previous runs are lost.

The GoFuzz tool allows the creation of a simple test function which is used as the basis of the test. Tribble used a similar approach and is based heavily on the work of GoFuzz and AFL, although not yet as advanced. This allows the tests to be run with out restarting the JVM, which makes things a lot faster. However it does mean that if you find a JVM crashing bug you will lose the node. We can recover from most out of memory errors. We use Jacoco to generate code coverage stats for a run, which lets us work out the paths through the application we have found.

Why the name Tribble?

Lots of small fuzzy creatures from Star trek. It seemed like a reasonable name when I started this.

"Letting engineers name things is like letting the marketing department build them"

"Advanced" settings

There are some extra settings you may wish to use if needed. They are accessible either from the maven plugin or the command line.

Tag	Parameter	Description	Default
target	--targetClass	Fully qualified class name of a class which implements org.catapult.sa.tribble.FuzzTest
corpusPath	--corpus	Path to the corpus folder	corpus
failedPath	--failed	Path to the failed folder	failed
threads	--threads	Number of threads to use	2
timeout	--timeout	Time out for an individual test run in milliseconds	1000
count	--count	number of iterations to run. Number greater than zero for a limit	-1
verbose	--verbose	Should verbose mutator stats be printed.	false
ignoreClasses		List of class prefixes e.g org.apache.hadoop. to ignore from coverage. This is useful if you find some classes are already loaded.
	--ignoreClasses	Comma Separated list of class prefixes to ignore. e.g org.apache.hadoop.
disabledMutators		List of mutator class names to disable
	--disabledMutations	Comma separated list of mutators to disable

Extensions

Custom mutators

It is possible to add extra mutators using a plugin like system. Create a class that extends the org.catapult.sa.tribble.mutators.Mutator trait. The list the class in a file called org.catapult.sa.tribble.mutators The default mutation engine will look for files with that name that contain lists of classes to use as mutators.

Extra keywords

If you are fuzzing a system which takes defined keywords it can take a while for the system to come up with them on its own. There is a keyword mutator, which can be extended by adding your own file called org.catapult.sa.tribble.keywords with one entry per line. On start up the keyword mutator will search the class path for files with that name.

Design goals:

1. Easy to use and setup
2. Easy to develop and extent
3. Fast

Where goals conflict the higher design goal should take precedence. Code that is harder to update but faster should be used with extreme caution.

Where next?

See the Issues list.

Contributing

See the CONTRIBUTING.md file and the Code of conduct

Contributors

• Emily Selwood
• Cameron Shiell
• Chen Chenglong
• Keegan Neave
• DittyTwo

Thanks

This project owes a huge thank you to the AFL project and go-fuzz for the ideas. JaCoCo for the code coverage tool this is built on top of.