Final Submission: Fuzzing Executable Tutorial

[bpalagi]

Assignment Proposal

Title

Katacoda Executable Tutorial demonstrating and explaining Fuzzing

Names and KTH ID

• Brad Palagi ([email protected])
• Ben Civjan ([email protected])

Deadline

Task 1

Category

Executable Tutorial

Description

Fuzz testing is an essential topic for testing and security.

We created a Katacoda Scenario with the goal of teaching the user how to effectively use Atheris, Google's Python Fuzzing tool which is a part of OSS-Fuzz and can be integrated into ones own git repository using ClusterFuzzLite.

In the tutorial we covered: Background information on fuzzing as a testing technique, the Pros and Cons of Fuzzing versus standard or other testing methods, how to integrate Google's Atheris Python fuzzer with a Python program, and how to add this type of fuzzing into a CI workflow with GitHub Actions using ClusterFuzzLite.

Final Submission Links:

Katacoda-Scenario Repository: https://github.com/bencivjan/katacoda-scenarios

Katacoda-Scenario Tutorial: https://www.katacoda.com/bencivjan/scenarios/fuzzing-exec-tutorial

Grading Criteria We Intended To Meet

	Yes	No
executable: The tutorial can be automatically executed from beginning to the end, in the browser or in CI (see below)	Mandatory	-
ilo: The tutorial states the intended learning outcomes.	Mandatory	-
motivation: The tutorial is clearly motivated (why it matters for Devops?)	Yes	No
browser-based: The tutorial can be successful executed in the browser (katacoda is recommended)	Yes	No
ci-based: The tutorial can successful be executed as a CI job	Yes	No
background: The tutorial gives enough background	Yes	No
illustrated: The tutorial is illustrated with an informative figure (eg a flowchart)	Yes	No
pedagogical: The tutorial is easy to follow	Yes	No
original: The tutorial is original, no or few similar tutorials exist on the web	Yes	No
easter-eggs: The tutorial contains an easter egg	Yes	No
language: The language is appropriate (structure, grammar, spelling)	Yes	No

[cesarsotovalero]

LGTM!

[monperrus]

@bpalagi in Step 7, the commands cannot be copy/pasted and the file contents (eg .clusterfuzzlite) are not put in the editor (as opposed to the previous steps). Also there is no final command to run.

could you fix step 7 that so that I can proceed with grading?

Thanks!

[bpalagi]

@monperrus

Professor, we were unable to find how to have a local git repository inside of the Katacoda environment which would allow any user to create git actions and push/pull to see the fuzzer run.

We have updated the scenario to make it clear that step 7 would need to be done on ones own local repository. Please advise us if it is better to just remove that step because it is unable to be performed in the browser. We wanted to leave it in just to emphasize the value that fuzzing adds when used in a CI workflow.

[monperrus]

@bpalagi ack, thanks.

[thomassrour]

Hey @bpalagi @bencivjan, @BastienFaivre and I find the topic of your tutorial interesting and would like to do our feedback task on it. Would you be open to that ?

[bencivjan]

@thomassrour @BastienFaivre Yes, we'd be open to it. However, we already submitted and received our grade for it so we wouldn't be able to implement any of the changes. If that works for you though you definitely have our permission!

[thomassrour]

Feedback for Katacoda Executable Tutorial demonstrating and explaining Fuzzing

This tutorial describes and showcases Fuzzing, which is a prominent and very used testing method, which can detect bugs and errors on code at any scale. This method is particularly important nowadays since it can automatically detect bugs that could lead to security breaches, which are hard to detect with usual testing strategies, thus making this tutorial relevant. After completing and analyzing this tutorial thoroughly, here is our feedback :

High level strengths and weaknesses

Strengths

• The tutorial provides a strong theoretical background about fuzzing and its different instances.
• The tutorial is easily readable, the language is errorless, there are no typos.
• There are no bugs in the code, all steps which require an execution work as intended.
• The real life example (finding a bug in a widely used library) that was given is a very nice feature, and clearly demonstrates the usefulness of this technique.
• Most of the material is well referenced.

Weaknesses and Improvements

• The definition of instrumentation in the context of fuzzing is missing in step 4. This term is often used later on in the tutorial, which makes the overall understanding of the further explanations harder. Another technical term whose explanation we couldn't find is ClusterFuzz, which is first referenced in Step 4.

  Possible improvement: Obviously, add the definition of instrumentation, explain what ClusterFuzz is, and describe the concepts a little more when mentionning them later.

• The use of Katacoda is kind of unecessary, most of the steps are just theoretical background. Furthermore, there are very few commands to execute, and they are limited to running some Python files.

  Possible alternative: You might have been better off making a demo/presentation about the subject. The way the tutorial is organized (high level explanation then an interesting showcase) would have been a good structure for a demo.

  Otherwise, a simple blog post or a README would have been enough for someone to grasp the concepts shown, as there is not that much to execute.

• Step 4 is essentially a copy paste of the link given at the end of the page, no additional information is provided.

  Possible improvement: We understand that it is important to understand these concepts before going into step 5, but again, such a step doesn't really belong in a tutorial. Simply giving the link to the definitions in a previous step and asking the reader to visit it and understand the definitions given would have been sufficient.

• Even though it's a bonus section, step 7 is a bit all over the place, it gives a shallow description of how to use fuzzing in CI without actually explaining the steps required to do so. We tried following the steps given in the Katacoda and found ourselves a bit lost, especially since some things are referenced without proper introduction (eg : oss-fuzz repo).

  Possible improvement: This part remains very interesting and important, so it would definitely warrant a tutorial of its own, since there is actually has a lot of hands-on potential. It could be very useful to people who want to implement this. Therefore, simply referencing an actual tutorial about this might have been better than going over it that quickly and broadly.

• Although it is good to have references, we didn't immediatly understand that the footnotes were actually on the finish page. Additionally, we cannot click on the reference number throughout the tutorial.

  Possible improvement: You could place the reference links on the bottom of the step where it is referenced, or you could make the number clickable (if possible).

Additional pointers

• We would have liked to have a deeper understanding of how Atheris understands how to choose inputs to maximize code coverage. In the given example, it is obvious that is chooses the input based on the branching condition, but in a more complex setting, choosing such an input would not be as simple. The Atheris README on GitHub is a good start.

• We also were wondering if implementing fuzzing in other languages than Python would be as straightforward. This might be a relevant topic to expand on, as your title doesn't limit the subject to Python. For example, here is how to implement fuzzing in Golang.

• Finally it could also be interesting to check out how to fuzz web applications (for example using Burp Suite)

Concluding Remarks

Despite not being a very hands-on tutorial, this Katacoda still gives very valuable information on what fuzzing is and how it works. Thank you for allowing us to review your tutorial.