Add Godot Engine to oss-fuzz

[fire]

Describe the project you are working on:

Robustness, and security is a core Godot Engine feature.

Describe the problem or limitation you are having in your project:

Godot commonly crashes and does not deal well with valid inputs.

Describe how this feature / enhancement will help you overcome this problem or limitation:

https://github.com/google/oss-fuzz

I was at the Google GSOC Mentor Summit. Google had a room dedicated to oss-fuzz.

oss-fuzz is a service ran by Google to help prominent or security-critical software find bugs via the method of fuzzing. They also offer up $20,000 in rewarding oss-fuzz integration.

Fuzzing is a method of generating valid inputs to Godot and finding invalid responses. These invalid responses can be used to discover issues to be fixed.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

Describe implementation detail for your proposal (in code), if possible:

The approach described by the oss-fuzz Googler is to take https://github.com/qarmin/The-worst-Godot-test-project exported as a Godot project. We would use the compiled gdscript bytecode and run the fuzzer on that input.

Why bytecode? Compiled gdscript is easier for the fuzzer to generate valid input.

oss-fuzz requires this project corpus to get the state of the whole project for fuzzing

oss-fuzz requires the Godot project to submit a pull request.

• with the name of the project,
• why the Godot project should be tested
• a list of developers to be emailed when bugs are found.
  • I nominate @fire, @akien-mga and @qarmin to have the associated email address to be set in the pr template.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

No, this requires integration by the compiler and Godot C++ code.

Is there a reason why this should be core and not an add-on in the asset library?:

This feature must be in the core because oss-fuzz requires a single executable and uses clang instrumentation to find bugs.

[Xrayez]

Godot has integrated doctest as seen in godotengine/godot#40659 now, and runs on CI currently with some tests already ported.

While porting some tests like godotengine/godot#40835, I've eventually stumbled upon stress/random tests there as well (they take "too many" cycles currently to be run on GitHub's CI). I think we could still have those kind of tests but manage them differently.

There's an option to filter tests with doctest: godot --test -tc="*filter*". I was thinking that we could come up with some convention to tag those test cases as [Stress] or [Fuzz] to be run on those fuzz machines specifically. 🙂

It's also possible to manage those tests independently of Godot core via modules: godotengine/godot#40720. For instance we could have godotengine/godot-oss-fuzz repository implemented as a module and write those tests there.

[RevoluPowered]

Godot has integrated doctest as seen in godotengine/godot#40659 now, and runs on CI currently with some tests already ported.

While porting some tests like godotengine/godot#40835, I've eventually stumbled upon stress/random tests there as well (they take "too many" cycles currently to be run on GitHub's CI). I think we could still have those kind of tests but manage them differently.

There's an option to filter tests with doctest: godot --test -tc="*filter*". I was thinking that we could come up with some convention to tag those test cases as [Stress] or [Fuzz] to be run on those fuzz machines specifically. 🙂

It's also possible to manage those tests independently of Godot core via modules: godotengine/godot#40720. For instance we could have godotengine/godot-oss-fuzz repository implemented as a module and write those tests there.

This would be nice and would mean the main logic would be reused too :) I like this idea

[Faless]

I really like the idea of having a fuzz-engine test our code. I'm not sure how feasible it is to run the whole demo project (I still have to look into oss-fuzz in depth), but I would really love to see at least dedicated submodules tested (e.g. marshalls, debugger serializers, etc).

[ellenhp]

It's not feasible to run an entire demo project. The way libfuzzer targets tend to work is they'll have an initialization method they expose that you go and implement. This can be slow to execute if need be, but ideally not. This will do things like set up singletons in Godot. Then they have another function that they call as fast as they can (ideally thousands of times a second) that gives you a block of data to push into whaterver that fuzz target is exercising. If we wanted to test a resource importer, we would take that data and treat it as a file and run the importer. Additionally, fuzzers tend to work best with small inputs. I've seen the number 4kb quoted as an arbitrary cutoff of where inputs start getting too large to effectively mutate. I don't think there are any valid godot demo projects that exercise a meaningful part of the codebase less than 4kb :)

Where fuzz testing really starts to shine is when you do things like fuzz test a parsing, deserialization or decompression operation. These sorts of things commonly have buffer overruns that are hard to spot reading the code but can be found by a fuzzer, sometimes in seconds or less. Heartbleed famously can be found in seconds on commodity hardware with an appropriately configured fuzzer.

The real trick will be getting enough of Godot's initialization running in the clusterfuzz container that we can write meaningful libfuzzer targets. Something you'll run into immediately if you start trying to write them in the current state of Godot is that malformed input will cause some parts of the engine to go grab the OS singleton and attempt to use it to print an error, which will crash the fuzz target if you forgot to initialize the singletons.

Perhaps someone more knowledgeable could help to write the initialization? I don't really have broad knowledge about Godot but I have experience working with fuzzers so I'm happy to help write the fuzz targets themselves.

[fire]

You can reuse the class_db initialization sequence in doctest.

https://github.com/godotengine/godot/blob/master/tests/test_class_db.h

[Xrayez]

Me and @qarmin have been successfully using https://github.com/qarmin/Qarminer to fuzz Godot via GDScript project, and I've even created a dedicated fuzzer instance for Goost that fuzzes Goost classes only: https://github.com/goostengine/goost-fuzzer, this way different classes can be fuzzed on demand. This kind of setup has already helped to reveal several crashes/memory leaks via cron GitHub Actions jobs so far, you can see a history of actions).

The only thing missing is a proper issue reporting for found bugs. You have to download huge logs and find leak/address sanitizer crashes manually.

I'm no longer convinced in effectiveness of C++ fuzz testing specifically unless there are cases where internal algorithms must be fuzzed, but as someone said to me, most of the time it's just implementation detail, testing via GDScript makes wonders, and it should be easier to maintain.

[ellenhp]

How are we supposed to test things like serialization/deserialization and resource import pipelines except by libfuzzer targets? These seem like they're built for entirely different use-cases. Qarminer finds bugs, libfuzzer finds security vulnerabilities.

We could have a stack buffer overflow in our netcode right now and we would have absolutely no idea it's there. We could be exposing all end-users of networked godot games to trivially exploitable RCE vulnerabilities. It's just not acceptable to be a project of this size and not be fuzzing in 2021, IMO. We don't need to use oss-fuzz, but we do need to be fuzzing critical C++ code somehow.

[ellenhp]

This is an example of a libfuzzer target that exercises the GLTF import pipeline. I had to modify it slightly to take a FileAccess* instead of a path, which allows me to use a custom InMemoryFileAccess.

#include "modules/gltf/gltf_state.h"
#include "in_memory_file_access.h"

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
	FileAccess *f = new InMemoryFileAccess(data, size);

	Ref<GLTFState> r_state;
	r_state.instantiate();

	Ref<GLTFDocument> gltf_document;
	gltf_document.instantiate();
	Error err = gltf_document->parse(r_state, f, "res://test.glb");

	delete f;

	return 0;
}

There's barely anything here to maintain, it's extremely simple. I've run this target fairly extensively with no issues, but I don't have the infrastructure to run my targets for a few hours each. oss-fuzz does.

[Xrayez]

It all depends on Godot's goals to be honest. According to what I've seen, a bug/security vulnerability (which is also a bug) is never seen as an actual bug unless it's reproducible by using public/surface API in Godot.

For example, I recall someone from Godot core developers said that reported issues like godotengine/godot#45984 should be immediately closed, just because those functions are not public, therefore cannot be normally reproduced. Obviously, I do not share this opinion. 😛

However, I hope I'm wrong and priorities have changed, in that case what you suggest also makes sense to implement (finding security vulnerabilities). But it's not like GDScript is unable to find some subset of vulnerabilities as well.