Integrate native go fuzzing #7055

AdamKorcz · 2021-12-22T17:13:29Z

This PR is the tracker for native Go fuzz integration. It is expected to be ready when native fuzzing becomes available in Go 1.18.

The purpose of this PR is to enable Golang projects to run their Go-1.18 fuzzers in OSS-fuzz.

Design

From a high level, a native fuzz harness is rewritten to a go-fuzz-like harness which is then built with libfuzzer instrumentation by way of an extended version of the go114-fuzz-build script which is currently being used by OSS-fuzz.
This process is enabled by a custom implementation of *testing.F which has the purpose of generating the arguments to the fuzzer based on the byte array from libFuzzer and calling the fuzz function. In a native Go fuzz harness, any number of arguments can be passed to the fuzzer, and these can be of other types than a byte slice.
Each argument is generated with the go-fuzz-headers library which deterministically can generate all the supported types from a byte slice.

Testing

The integration has been tested on the following fuzzers:

Vitess

FuzzTabletManager_ExecuteFetchAsDba (src)
FuzzEqualsSQLNode (src)
FuzzIsDML (src)
FuzzNodeFormat (src)
FuzzNormalizer (src)
FuzzParser (src)
FuzzSplitStatementToPieces (src)

go-ethereum

FuzzRuntimeNative (src)
FuzzLesNative (src)
FuzzRlpNative (src)

istio

FuzzParseInputs (src)
FuzzKubeCRD (src)
FuzzParseMeshNetworks (src)
FuzzValidateMeshConfig (src)

As @DavidKorczynski and I work more on the integration, updates and documentation will be added to this PR.

Features

Does not require upstream changes to native Go fuzzing.
Does not require changes to OSS-fuzz runtime environment.
Works with OSS-fuzz's seed corpus and dictionaries.
Works with any number of parameters in the native harness.
Requires no functional changes to existing go fuzz coverage build. Some code will be reused and therefore reorganized. (Expected/pending)

Supported arg types:

[]byte
string
int
int8
int16
int32
int64
uint
uint8
uint16
uint32
uint64
rune
float32
float64

All native supported arg types: https://github.com/golang/go/blob/master/src/testing/fuzz.go#L165. All of these will be supported by OSS-fuzz.

prestonvanloon · 2022-01-09T21:01:10Z

@AdamKorcz is it possible to support fuzz targets with 2+ fuzz arguments?

AdamKorcz · 2022-01-10T10:10:22Z

is it possible to support fuzz targets with 2+ fuzz arguments?

Yes. This is already supported at this stage.

prestonvanloon · 2022-01-10T16:48:33Z

is it possible to support fuzz targets with 2+ fuzz arguments?

Yes. This is already supported at this stage.

Thanks @AdamKorcz. To clarify, is it possible to support fuzz targets with 2+ fuzz args with the LLVMFuzzerTestOneInput interface?

Curious how that would work given that there is one input sent to n args.
Could you kindly add an example to the PR? Thanks in advance.

AdamKorcz · 2022-01-10T17:00:37Z

is it possible to support fuzz targets with 2+ fuzz args with the LLVMFuzzerTestOneInput interface?

Yes. Native Go fuzzers with 1+ arguments are compatible with the current libfuzzer mode that OSS-fuzz will support for native Go fuzzers. Any native Go fuzzer with 1+ arguments of any of the natively supported types will be compiled with the LLVMFuzzerTestOneInput interface and built with -fsanitize=fuzzer. This is currently implemented and should be available when Go 1.18 is released. Hope it answers the question.

Curious how that would work given that there is one input sent to n args.

It is solved with a custom implementation of *testing.F.Fuzz() that checks which and how many arguments are passed, creates the arguments of those types from the byte array provided by libFuzzer and finally calls the original func(*testing.T, any) with those arguments. The implementation can be found here.

Could you kindly add an example to the PR? Thanks in advance.

Will do!

oliverchang · 2022-01-24T01:46:56Z

Any updates on this? It would be good to get this out and the documentation ready soon together with the Go 1.18 release date.

AdamKorcz · 2022-01-24T22:40:37Z

Any updates on this? It would be good to get this out and the documentation ready soon together with the Go 1.18 release date.

We are ready for review this week.

oliverchang

thanks!! @jonathanmetzman could you please take a pass as well?

docs/getting-started/new-project-guide/go_lang.md

jonathanmetzman

Did a first pass, most files LGTM except compile_native_go_fuzzer which I haven't finished reviewing.
To help me understand it, can you say why we rewrite the fuzz target?

docs/getting-started/new-project-guide/go_lang.md

jonathanmetzman · 2022-02-02T15:30:19Z

infra/base-images/base-runner/coverage

  $OUT/$target -test.coverprofile $DUMPS_DIR/$target.profdata &> $LOGS_DIR/$target.log
+
+  # The Go 1.18 fuzzers are renamed to "*_fuzz_.go" during "infra/helper.py build_fuzzers".


infra/base-images/base-runner/coverage

jonathanmetzman · 2022-02-02T15:31:39Z

infra/base-images/base-builder/install_go.sh

@@ -26,3 +26,16 @@ echo 'Set "PATH=$PATH:/root/.go/bin:$GOPATH/bin"'

 go get -u github.com/mdempsky/go114-fuzz-build
 ln -s $GOPATH/bin/go114-fuzz-build $GOPATH/bin/go-fuzz
+
+go install golang.org/dl/gotip@latest \


I'm confused. Youre installing gotip in base-builder but using it in base-runner. Is this intended?

gotip is also being used base-builder in the coverage build in compile_native_go_fuzzer

Right but where is it coming from in base-runner then?

infra/base-images/base-builder/compile_native_go_fuzzer

jonathanmetzman · 2022-02-02T17:08:59Z

Adam could you check if these target binaries will contain the string LLVMFuzzerTestOneInput? If not I think it could break some things.

AdamKorcz · 2022-02-02T17:55:26Z

To help me understand it, can you say why we rewrite the fuzz target?

2 things happen in rewrite_go_fuzz_harness:

The file name is changed so it does not end with _test.go. This is so the fuzzer can be built with go build.
*testing.F is replaced with *github.com/AdamKorcz/go-118-fuzz-build/utils.F to allow the fuzzer to generate any number of parameters of any type based on the byte array from libFuzzer. This is to accommodate the feature in Native Go Fuzzing where a fuzzer can have multiple arguments and they can be other types than the usual byte array. By using *github.com/AdamKorcz/go-118-fuzz-build/utils.F, any argument in a Native Go fuzzer will be created from the libFuzzer byte array.

@jonathanmetzman

jonathanmetzman · 2022-02-02T18:54:07Z

random thought: I wonder if we should just be using golang's engine, would probably be worse and more effort on our part, but seems less hacky and perhaps they've optimized their fuzzer for Go code in a way libfuzzer isn't

jonathanmetzman · 2022-02-02T18:54:55Z

To help me understand it, can you say why we rewrite the fuzz target?

2 things happen in rewrite_go_fuzz_harness:

The file name is changed so it does not end with _test.go. This is so the fuzzer can be built with go build.

*testing.F is replaced with *github.com/AdamKorcz/go-118-fuzz-build/utils.F to allow the fuzzer to generate any number of parameters of any type based on the byte array from libFuzzer. This is to accommodate the feature in Native Go Fuzzing where a fuzzer can have multiple arguments and they can be other types than the usual byte array. By using *github.com/AdamKorcz/go-118-fuzz-build/utils.F, any argument in a Native Go fuzzer will be created from the libFuzzer byte array.

@jonathanmetzman

Ah so we're not really using golang's native fuzzer. Two questions:

are the command line arguments different from the native fuzzer?
Is the corpus compatible between the two?

AdamKorcz · 2022-02-02T19:33:40Z

random thought: I wonder if we should just be using golang's engine, would probably be worse and more effort on our part, but seems less hacky and perhaps they've optimized their fuzzer for Go code in a way libfuzzer isn't

We discussed this with @oliverchang and the Go team. The conclusion was to go with this solution for now and perhaps switch to the Native Go fuzzing engine at a later stage. Several pros and cons came up, but on the pros side of the libFuzzer approach is that support for Native Go harnesses is available in OSS-fuzz when Go 1.18 releases, and users can start running their fuzzers continuously right away. In addition to that: No infra/runtime changes are required on the OSS-fuzz side. On the con side: Avoiding the rewrites in this PR, having similar UX between local fuzzing and continuous fuzzing.

Another pro of the approach in this PR is that it follows the paradigm of how go-fuzz is currently used in OSS-fuzz: i.e. the existing go-fuzz already has their own engine but the oss-fuzz fuzzers run by way of libFuzzer.

By the way, related to this we thought it might be interesting to implement go fuzz engines in fuzzbench as it seems there are a bunch now.

AdamKorcz · 2022-02-02T19:58:08Z

are the command line arguments different from the native fuzzer?

Is the corpus compatible between the two?

Am not entirely sure if you're asking about the native go-fuzzing engine in general or the additions in this PR? As in, the corpus is not compatible by default but we have added support for this (see the feature list in the initial description).

Note that we have tested this work in oss-fuzz by way of a test project here: https://github.com/google/oss-fuzz/tree/master/projects/testing-native-go-fuzzing.

jonathanmetzman · 2022-02-02T20:00:33Z

random thought: I wonder if we should just be using golang's engine, would probably be worse and more effort on our part, but seems less hacky and perhaps they've optimized their fuzzer for Go code in a way libfuzzer isn't

We discussed this with @oliverchang and the Go team. The conclusion was to go with this solution for now and perhaps switch to the Native Go fuzzing engine at a later stage. Several pros and cons came up, but on the pros side of the libFuzzer approach is that support for Native Go harnesses is available in OSS-fuzz when Go 1.18 releases, and users can start running their fuzzers continuously right away. In addition to that: No infra/runtime changes are required on the OSS-fuzz side. On the con side: Avoiding the rewrites in this PR, having similar UX between local fuzzing and continuous fuzzing.

Makes sense, I hadn't seen what was decided.

Another pro of the approach in this PR is that it follows the paradigm of how go-fuzz is currently used in OSS-fuzz: i.e. the existing go-fuzz already has their own engine but the oss-fuzz fuzzers run by way of libFuzzer.

ACK.

By the way, related to this we thought it might be interesting to implement go fuzz engines in fuzzbench as it seems there are a bunch now.

+1

DavidKorczynski · 2022-02-02T20:00:39Z

Note that we have tested this work in oss-fuzz by way of a test project here: https://github.com/google/oss-fuzz/tree/master/projects/testing-native-go-fuzzing.

In addition we have tested on the Go-ethereum and Istio projects as well (as described in the PR text), i.e. we have added native go-fuzzers to those projects and their have been running for a month or so

jonathanmetzman · 2022-02-02T20:02:44Z

are the command line arguments different from the native fuzzer?

Is the corpus compatible between the two?

Am not entirely sure if you're asking about the native go-fuzzing engine in general or the additions in this PR? As in, the corpus is not compatible by default but we have added support for this (see the feature list in the initial description).

Note that we have tested this work in oss-fuzz by way of a test project here: https://github.com/google/oss-fuzz/tree/master/projects/testing-native-go-fuzzing.

Perhaps I'm being unclear but here is what i mean.
Suppose a go developer runs the fuzzer locally gets some interesting testcases (ones that caused crashes) and then includes this in OSS-Fuzz as a seed corpus.
Will these testcases be interpreted the same by libFuzzer. What I'm really getting at is: is the way libfuzzer (with your shim) and go native fuzzing represent multi-arg testcases the same?

jonathanmetzman

LGTM
The change seems fine to me based on what was decided for this feature.

infra/base-images/base-builder-go/native_ossfuzz_coverage_runner.go

AdamKorcz · 2022-02-02T20:17:16Z

What I'm really getting at is: is the way libfuzzer (with your shim) and go native fuzzing represent multi-arg testcases the same?

We have not performed this test, but my guess is that the answer is no. We're happy to explore this, and, perhaps do it simultaneously with migrating the native engine itself

jonathanmetzman · 2022-02-02T20:19:10Z

What I'm really getting at is: is the way libfuzzer (with your shim) and go native fuzzing represent multi-arg testcases the same?

We have not performed this test, but my guess is that the answer is no. We're happy to explore this, and, perhaps do it simultaneously with migrating the native engine itself

I think this is somewhat of a problem. most likely users won't noticed this (or the fact that .options file may also not work as they expect). Is this the same behavior wrt to libfuzzer for gofuzz targets?

AdamKorcz · 2022-02-02T20:45:47Z

I think this is somewhat of a problem. most likely users won't noticed this (or the fact that .options file may also not work as they expect).

I agree. The current implementation is focused on a first step in this context, i.e. OSS-fuzz will support running Native Go fuzzers instantly. There are more steps in maturing it and this is one of those for sure!

Note that at the moment Go 1.18 fuzzing has very few users: I have found a single OSS-fuzz project with two 1.18 fuzzers upstream.

Is this the same behavior wrt to libfuzzer for gofuzz targets?

No. These can be used interchangeably.

AdamKorcz · 2022-02-03T12:20:30Z

Adam could you check if these target binaries will contain the string LLVMFuzzerTestOneInput?

Can confirm all target binaries this was tested on (all mentioned in PR text) contain the string "LLVMFuzzerTestOneInput".

AdamKorcz force-pushed the native-go-fuzz-integration branch from 284e1dc to c9c77de Compare December 22, 2021 17:26

[draft] Integrate native go fuzzing

d9ebee7

AdamKorcz force-pushed the native-go-fuzz-integration branch from c9c77de to d9ebee7 Compare December 22, 2021 17:28

This was referenced Jan 5, 2022

cmd/go: support continuously running fuzzing with OSS-Fuzz golang/go#50192

Closed

Extend support go 1.18 - native fuzzing #7020

Closed

Migrate to go native fuzzing in 1.18 sigstore/sigstore#205

Closed

AdamKorcz added 2 commits January 6, 2022 17:13

Use script to change imports of fuzz harness

f61d935

Add comment

f92d4b8

AdamKorcz mentioned this pull request Jan 6, 2022

Fuzzing: Rewrite fuzzers to native Go 1.18 fuzzers vitessio/vitess#9479

Open

AdamKorcz added 4 commits January 6, 2022 18:35

Add more fuzzers

9ce510f

wip: wrap compile_go_fuzzer and compile_native_go_fuzzer

f7b0cbd

vitess specific: Disable parser logging

05e94b8

Add support for coverage

668f3d4

AdamKorcz force-pushed the native-go-fuzz-integration branch from 85e5e03 to 668f3d4 Compare January 7, 2022 13:39

AdamKorcz mentioned this pull request Jan 7, 2022

Test drive native fuzzing #7104

Merged

AdamKorcz force-pushed the native-go-fuzz-integration branch 3 times, most recently from aa478dd to 605e181 Compare January 10, 2022 15:23

fix coverage build

3783183

AdamKorcz force-pushed the native-go-fuzz-integration branch from 605e181 to 3783183 Compare January 10, 2022 15:27

AdamKorcz force-pushed the native-go-fuzz-integration branch 2 times, most recently from 20da06b to 65e5046 Compare January 24, 2022 15:20

Move logic to base images. Remove test fuzzers

48de2c6

AdamKorcz force-pushed the native-go-fuzz-integration branch from 65e5046 to 48de2c6 Compare January 24, 2022 15:20

AdamKorcz added 3 commits January 27, 2022 11:56

minor refactoring

f6e0465

Add documentation

6d13364

minor refactoring

81fab98

AdamKorcz marked this pull request as ready for review January 27, 2022 12:25

AdamKorcz requested a review from oliverchang January 27, 2022 12:26

oliverchang requested a review from jonathanmetzman January 28, 2022 03:46

oliverchang reviewed Jan 28, 2022

View reviewed changes

docs/getting-started/new-project-guide/go_lang.md Outdated Show resolved Hide resolved

AdamKorcz added 2 commits February 2, 2022 12:48

Improve sed command

37bc302

Update docs

a2ae977

jonathanmetzman reviewed Feb 2, 2022

View reviewed changes

Fix formatting of comments

7f8c0cb

jonathanmetzman approved these changes Feb 2, 2022

View reviewed changes

infra/base-images/base-builder-go/native_ossfuzz_coverage_runner.go Show resolved Hide resolved

infra/base-images/base-builder-go/native_ossfuzz_coverage_runner.go Show resolved Hide resolved

oliverchang merged commit 4fdde05 into google:master Feb 8, 2022

AdamKorcz changed the title ~~[draft] Integrate native go fuzzing~~ Integrate native go fuzzing Feb 8, 2022

oliverchang mentioned this pull request Feb 15, 2022

go coverage reports are broken #7281

Closed

elias-orijtech mentioned this pull request Apr 21, 2022

Native support for Go 1.18 corpus format #7600

Open

MartinPetkov pushed a commit to MartinPetkov/oss-fuzz that referenced this pull request Aug 15, 2022

[draft] Integrate native go fuzzing (google#7055)

cabe177

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate native go fuzzing #7055

Integrate native go fuzzing #7055

AdamKorcz commented Dec 22, 2021 •

edited

Loading

prestonvanloon commented Jan 9, 2022

AdamKorcz commented Jan 10, 2022 •

edited

Loading

prestonvanloon commented Jan 10, 2022

AdamKorcz commented Jan 10, 2022 •

edited

Loading

oliverchang commented Jan 24, 2022

AdamKorcz commented Jan 24, 2022

oliverchang left a comment

jonathanmetzman left a comment

jonathanmetzman Feb 2, 2022

jonathanmetzman Feb 2, 2022

AdamKorcz Feb 2, 2022

jonathanmetzman Feb 2, 2022

jonathanmetzman commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022 •

edited

Loading

jonathanmetzman commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022 •

edited

Loading

DavidKorczynski commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022

jonathanmetzman left a comment

AdamKorcz commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022 •

edited

Loading

AdamKorcz commented Feb 2, 2022

AdamKorcz commented Feb 3, 2022

		$OUT/$target -test.coverprofile $DUMPS_DIR/$target.profdata &> $LOGS_DIR/$target.log

		# The Go 1.18 fuzzers are renamed to "*_fuzz_.go" during "infra/helper.py build_fuzzers".

Integrate native go fuzzing #7055

Integrate native go fuzzing #7055

Conversation

AdamKorcz commented Dec 22, 2021 • edited Loading

Design

Testing

Vitess

go-ethereum

istio

Features

Supported arg types:

prestonvanloon commented Jan 9, 2022

AdamKorcz commented Jan 10, 2022 • edited Loading

prestonvanloon commented Jan 10, 2022

AdamKorcz commented Jan 10, 2022 • edited Loading

oliverchang commented Jan 24, 2022

AdamKorcz commented Jan 24, 2022

oliverchang left a comment

Choose a reason for hiding this comment

jonathanmetzman left a comment

Choose a reason for hiding this comment

jonathanmetzman Feb 2, 2022

Choose a reason for hiding this comment

jonathanmetzman Feb 2, 2022

Choose a reason for hiding this comment

AdamKorcz Feb 2, 2022

Choose a reason for hiding this comment

jonathanmetzman Feb 2, 2022

Choose a reason for hiding this comment

jonathanmetzman commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022 • edited Loading

jonathanmetzman commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022

AdamKorcz commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022 • edited Loading

DavidKorczynski commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022

jonathanmetzman left a comment

Choose a reason for hiding this comment

AdamKorcz commented Feb 2, 2022

jonathanmetzman commented Feb 2, 2022 • edited Loading

AdamKorcz commented Feb 2, 2022

AdamKorcz commented Feb 3, 2022

AdamKorcz commented Dec 22, 2021 •

edited

Loading

AdamKorcz commented Jan 10, 2022 •

edited

Loading

AdamKorcz commented Jan 10, 2022 •

edited

Loading

AdamKorcz commented Feb 2, 2022 •

edited

Loading

jonathanmetzman commented Feb 2, 2022 •

edited

Loading

jonathanmetzman commented Feb 2, 2022 •

edited

Loading