Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Centipede a default engine #9427

Closed
wants to merge 13 commits into from
Closed

Make Centipede a default engine #9427

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
be42b22
Make Centipede a default engine
DonggeLiu Jan 16, 2023
e2a15d8
Add Centipede's build steps to expected build steps
DonggeLiu Jan 16, 2023
87b26c4
Use -Wl for linker flags
DonggeLiu Jan 17, 2023
3b3c1da
Fix flag
DonggeLiu Jan 17, 2023
3e7af92
Add week.o to LDFLAGS
DonggeLiu Jan 18, 2023
1b4b623
A wrapper of ld that always append centipede flags
DonggeLiu Jan 19, 2023
715471f
Update Centipede
DonggeLiu Feb 2, 2023
447e163
Fix Undefined reference to operator new/delete
DonggeLiu Feb 2, 2023
9c536a4
Add -lc++ to ld
DonggeLiu Feb 2, 2023
334aff6
Build weak.o with $CC and omit -lc++
DonggeLiu Feb 5, 2023
d43e235
Revert "Build weak.o with $CC and omit -lc++"
DonggeLiu Feb 5, 2023
e633dfc
Merge branch 'master' into make_centipede_default_engine
jonathanmetzman Feb 6, 2023
38d53f2
Merge branch 'master' into make_centipede_default_engine
jonathanmetzman Feb 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions infra/base-images/base-builder/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,9 @@ COPY precompile_honggfuzz /usr/local/bin/
RUN precompile_honggfuzz

RUN cd $SRC && \
git clone https://github.com/google/centipede.git && \
git clone -n https://github.com/google/centipede.git && \
cd centipede && \
git checkout eb91dd2157710e6c82579f8be19d7fab9423b781 && \
git checkout 2a2c78a2c161d99f5962b9710bce61feb00acc3d && \
rm -rf .git

COPY precompile_centipede /usr/local/bin/
Expand Down
8 changes: 7 additions & 1 deletion infra/base-images/base-builder/compile_centipede
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,15 @@ cp "$BIN_DIR/libcentipede_runner.pic.a" "$LIB_FUZZING_ENGINE"

export DFTRACING_FLAGS='-fsanitize-coverage=trace-loads'
export CENTIPEDE_FLAGS=`cat "$SRC/centipede/clang-flags.txt" | tr '\n' ' '`
export LIBRARIES_FLAGS="-Wno-unused-command-line-argument -ldl -lrt -lpthread $SRC/centipede/weak.o"
export LIBRARIES_FLAGS="-lc++ -Wno-unused-command-line-argument"

export CFLAGS="$CFLAGS $DFTRACING_FLAGS $CENTIPEDE_FLAGS $LIBRARIES_FLAGS"
export CXXFLAGS="$CXXFLAGS $DFTRACING_FLAGS $CENTIPEDE_FLAGS $LIBRARIES_FLAGS"
# For Meson.
export LDFLAGS='/src/centipede/weak.o'
Comment on lines +34 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not clear why centipede is so unique that we need to support LDFLAGS after six years of not needing it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which projects does this help? Most meson projects seem broken still.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I will test removing it here.
I've added weak.o to ld wrapper, not sure if that works for meson.


# For Centipede's ld
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

end with period.

export CENTIPEDE_BIN="$SRC/centipede/bin"
export PATH="$CENTIPEDE_BIN":$PATH
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put ending quote on the end of this line.


echo 'done.'
9 changes: 9 additions & 0 deletions infra/base-images/base-builder/precompile_centipede
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ bazel --bazelrc=/tmp/centipede.bazelrc build -c opt :all
# The weak symbols provides fake implementations for intermediate binaries.
$CXX "$SRC/centipede/weak_sancov_stubs.cc" -c -o "$SRC/centipede/weak.o"

# Centipede's ld, which always appends weak.o as a parameter.
export CENTIPEDE_BIN="$SRC/centipede/bin"
mkdir -p $CENTIPEDE_BIN
cat << EOF > ${CENTIPEDE_BIN}/ld
#!/bin/bash
/usr/bin/ld \$@ -ldl -lrt -lpthread -lc++ /src/centipede/weak.o
EOF
chmod 777 ${CENTIPEDE_BIN}/ld
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we setting this to be the linker somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 37-39 in compile_centipede.

Copy link
Contributor

@jonathanmetzman jonathanmetzman Feb 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got it. Do you mind giving me an hour to test #9300? I have a hunch most of the major fixes have been pushed and I think this PR has some unecessary complexity. I suspect this LD wrapper isn't necessary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I am working on google/fuzzbench#1632 now.


echo 'Removing extra stuff leftover to avoid bloating image.'

rm -rf /clang-*.tgz /clang
Expand Down
2 changes: 1 addition & 1 deletion infra/build/functions/build_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
GCB_LOGS_BUCKET = 'oss-fuzz-gcb-logs'

DEFAULT_ARCHITECTURES = ['x86_64']
DEFAULT_ENGINES = ['libfuzzer', 'afl', 'honggfuzz']
DEFAULT_ENGINES = ['libfuzzer', 'afl', 'honggfuzz', 'centipede']
DEFAULT_SANITIZERS = ['address', 'undefined']

LATEST_VERSION_FILENAME = 'latest.version'
Expand Down
282 changes: 282 additions & 0 deletions infra/build/functions/test_data/expected_build_steps.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,288 @@
"rm -r /workspace/out/afl-address-x86_64"
]
},
{
"name": "gcr.io/cloud-builders/docker",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-address-x86_64",
"SANITIZER=address"
],
"args": [
"run",
"--platform",
"linux/amd64",
"-v",
"/workspace:/workspace",
"--privileged",
"--cap-add=all",
"-e",
"ARCHITECTURE=x86_64",
"-e",
"FUZZING_ENGINE=centipede",
"-e",
"FUZZING_LANGUAGE=c++",
"-e",
"HOME=/root",
"-e",
"OUT=/workspace/out/centipede-address-x86_64",
"-e",
"SANITIZER=address",
"-t",
"gcr.io/oss-fuzz/test-project",
"bash",
"-c",
"rm -r /out && cd /src && cd /src && mkdir -p /workspace/out/centipede-address-x86_64 && compile || (echo \"********************************************************************************\nFailed to build.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer address --engine centipede --architecture x86_64 test-project\n********************************************************************************\" && false)"
],
"id": "compile-centipede-address-x86_64"
},
{
"name": "gcr.io/cloud-builders/docker",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-address-x86_64",
"SANITIZER=address"
],
"args": [
"run",
"--platform",
"linux/amd64",
"-v",
"/workspace:/workspace",
"--privileged",
"--cap-add=all",
"-e",
"ARCHITECTURE=x86_64",
"-e",
"FUZZING_ENGINE=centipede",
"-e",
"FUZZING_LANGUAGE=c++",
"-e",
"HOME=/root",
"-e",
"OUT=/workspace/out/centipede-address-x86_64",
"-e",
"SANITIZER=address",
"-t",
"gcr.io/oss-fuzz-base/base-runner",
"bash",
"-c",
"test_all.py || (echo \"********************************************************************************\nBuild checks failed.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer address --engine centipede --architecture x86_64 test-project\npython infra/helper.py check_build --sanitizer address --engine centipede --architecture x86_64 test-project\n********************************************************************************\" && false)"
],
"id": "build-check-centipede-address-x86_64"
},
{
"name": "gcr.io/oss-fuzz-base/base-runner",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-address-x86_64",
"SANITIZER=address"
],
"args": [
"bash",
"-c",
"targets_list > /workspace/targets.list.address"
]
},
{
"name": "gcr.io/oss-fuzz/test-project",
"args": [
"bash",
"-c",
"cd /workspace/out/centipede-address-x86_64 && zip -r test-project-address-202001010000.zip *"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/srcmap.json",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/out/centipede-address-x86_64/test-project-address-202001010000.zip",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/targets.list.address",
"test_url"
]
},
{
"name": "gcr.io/cloud-builders/curl",
"args": [
"-H",
"Content-Type: text/plain",
"-X",
"PUT",
"-d",
"test-project-address-202001010000.zip",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz/test-project",
"args": [
"bash",
"-c",
"rm -r /workspace/out/centipede-address-x86_64"
]
},
{
"name": "gcr.io/cloud-builders/docker",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-none-x86_64",
"SANITIZER=none"
],
"args": [
"run",
"--platform",
"linux/amd64",
"-v",
"/workspace:/workspace",
"--privileged",
"--cap-add=all",
"-e",
"ARCHITECTURE=x86_64",
"-e",
"FUZZING_ENGINE=centipede",
"-e",
"FUZZING_LANGUAGE=c++",
"-e",
"HOME=/root",
"-e",
"OUT=/workspace/out/centipede-none-x86_64",
"-e",
"SANITIZER=none",
"-t",
"gcr.io/oss-fuzz/test-project",
"bash",
"-c",
"rm -r /out && cd /src && cd /src && mkdir -p /workspace/out/centipede-none-x86_64 && compile || (echo \"********************************************************************************\nFailed to build.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer none --engine centipede --architecture x86_64 test-project\n********************************************************************************\" && false)"
],
"id": "compile-centipede-none-x86_64"
},
{
"name": "gcr.io/cloud-builders/docker",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-none-x86_64",
"SANITIZER=none"
],
"args": [
"run",
"--platform",
"linux/amd64",
"-v",
"/workspace:/workspace",
"--privileged",
"--cap-add=all",
"-e",
"ARCHITECTURE=x86_64",
"-e",
"FUZZING_ENGINE=centipede",
"-e",
"FUZZING_LANGUAGE=c++",
"-e",
"HOME=/root",
"-e",
"OUT=/workspace/out/centipede-none-x86_64",
"-e",
"SANITIZER=none",
"-t",
"gcr.io/oss-fuzz-base/base-runner",
"bash",
"-c",
"test_all.py || (echo \"********************************************************************************\nBuild checks failed.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer none --engine centipede --architecture x86_64 test-project\npython infra/helper.py check_build --sanitizer none --engine centipede --architecture x86_64 test-project\n********************************************************************************\" && false)"
],
"id": "build-check-centipede-none-x86_64"
},
{
"name": "gcr.io/oss-fuzz-base/base-runner",
"env": [
"ARCHITECTURE=x86_64",
"FUZZING_ENGINE=centipede",
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/centipede-none-x86_64",
"SANITIZER=none"
],
"args": [
"bash",
"-c",
"targets_list > /workspace/targets.list.none"
]
},
{
"name": "gcr.io/oss-fuzz/test-project",
"args": [
"bash",
"-c",
"cd /workspace/out/centipede-none-x86_64 && zip -r test-project-none-202001010000.zip *"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/srcmap.json",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/out/centipede-none-x86_64/test-project-none-202001010000.zip",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz-base/uploader",
"args": [
"/workspace/targets.list.none",
"test_url"
]
},
{
"name": "gcr.io/cloud-builders/curl",
"args": [
"-H",
"Content-Type: text/plain",
"-X",
"PUT",
"-d",
"test-project-none-202001010000.zip",
"test_url"
]
},
{
"name": "gcr.io/oss-fuzz/test-project",
"args": [
"bash",
"-c",
"rm -r /workspace/out/centipede-none-x86_64"
]
},
{
"name": "gcr.io/cloud-builders/docker",
"env": [
Expand Down
2 changes: 1 addition & 1 deletion infra/ci/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
CANARY_PROJECT = 'skcms'

DEFAULT_ARCHITECTURES = ['x86_64']
DEFAULT_ENGINES = ['afl', 'honggfuzz', 'libfuzzer']
DEFAULT_ENGINES = ['afl', 'honggfuzz', 'libfuzzer', 'centipede']
DEFAULT_SANITIZERS = ['address', 'undefined']


Expand Down