Simplify rust project setup. #3830
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
inferno-chromium
commented
May 17, 2020
•
inferno-chromium
commented
- Add rust and cargo-fuzz in base builder.
- Set RUSTC_BOOSTRAP to make ASan available.
- Set RUSTFLAGS and C,CXXFLAGS properly.
Travis tests have failedHey @inferno-chromium, 1st Build./infra/travis/travis_build.py5th Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: 66052630-985f-11ea-b97e-ed27cf305a90 |
inferno-chromium
changed the title
- Add rust and cargo-fuzz in base builder. - Set RUSTC_BOOSTRAP to make ASan available. - Set RUSTFLAGS and C,CXXFLAGS properly.
Travis tests have failedHey @inferno-chromium, 1st Build./infra/travis/travis_build.py5th Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: 5588caa0-9882-11ea-b97e-ed27cf305a90 |
|
Ignore travis failures, they dont have updated image. |
Travis tests have failedHey @inferno-chromium, 1st Build./infra/travis/travis_build.py5th Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: d331f3c0-9885-11ea-b97e-ed27cf305a90 |
oliverchang
approved these changes
May 17, 2020
|
Seeing build failures now from other projects: |
|
Workaround in #3832. |
inferno-chromium
added a commit
that referenced
this pull request
May 18, 2020
This reverts commit 4f7cf1b.
inferno-chromium
added a commit
that referenced
this pull request
May 18, 2020
|
@jonathanmetzman as fyi |
inferno-chromium
added a commit
that referenced
this pull request
May 18, 2020
| @@ -63,8 +63,24 @@ if [ $FUZZING_ENGINE = "none" ]; then | |||
| export COVERAGE_FLAGS= | |||
| fi | |||
|
|
|||
| export CFLAGS="$CFLAGS $SANITIZER_FLAGS $COVERAGE_FLAGS" | |||
| export CXXFLAGS="$CFLAGS $CXXFLAGS_EXTRA" | |||
| if [[ $FUZZING_LANGUAGE == "rust" ]]; then | |||
There was a problem hiding this comment.
I'm confused why if-else is necessary here. Why can't these rustflags be defined in addition to the C/C++ ones so we don't need to parse the yaml files in helper.py?
There was a problem hiding this comment.
Reason is bc45406#diff-7a7598e90ce6e156a92aad21f6f27990L24, also i tried putting it in comments, as per project owners, it breaks stuff.
There was a problem hiding this comment.
I think it's a bit of a weird workflow for rust now that the project.yaml must be filled it out in order for the build to work (for new projects especially). Wonder if just forcing them to reset some env vars is a better option.
There was a problem hiding this comment.
Max added it to presubmit long time back when he needed it for code coverage stuff. language is a required attribute and added in all projects. Having language there is important i feel, it makes us track what project types we have etc.
https://github.com/google/oss-fuzz/blob/master/infra/presubmit.py#L127
My breakages was trying to fix that language support to be properly available in all places.
There was a problem hiding this comment.
What about Suricata which is a project with both C and Rust ?
There was a problem hiding this comment.
I checked yesterday, we can actually get rid of FUZZING_LANGUAGE and still keep sanitizer flags without breaking rust projects (this is contrary to what developers had in their build.sh). I am unsure what will happen with code coverage support coming soon. What about keeping rust targets in a seperate project.
There was a problem hiding this comment.
Suricata project has fuzz targets written in C that will run C and Rust code.
I am not sure if MSAN can work with this
It seems it works with ASAN (I had caught a panic in Rust with a buggy fuzz target)
chiang831
reviewed
May 25, 2020
| # Instead, use RUSTFLAGS. | ||
| # FIXME: Support code coverage once support is in. | ||
| # See https://github.com/rust-lang/rust/issues/34701. | ||
| export RUSTFLAGS="--cfg fuzzing -Zsanitizer=${SANITIZER} -Cdebuginfo=1 -Cforce-frame-pointers" |
There was a problem hiding this comment.
Hi !
Should we skip -Zsanitizer when $SANITIZER is "coverage" ?
It is not supported by rustc:
https://rustc-dev-guide.rust-lang.org/sanitizers.html
This build failure on cras happened recently: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22447#c2
Build log: https://oss-fuzz-build-logs.storage.googleapis.com/log-3e1434f6-4936-489f-ac03-690ebcbd0ebd.txt
Step #3: Caused by:
Step #3: process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --cfg fuzzing -Zsanitizer=coverage -Cdebuginfo=1 -Cforce-frame-pointers --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
Step #3: --- stderr
Step #3: error: incorrect value `coverage` for debugging option `sanitizer` - one of: `address`, `leak`, `memory` or `thread` was expected
Could you please take a look?
Thanks!
There was a problem hiding this comment.
Sorry about that, excluded it in d70af2f
There was a problem hiding this comment.
Thanks a lot for the help and quick fix!
zchcai
added a commit
to zchcai/oss-fuzz
that referenced
this pull request
Jun 3, 2020
* [cryptofuzz] Add Microsoft SymCrypt (google#3826) * [gRPC-gateway] Initial integration (google#3807) * [doc] Update new_project_guide.md (google#3828) * [knot-dns] link against liblmdb statically (google#3797) Relates to https://gitlab.labs.nic.cz/knot/knot-dns/-/commit/d4ec3a3aa8b7e2ba4d196e2f7984173069e3d91b Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22191 * Simplify rust project setup. (google#3830) * Simplify rust project setup. - Add rust and cargo-fuzz in base builder. - Set RUSTC_BOOSTRAP to make ASan available. - Set RUSTFLAGS and C,CXXFLAGS properly. * Quick fix for helper.py build_fuzzers. (google#3832) * Parse project language and use in infra/helper.py (google#3834) * Parse project language and use in infra/helper.py * Fix exception message. * Add FUZZING_LANGUAGE to build step. (google#3835) * Add fuzzing language correctly during compile step. (google#3837) * Remove unneeded rust hacks after google#3830. (google#3840) * Fix missing FUZZING_LANGUAGE in coverage build. (google#3843) * Use hardcoded FUZZING_LANGUAGE to unbreak CIFuzz. (google#3844) * solidity: Disable Z3 build in dockerfile (google#3831) * cifuzz: use pull_request.number instead of GITHUB_REF (google#3845) Closes google#3732 * wasmtime: build fuzz targets with --all-features (google#3850) This enables not only the binaryen-using fuzz targets, but also the peepmatic fuzz targets (which is necessary after bytecodealliance/wasmtime#1727). * Relax project language requirement. (google#3846) This is necessary for the bisector to be able to build older revisions. Print a warning instead. * libzmq: add maintainer, add UBSAN (google#3829) * Add another maintainer to libzmq's CC list * Enable UBSAN for libzmq * libzmq: disable afl fuzzer, CI fails * libsodium: do not let libsodium's autogen.sh download files from gnu.org There is no sanity check and if the download fails because gnu.org is down the build fails with unhelpful errors * Fixed changes added in clang-11.0. (google#3852) * Bump httplib2 from 0.11.3 to 0.18.0 in /infra/gcb (google#3853) Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.11.3 to 0.18.0. - [Release notes](https://github.com/httplib2/httplib2/releases) - [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG) - [Commits](httplib2/httplib2@v0.11.3...v0.18.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [TiDB] initial integration (google#3849) * [Minify] Initial integration (google#3848) * Set rust flags always in compile step. (google#3859) This is needed for projects that mix both rust and c/c++. * Remove unneeded RUSTC_BOOTSTRAP since nightly is used * [fasthttp] Initial integration (google#3858) * [libfido2] bump libcbor version from 0.5.0 to 0.7.0 (google#3861) * Fix cras build failure, use default rust nightly. * Don't add Rust sanitizer flags when using ubsan or i386. (google#3862) Rust sanitizer flags are not supported for these configs. Fixes Cras and ecc-diff-fuzzer builds. * [hermes] Initial setup for Hermes (google#3698) * [poppler] Exercise a few more methods (google#3857) * [minify] Moved build script upstream (google#3867) * Use available nightly rust for spidermonkey build * Use available rust nightly in spidermonkey-ufi * Use available rust nightly in firefox * [python3-libraries] Fix build (google#3869) * Use latest git version. (google#3870) Adds features needed by firefox and spidermonkey-ufi builds. * libzmq: fix MSAN (google#3868) * libzmq: build libsodium with --disable-asm to avoid false positives in MSAN checks * libzmq: set autoconf install prefix to avoid /usr/local being added to the linker library paths A non-working version of the standard library is installed in oss-fuzz's images under /usr/local/lib. Using it breaks MSAN's tests with false positives. Set the prefix in autoconf to something different from the default /usr/local, as /lib is added by autoconf automatically to the linker flags, which means this broken standard library is used instead of the instrumented one in /usr/lib. * build_specified_commit: revert to using cp. (google#3871) rsync seems to have disappeared in the latest images somehow, and we can't rely on it always existing. * Don't add sanitizer=coverage in RUSTFLAGS * Remove libpcap patch merged upstream (google#3865) * Install Python 3.8.3 on builder image. (google#3874) Part of fixing google#3756. * [qt] Get a shallow clone of qtbase (google#3872) Reduces the size by nearly 200 MB * [qt] Add fuzzer for QImage::loadFromData() (google#3873) * Fix jsoncpp build * build_specified_commit: Don't replace gitdir if already relative. (google#3875) Upgraded git broke this. * [skia] Speculative build fix (google#3847) * [stb] extend code coverage for stb_image (google#3784) * [stb] extend code coverage for stb_image * remove wget's from build.sh * add wget's to Dockerfile * fix Dockerfile * [hermes] Fix coverage build (google#3877) * Update email address to non-alias for clusterfuzz login (google#3878) * add hosts and interns (google#3879) Signed-off-by: Asra Ali <[email protected]> * [qt] Don't install build-essential (google#3880) It's in base-builder now. * [hermes] Add CC for issues (google#3882) * [systemd] turn off hongfuzz (google#3889) See google#3887 * Remove libbz2-dev dependency installed in base-builder for python3 (google#3890) * Remove libbz2-dev dependency installed in base-builder for python3 Fixes google#3888 * Revert "[systemd] turn off hongfuzz (google#3889)" (google#3891) This reverts commit d638fac. * Set up python in one docker layer, remove python deps. (google#3893) * Set up python in one docker layer. Address review comment in google#3890 (review) * Remove all python dependencies, keep build-essential. * Makes suricata work with rust sanitizers (google#3897) * [firefox] Fix libFuzzer cloning process (google#3896) * Fix spidermonkey-ufi build. * [nanopb] Use $(which python3) instead of hardcoding /usr/bin/python3 (google#3895) After the addition of Python 3.8 in google#3874, the default python3 is now /usr/local/bin/python3. The nanopb Dockerfile hardcoded a path of /usr/bin/python3, causing again problems of Python packages being installed for different version than attempting to run with. * Fix CIFuzz issue where targets assumed in OSS-Fuzz build if exists (google#3817) Make some other changes: 1. Refactor fuzz_target.py and fuzz_target_test.py 2. Introduce pyfakefs and parameterized as dependencies and use them in tests. 3. Fix infra-tests in CI so that they use installed dependencies * Fix spidermonkey-ufi build * Extend Zeek project auto_ccs (google#3898) * [libgd] Add another fuzzer to libgd (google#3892) * [qt] Add qtdeclarative (google#3901) * [qt] Add qtdeclarative * [qt] Ignore stdout from make "The job exceeded the maximum log length, and has been terminated." * Updating jsoncpp primary contact (google#3900) * [php] Remove --enable-json (google#3899) JSON is now always compiled in, and cannot be explicitly enabled. * Don't create empty corpus archive as it wont unpack. (google#3903) * Update email addresses for Prometheus (google#3905) Signed-off-by: Julius Volz <[email protected]> * [cryptofuzz] wolfCrypt: Build with support for AES-CFB, AES-OFB (google#3904) * libzmq: enable honggfuzz (google#3909) * Update Prometheus maintainer in Dockerfile (google#3906) ...to myself (Prometheus co-creator). Signed-off-by: Julius Volz <[email protected]> * Revert "[systemd] remove my non-gmail address (google#3606)" (google#3910) This reverts commit 65dcabc. * [GDAL] Move clone of dependencies to upstream repository (google#3911) Cloning of build dependencies and patching is now done by https://github.com/OSGeo/gdal/blob/master/gdal/fuzzers/build.sh * bump libtorrent to fuzz the 2.0 release branch (google#3913) * [FFmpeg] install rsync as it is required for the seed corpus (google#3912) Fixes: Issue 22500 (build failure) * Fix builds_status cron exception. (google#3914) When build log is not found, skip it and avoid 404. Fixes exception ``` Traceback (most recent call last): File "oss-fuzz/infra/gcb/builds_status.py", line 243, in <module> main() File "oss-fuzz/infra/gcb/builds_status.py", line 228, in main status_filename='status.json') File "oss-fuzz/infra/gcb/builds_status.py", line 159, in update_build_status last_build = find_last_build(builds, project, build_tag_suffix) File "oss-fuzz/infra/gcb/builds_status.py", line 102, in find_last_build log.download_to_filename(f.name) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 565, in download_to_filename file_obj, client=client, start=start, end=end) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 537, in download_to_file _raise_from_invalid_response(exc) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1873, in _raise_from_invalid_response raise exceptions.from_http_response(error.response) google.api_core.exceptions.NotFound: 404 GET https://www.googleapis.com/download/storage/v1/b/oss-fuzz-gcb-logs/o/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt?alt=media: No such object: oss-fuzz-gcb-logs/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt Build step 'Execute shell' marked build as failure Finished: FAILURE ``` * Retry building project images in build_specified_commit. (google#3915) To mitigate transient network issues. * Prometheus: Don't use WORKDIR $SRC (google#3908) This makes build_fuzzers work with local checkouts. Without this, it complains with 'Cannot use local checkout with "WORKDIR /src".' Signed-off-by: Julien Pivotto <[email protected]> * [systemd] add DaanDeMeyer's address (google#3916) * [muparser] initial integration (google#3814) * [haproxy] Fix build failures. (google#3884) * Updated the haproxy fuzzers to build again. * The frame decoder needs additional updates since we need to call init_h2 to initialise a memory pool. Disabling this for now as this is a larger change in the code base and will fix up in the coming week. * Fixed hpack decode. * Updated the yaml since we dont want memory sanitizer. * Update project.yaml (google#3917) * install rsync (google#3920) * [LibRaw] Initial integration (google#3918) * Init integration * Fix formatting * Add size check * Update maintainers * PR Feedback * Add newline * Disable UBSan vptr * Disable building examples * Remove disabling UBSan vptr Co-authored-by: Jamie Pinheiro <[email protected]> * ecc-diff-fuzzer buils nettle without openssl (google#3922) * Update Dockerfile (google#3923) * Added contact to zlib-ng project auto_ccs. (google#3924) * [qt] Add pdf to image corpus (google#3926) * [haproxy] Minor fix (google#3928) * Revert "[qt] Add pdf to image corpus (google#3926)" (google#3929) To handle PDFs, QImage requires qtwebengine which doesn't build statically. This reverts commit 934c770. * Fix TensorFlow build (google#3930) * Remove [email protected] as he left Google * Use python3 as python2 is deprecated * Need to manually install numpy now as it is used in the toolchain * Use bazelisk instead of the grep configure -> get Bazel version -> curl hack * Remove C++11 constraint as TF now builds and uses C++14 * Handle review * Remove source sed/replace as it is no longer needed Co-authored-by: Guido Vranken <[email protected]> Co-authored-by: AdamKorcz <[email protected]> Co-authored-by: Daniel Salzman <[email protected]> Co-authored-by: Abhishek Arya <[email protected]> Co-authored-by: Oliver Chang <[email protected]> Co-authored-by: Bhargava Shastry <[email protected]> Co-authored-by: Evgeny Vereshchagin <[email protected]> Co-authored-by: Nick Fitzgerald <[email protected]> Co-authored-by: Luca Boccassi <[email protected]> Co-authored-by: DavidKorczynski <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gabriel Kihlman <[email protected]> Co-authored-by: neildhar <[email protected]> Co-authored-by: tsdgeos <[email protected]> Co-authored-by: Catena cyber <[email protected]> Co-authored-by: Robert Löhning <[email protected]> Co-authored-by: Kevin Lubick <[email protected]> Co-authored-by: Randy <[email protected]> Co-authored-by: Will <[email protected]> Co-authored-by: asraa <[email protected]> Co-authored-by: Christian Holler (:decoder) <[email protected]> Co-authored-by: Petteri Aimonen <[email protected]> Co-authored-by: jonathanmetzman <[email protected]> Co-authored-by: Jon Siwek <[email protected]> Co-authored-by: Google AutoFuzz Team <[email protected]> Co-authored-by: kabeer27 <[email protected]> Co-authored-by: Nikita Popov <[email protected]> Co-authored-by: Julius Volz <[email protected]> Co-authored-by: Even Rouault <[email protected]> Co-authored-by: Arvid Norberg <[email protected]> Co-authored-by: Michael Niedermayer <[email protected]> Co-authored-by: Julien Pivotto <[email protected]> Co-authored-by: Ingo Berg <[email protected]> Co-authored-by: Teju Nareddy <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Nathan Moinvaziri <[email protected]> Co-authored-by: Mihai Maruseac <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.