bitcoin-core: Migrate to CMake #1

hebasto · 2024-01-19T08:19:29Z

This testing PR was requested by reviewers of hebasto/bitcoin#43 and is not intended to be merged.

hebasto · 2024-01-19T09:45:00Z

projects/bitcoin-core/build.sh

 if [ "$SANITIZER" = "memory" ]; then
-  CONFIG_SITE="$PWD/depends/$BUILD_TRIPLET/share/config.site" ./configure --enable-fuzz SANITIZER_LDFLAGS="$LIB_FUZZING_ENGINE" --disable-hardening --with-asm=no
+  cmake -S .. -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" --toolchain depends/${BUILD_TRIPLET}/share/toolchain.cmake -DCMAKE_BUILD_TYPE=None -DFUZZ=ON -DSANITIZER_LDFLAGS="$LIB_FUZZING_ENGINE" -DHARDENING=OFF -DASM=OFF


Moved from google#11516 (comment)

Shouldn't CC and CXX be coming from the toolchain file? Why do we need to (re-)set it explicitly here?

@fanquake

Thanks! ~~Fixed~~.

Sorry, but I recalled the reason of explicit setting the CMAKE_C_COMPILER and CMAKE_CXX_COMPILER variables. It makes the toolchain file skip processing of the CFLAGS and CXXFLAGS, which avoids their duplication.

Otherwise, it might look like this:

C compiler ............................ /usr/local/bin/clang CFLAGS ................................ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument C++ compiler .......................... /usr/local/bin/clang++ CXXFLAGS .............................. -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument -stdlib=libc++ -flto=thin -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument -stdlib=libc++ -flto=thin

So, reverted back and commented.

I'm not sure. Doesn't that mean anyone building depends will also always have this problem of duplicate flags, and need to use this "workaround"? Seems like something that needs solving, otherwise it defeats the point of the toolchain file?

Doesn't that mean anyone building depends will also always have this problem of duplicate flags, and need to use this "workaround"?

One, who builds with depends, usually provides CFLAGS and / or CXXFLAG to the make command directly, avoiding of setting these environment variables globally.

Sure, but that shouldn't matter? Because if we pass them to depends, they should get put into the toolchain file, which is then used for compilation?

That is what makes the duplication, because CMake still considers the CFLAGS and CXXFLAGS environment variables.

Right. So this working around a CMake thing, that would also be a problem for anyone building using CMake somewhere that has ENV vars set? Because the flags they think they are using, might actually be getting overridden by environment variables? This would potentially be an issue for anyone using a toolchain file?

FWIW, the flag duplication is a fact in the master branch. For example, https://github.com/google/oss-fuzz/actions/runs/7529480132/job/20493782120:

CC = clang CFLAGS = -pthread -m32 -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -m32 -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link CPPFLAGS = -DABORT_ON_FAILED_ASSUME -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -DHAVE_BUILD_INFO -D_FILE_OFFSET_BITS=64 -DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE -m32 -I/src/bitcoin-core/depends/i686-pc-linux-gnu/include/ -DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE -m32 CXX = clang++ -std=c++20 CXXFLAGS = -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -m32 -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -flto=thin -m32 -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -flto=thin LDFLAGS = -lpthread -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -pie -flto=thin -L/src/bitcoin-core/depends/i686-pc-linux-gnu/lib -flto=thin AR = llvm-ar ARFLAGS = cr

This would potentially be an issue for anyone using a toolchain file?

I lean to agree with you. However, the current CMake-based build system behavior just mirrors Autotools-based one.

And the user experience could be improved after migration, no?

hebasto · 2024-01-19T15:07:42Z

Ready for review :)

9b9f562 cmake: Add fuzzing options (Hennadii Stepanov) ea7861e cmake: Add `SANITIZERS` option (Hennadii Stepanov) Pull request description: New CMake variables that affect the build system configuration: - `SANITIZERS` - `BUILD_FUZZ_BINARY` - `FUZZ` In the light of bitcoin#29189, this PR is no longer based on #41. However, the `test/fuzz/script_bitcoin_consensus.cpp` might be easily added anytime. For OSS-Fuzz integration, please refer to hebasto/oss-fuzz#1. ACKs for top commit: theuni: Lightly tested ACK 9b9f562. Tree-SHA512: f762d1218f2f8bfe26bdd43f431cb37a26093a6899db7120b50df3083f81d6ad6aa867937e69930faff6ab4519532cfaca48aaf97831d4c2593b93423cb6d41a

dergoegge · 2024-02-01T09:49:25Z

projects/bitcoin-core/build.sh

+  `# This avoids CFLAGS and CXXFLAGS duplication by skipping their processing in the toolchain file.` \
+  -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" \
+  --toolchain depends/${BUILD_TRIPLET}/share/toolchain.cmake \
+  -DCMAKE_BUILD_TYPE=None \


https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html

None is not documented, what does it do?

It does nothing, which means, no per-configuration flags will be appended after ones provided by the toolchain file.

FWIW, it is a part of the CMake code:

set(CMAKE_BUILD_TYPE "${CMAKE_BUILD_TYPE_INIT}" CACHE STRING "Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel ...")

Base PR apache/brpc#2420 ; NOTE: I can't enable memory sanitizer due to ```log BAD BUILD: /tmp/not-out/tmpmptlk01q/fuzz_esp seems to have either startup crash or exit: /tmp/not-out/tmpmptlk01q/fuzz_esp -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null Uninitialized bytes in MemcmpInterceptorCommon at offset 15 inside [0x7030000000f0, 19) ==428==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x682b90 in __interceptor_memcmp /src/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:892:10 #1 0x7fa8ef4cf62a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::FindLastLessOrEqual(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15062a) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #2 0x7fa8ef4d259f in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15359f) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #3 0x7fa8ef4d2a15 in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddFile(google::protobuf::FileDescriptorProto const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x153a15) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #4 0x7fa8ef4cebef in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x14fbef) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #5 0x7fa8ef499f43 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11af43) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #6 0x7fa8ef49281d in protobuf_google_2fprotobuf_2fapi_2eproto::AddDescriptorsImpl() (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11381d) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) ``` Signed-off-by: Arjun Singh <[email protected]>

@AdamKorcz

Fix build error https://oss-fuzz-build-logs.storage.googleapis.com/log-17235690-03af-4f88-a2c8-1f7203f4695c.txt ```Step #1: Step 3/4 : COPY build.sh *.dict $SRC Step #1: When using COPY with more than one source file, the destination must be a directory and end with a / Finished Step #1 ``` @AdamKorcz

- OSS-Fuzz builds are failing after pigweed was rebased in project-chip/connectedhomeip#35644 - One of the failures is related to pigweed becoming incompatible with python <3.9. Such as using subscript notation in the type hints. - Fix: Base images in OSS-Fuzz use python 3.8, This PR aims to force the usage of python3.10 instead ### Example Error ``` Step #1: Traceback (most recent call last): Step #1: File "../../third_party/pigweed/repo/pw_build/py/pw_build/python_runner.py", line 38, in <module> Step #1: import gn_resolver # type: ignore Step #1: File "/src/connectedhomeip/third_party/pigweed/repo/pw_build/py/pw_build/gn_resolver.py", line 319, in <module> Step #1: _Actions = Iterator[tuple[_ArgAction, str]] Step #1: TypeError: 'type' object is not subscriptable Step #1: [137/1234] ln -f ../../third_party/pigweed/repo/pw_thread/pw_thread_protos/thread_snapshot_service.proto ```

hebasto force-pushed the 240119-cmake branch from 3be975a to 32fb361 Compare January 19, 2024 08:54

hebasto commented Jan 19, 2024

View reviewed changes

hebasto marked this pull request as draft January 19, 2024 09:56

hebasto force-pushed the 240119-cmake branch 2 times, most recently from eb22249 to 0b23dc7 Compare January 19, 2024 11:39

hebasto mentioned this pull request Jan 19, 2024

cmake: Add fuzzing facilities hebasto/bitcoin#43

Merged

hebasto force-pushed the 240119-cmake branch 2 times, most recently from b293af6 to deb912a Compare January 19, 2024 13:58

bitcoin-core: Migrate to CMake

28d8c9f

hebasto force-pushed the 240119-cmake branch from deb912a to 28d8c9f Compare January 19, 2024 14:28

hebasto mentioned this pull request Jan 24, 2024

cmake: Rework handling compiler flags hebasto/bitcoin#82

Merged

dergoegge reviewed Feb 1, 2024

View reviewed changes

hebasto closed this Mar 24, 2024

hebasto deleted the 240119-cmake branch August 28, 2024 13:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bitcoin-core: Migrate to CMake #1

bitcoin-core: Migrate to CMake #1

hebasto commented Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024

hebasto Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024

fanquake Jan 19, 2024

hebasto Jan 19, 2024

fanquake Jan 19, 2024

hebasto Jan 19, 2024 •

edited

Loading

fanquake Jan 19, 2024

hebasto Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024

hebasto commented Jan 19, 2024

dergoegge Feb 1, 2024 •

edited

Loading

hebasto Feb 1, 2024

hebasto Feb 1, 2024

bitcoin-core: Migrate to CMake #1

bitcoin-core: Migrate to CMake #1

Conversation

hebasto commented Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

hebasto Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto commented Jan 19, 2024

dergoegge Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto commented Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024 •

edited

Loading

hebasto Jan 19, 2024 •

edited

Loading

dergoegge Feb 1, 2024 •

edited

Loading