ci: Run ASan/LSan and reorganize sanitizer and Valgrind jobs

[real-or-random]

This enables asan (including lsan), and restructures the sanitizer builds. It also removes -fno-omit-frame-pointer again. This is debatable. The main reason for removing this is that GCC (but not clang) fails to build (runs out of registers) when combining asan, -fno-omit-frame-pointer, and the x86_64 asm. But I believe without -fno-omit-frame-pointer, the instrumented binary may be closer to the real binary, which is good. If we get unusable debugging output on CI without frame pointers, we can still reproduce the build locally.

• ECMULTGENPRECISION=8 : We should disable this entirely, not only on CI, because it's slow and needs too much stack space

[real-or-random]

A few things are weird on MacOS, and I don't really understand what's going on:

• jobs 49 and 50 fail because lsan does not work with Apple's clang, see config.log. That's expected according to Mac OS X Sierra: detect_leaks is not supported on this platform google/sanitizers#1026
• jobs 67 succeeds but 68 fails, with the same error message as above. But why the difference? I have no idea, they should use the same toolchain etc. Is this related to gmp? But 67 seem to fail with the "native compiler", and we don't set $CC_FOR_BUILD correctly for gcc. I've added this now.
• I'm not convinced that the "gcc" jobs (including 67, 68) really use gcc. It's again Apple's fake gcc according to the $CC --version in after_script. @elichai I added a $CC --version in the script, let's see if this makes a difference. (This will be obsolete anyway when we switch away from travis).

And job 32 times out... well that's not nice but at least I understand what's going on ...

[elichai]

• I'm not convinced that the "gcc" jobs (including 67, 68) really use gcc. It's again Apple's fake gcc according to the $CC --version in after_script. @elichai I added a $CC --version in the script, let's see if this makes a difference. (This will be obsolete anyway when we switch away from travis).

IIRC that's why I did CC=gcc-9 because OSX has an alias for gcc=clang
https://github.com/bitcoin-core/secp256k1/pull/750/files#diff-5c41b2883cfdc97e6b8a07ebe6456643de0160099b7f15ed04afeb2fd627f630R14

secp256k1/contrib/travis.sh

Lines 10 to 13 in 4232e5b

	if [ "$TRAVIS_OS_NAME" = "osx" ] && [ "$TRAVIS_COMPILER" = "gcc" ]
	then
	export CC="gcc-9"
	fi

[real-or-random]

IIRC that's why I did CC=gcc-9 because OSX has an alias for gcc=clang

Apparently that does not seem to work (anymore?).

I think we shouldn't try to fix this now but first move to another CI platform. For example, Cirrus seems to ship with GCC cirruslabs/cirrus-ci-docs#67 .

[real-or-random]

Rebased to work with Cirrus. I also updated the initial comment.

[sipa]

utACK 1191bb0

[jonasnick]

Remind me, what's the reason for not running this (and valgrind) in make check? If it was just about being able to set the test "count"/TEST_ITERS separately then we do have an env var for this now, and could remove it like this https://github.com/jonasnick/secp256k1/commits/202010_ci_matrix-jn (untested).

[sipa]

I think it's just that make check will try to run without the wrappers.

[jonasnick]

You mean make check is run without a wrapper even when a wrapper is set? Afaics no, because we unset BUILD whenever we set WRAPPER_CMD.

[real-or-random]

The main issue (and I think this is what Pieter is partly saying) is that ${WRAPPER_CMD} make check typically won't work for all wrapper commands.

• valgrind make check will memcheck make but will not memcheck our tests because they'll be child processes. (Now you could try valgrind --trace-children but then it will still additionally memcheck make which gives false positives and wastes CI time.)
• wine make check just won't work at all because make is not a Windows executable.
• I haven't tried QEMU but I expect the same as for Wine (because we cross-build).

What we could is to tell make to use the wrapper command. Now that I'm reading up on this again, I recall that it's not even more work to do this. There's the (obscurely named) LOG_COMPILER env variable, which does exactly this. You can even pass LOG_COMPILER="./libtool --mode=execute valgrind" etc, which we don't need now but maybe in the future. Running all tests using make check will have the benefit that the logs always end up in the same place in the CI output. Let me give it a shot.

And thanks for reminding me of SECP256K1_TEST_ITERS. When I was writing this, I didn't know that this exists (I reinvented the name TEST_ITERS). So we should simply set this variable instead, like we do for the benchmarks. This has the nice side effect that it would apply to the ordinary make check builds if we'd like to set it there for some reason.

[real-or-random]

updated as described above

edit: pushed again, I had forgotten the use_aligned=1.

[real-or-random]

Hm, it's still timing out. But everything looks correct including the test iterations (like in the revision yesterday, when the longest job took ~ 40 min.) If I'm not mistaken, the only difference to now is that we run the tests in make check and this means the only real difference is that they run concurrently with -j2 and we allocate only one CPU. I pushed a commit to see if this fixes the timeout.

[real-or-random]

Ok, ignore my last comment. The tests were running twice because I screwed up when rebasing. I hope it's really ready now...

[jonasnick]

Nice!

ACK 02dcea1 spot-checked ci output, checked that when valgrind ./tests crashes then LOG_COMPILER=valgrind make check also crashes.

[sipa]

utACK 02dcea1