Improve asan and diagnostics in CI

[MonsieurNicolas]

This PR rerun tests under a debugger as to catch the backtrace and output it in the logs.

This allows to diagnose issues caught by asan much quicker as it reduces the need to repro issues locally.

We can probably improve this further in the future to give better context outside of crashes (by passing -b option when running tests for example).

Example below (output from make check under the same conditions than ci-build.sh):

local:1/0/100%/0.0s 1   PostgreSQL enabled for tests using temporary database cluster
1       Filters: bucketmanager do not leak empty-merge futures
1       "bucketmanager do not leak empty-merge futures" bucket/test/BucketManagerTests.cpp:434
1       .Test failed, rerunning with debugger
1       ./stellar-core test --ll fatal -w NoTests -a -r simple --all-versions --base-instance 0 bucketmanager do not leak empty-merge futures
1       ./test/run-selftest-pg: line 109: 15238 Aborted                 ./stellar-core test $STELLAR_CORE_TEST_PARAMS --base-instance $BASE_INSTANCE "$TESTS" 2> /dev/null
local:1/0/100%/0.0s 1   (lldb) target create "./stellar-core"
1       Traceback (most recent call last):
1         File "<string>", line 1, in <module>
1       ModuleNotFoundError: No module named 'lldb.embedded_interpreter'
local:1/0/100%/0.0s 1   Current executable set to '/home/nicolas/src/stellar-core/src/stellar-core' (x86_64).
1       (lldb) settings set -- target.run-args  "test" "--ll" "fatal" "-w" "NoTests" "-a" "-r" "simple" "--all-versions" "--base-instance" "0" "bucketmanager do not leak empty-merge futures"
1       (lldb) r
1ocal:1/0/100%/0.0s 1   Filters: bucketmanager do not leak empty-merge futures
1       "bucketmanager do not leak empty-merge futures" bucket/test/BucketManagerTests.cpp:434
local:1/0/100%/0.0s 1   .Process 15285 stopped
1       * thread #1, name = 'stellar-core', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
1           frame #0: 0x0000000000b87cf5 stellar-core`::resetEvictionState() at TxQueueLimiter.cpp:229:34
1          226  TxQueueLimiter::resetEvictionState()
1          227  {
1          228      mLaneEvictedFeeBid.assign(
1       -> 229          mSurgePricingLaneConfig->getLaneOpsLimits().size(), {0, 0});
1                                                ^
1          230  }
1          231  }
1       Process 15285 launched: '/home/nicolas/src/stellar-core/src/stellar-core' (x86_64)
1       (lldb) bt
local:1/0/100%/0.0s 1   * thread #1, name = 'stellar-core', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
1         * frame #0: 0x0000000000b87cf5 stellar-core`::resetEvictionState() at TxQueueLimiter.cpp:229:34
1           frame #1: 0x0000000000b3596f stellar-core`::shift() at TransactionQueue.cpp:837:22
1           frame #2: 0x00000000008f9339 stellar-core`::updateTransactionQueue() at HerderImpl.cpp:1866:23
1           frame #3: 0x00000000008f8844 stellar-core`::newSlotExternalized() at HerderImpl.cpp:222:9
1           frame #4: 0x00000000008fd232 stellar-core`::valueExternalized() at HerderImpl.cpp:339:9
1           frame #5: 0x00000000009eefb9 stellar-core`::valueExternalized() at HerderSCPDriver.cpp:801:17
1           frame #6: 0x0000000000913cf4 stellar-core`::externalizeValue() at HerderImpl.cpp:718:26
1           frame #7: 0x0000000001b4f9bf stellar-core`::closeLedger() at BucketTestUtils.cpp:62:21
1           frame #8: 0x0000000001b507d6 stellar-core`::closeLedger() at BucketTestUtils.cpp:71:12
1           frame #9: 0x0000000001a30a31 stellar-core`::C_A_T_C_H_T_E_S_T_10() at BucketManagerTests.cpp:464:13
1           frame #10: 0x0000000002d1965d stellar-core`::invokeActiveTestCase() [inlined] invoke at catch.hpp:14171:15
1           frame #11: 0x0000000002d19611 stellar-core`::invokeActiveTestCase() at catch.hpp:13027:27
1           frame #12: 0x0000000002d11e73 stellar-core`::runCurrentTest() at catch.hpp:13000:17
1           frame #13: 0x0000000002d0e934 stellar-core`::runTest() at catch.hpp:12761:13
1           frame #14: 0x0000000002d2609f stellar-core`::runInternal() [inlined] execute at catch.hpp:13354:45
1           frame #15: 0x0000000002d25ce5 stellar-core`::runInternal() at catch.hpp:13564:35
1           frame #16: 0x0000000002d23072 stellar-core`::run() at catch.hpp:13520:24
1           frame #17: 0x0000000002d8de87 stellar-core`::runTest() at test.cpp:430:22
1           frame #18: 0x0000000001203c53 stellar-core`::handleCommandLine() [inlined] operator() at std_function.h:590:9
1           frame #19: 0x0000000001203c06 stellar-core`::handleCommandLine() [inlined] run at CommandLine.cpp:539:12
1           frame #20: 0x0000000001203c06 stellar-core`::handleCommandLine() at CommandLine.cpp:1845:28
1           frame #21: 0x0000000001445195 stellar-core`main at main.cpp:230:12
1           frame #22: 0x00007ffff7a33d90 libc.so.6`__libc_start_call_main(main=(stellar-core`main at main.cpp:207), argc=13, argv=0x00007fffffffd9d8) at libc_start_call_main.h:58:16
1           frame #23: 0x00007ffff7a33e40 libc.so.6`__libc_start_main_impl(main=(stellar-core`main at main.cpp:207), argc=13, argv=0x00007fffffffd9d8, init=0x00007ffff7ffd040, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffd9c8) at libc-start.c:392:3
1           frame #24: 0x000000000048ac45 stellar-core`_start + 37
1       (lldb) exit
local:1/0/100%/0.0s 1   waiting for server to shut down.... done
1       server stopped
local:1/0/100%/0.0s parallel: This job failed:
runpart 1 'bucketmanager do not leak empty-merge futures'

[dmkozh]

This is great, thank you!

[dmkozh]

r+ c026231