test executable_dll_and_plugin fails on Linux, GCC 8.1.0, -fsanitize=address

[jktjkt]

Description

Using commit b2611a1 (latest master), test executable_dll_and_plugin fails:

31/32 Test #31: executable_dll_and_plugin ...........***Failed    0.02 sec
Files "/home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt" to "/home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt" are different.
==========================================================================
== CONTENTS OF /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt
==========================================================================
[doctest] run with "--help" for options
I am a test from the dll!
I am a test from the implementation!
I am a test from the implementation_2!
I am a test from the executable!
===============================================================================
main.cpp(0):
TEST CASE:  executable

main.cpp(0): ERROR: test case THREW exception: char: 97

I am a test from the plugin!
===============================================================================
[doctest] test cases:      5 |      4 passed |      1 failed |      0 skipped
[doctest] assertions:      0 |      0 passed |      0 failed |
[doctest] Status: FAILURE!

==========================================================================
== CONTENTS OF /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt
==========================================================================
[doctest] run with "--help" for options
I am a test from the dll!
I am a test from the implementation!
I am a test from the implementation_2!
I am a test from the executable!
===============================================================================
main.cpp(0):
TEST CASE:  executable

main.cpp(0): ERROR: test case THREW exception: unknown exception

===============================================================================
[doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
[doctest] assertions:      0 |      0 passed |      0 failed |
[doctest] Status: FAILURE!

==========================================================================
== CONTENTS END
==========================================================================
CMake Error at /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/scripts/cmake/exec_test.cmake:57 (message):
  Running
  '/home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/executable_dll_and_plugin
  --no-version --dt-no-path-filenames=1 --dt-no-line-numbers=1
  --dt-no-exitcode=1 --dt-gnu-file-line=0' ended with code 'Output is
  different from reference file!'

It would be nice if the test suite run diff automatically, but anyway, here we go:

--- /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt    2019-03-19 11:40:17.881073684 +0100
+++ /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt 2019-03-19 11:47:10.422304360 +0100
@@ -7,10 +7,9 @@
 main.cpp(0):
 TEST CASE:  executable
 
-main.cpp(0): ERROR: test case THREW exception: char: 97
+main.cpp(0): ERROR: test case THREW exception: unknown exception
 
-I am a test from the plugin!
 ===============================================================================
-[doctest] test cases:      5 |      4 passed |      1 failed |      0 skipped
+[doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
 [doctest] assertions:      0 |      0 passed |      0 failed |
 [doctest] Status: FAILURE!

Extra information

• doctest version: b2611a1
• Operating System: Gentoo Linux, amd64
• Compiler+version: GCC 8.1.0
• cmake invocation (via QtCreator IDE, sorry for a non-exact form of parameters)
  Starting to parse CMake project, using: "-DCMAKE_BUILD_TYPE:STRING=Debug", "-DCMAKE_CXX_COMPILER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-g++", "-DCMAKE_CXX_FLAGS:STRING=-fsanitize=address", "-DCMAKE_C_COMPILER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-gcc-8.1.0", "-DCMAKE_C_FLAGS:STRING=-fsanitize=address", "-DCMAKE_EXE_LINKER_FLAGS:STRING=-fsanitize=address", "-DCMAKE_EXPORT_COMPILE_COMMANDS:INTERNAL=ON", "-DCMAKE_INSTALL_PREFIX:STRING=/opt/nc", "-DCMAKE_INSTALL_RPATH:INTERNAL=/opt/nc/lib64", "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:INTERNAL=ON", "-DCMAKE_LINKER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-gcc-8.1.0", "-DCMAKE_MODULE_LINKER_FLAGS:STRING=-fsanitize=address", "-DCMAKE_PREFIX_PATH:STRING=/opt/nc", "-DCMAKE_SHARED_LINKER_FLAGS:STRING=-fsanitize=address", "-DENABLE_BUILD_TESTS:STRING=ON", "-DGEN_CPP_BINDINGS:BOOL=ON", "-DGEN_JAVA_BINDINGS:BOOL=OFF", "-DGEN_LANGUAGE_BINDINGS:BOOL=ON", "-DGEN_LUA_BINDINGS:BOOL=OFF", "-DGEN_PYTHON_BINDINGS:BOOL=OFF", "-DKEYSTORED_DEFER_SSH_KEY:BOOL=ON", "-DKEYSTORED_KEYS_DIR:INTERNAL=/opt/nc/etc/keystored/keys", "-DSYSREPOCTL_ROOT_PERMS:INTERNAL=-p 600", "-DTHREADS_HAVE_PTHREAD_ARG:BOOL=ON", "-DUSE_SR_MEM_MGMT:BOOL=OFF".

Custom env:

CC=gcc-8.1.0
CFLAGS=-ggdb -O2 -march=native -fsanitize=address
CXX=g++-8.1.0
CXXFLAGS=-ggdb -O2 -march=native -fsanitize=address
LDFLAGS=-fsanitize=address
LD_LIBRARY_PATH=/opt/nc/lib64
PKG_CONFIG_PATH=/opt/nc/lib64/pkgconfig:/opt/nc/lib/pkgconfig 

I have not dug into this, but given that this is an ASAN build, it would be an error if the test links together TUs compiled or linked with and without -fsanitize=address. Maybe that's irrelevant. Either way, I'm quite sure that my build env is sane.

[onqtam]

doctest runs all the tests on the travis CI with asan/ubsan/tsan and this passes there without trouble...

Can you test with Clang as well? The sanitizers are usually best integrated with clang.

I've hit many such toolchain issues in the past where a specific version of the compiler or something else is causing problems and I'm highly suspicious that this is the case here as well.

[jktjkt]

I am not hitting this with clang, neither v6, nor v7. However, the issue persists when I try with the latest GCC (8.3.0). I believe that sanitizers actually share the same backend these days (upstream says so, at least), and given that the latest compiler release from GCC fails the test suite, I think that this is relevant.

[onqtam]

So the problem is present both with gcc 8.1 and 8.3? This is the GCC that is used on the CI: g++-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0 and there is no such issue with any of these compilers: https://travis-ci.org/onqtam/doctest (all tested in Debug/Release - but I'm using the gold linker for all sanitizer tests).

About the sanitizers - I've previously reported bugs to GCC on linux about their integration with the sanitizers (example) - there were issues not present with clang, so it is very much a possibility that something isn't right for a particular patch of gcc. Maybe -march=native is the problem - I don't know - could you test without it? Does the issue persist without the sanitizers? I've had spectacular failures of normal exception use on OSX using an array of different gcc versions installed through homebrew - and there are no longer any CI builds on OSX with gcc - only clang from XCode. Toolchains can be amazingly broken.

Maybe something else on your platform is causing this since it is a project with multiple shared objects. Try with the gold linker? I use it because there were issues with unrecognized flags by the default linker on the CI like --push-state (or something like that) which when the sanitizers were enabled were passed to the linker (link).

doctest is using exceptions in a standards-conforming and simple way so I seriously doubt that there is an issue on the side of the framework. I'm keeping the issue open but for now I have no clue as to what the reason might be.

[onqtam]

seems that this is the flag which fixes the issue: -static-libasan - see this comment: https://github.com/onqtam/doctest/pull/285/files#r333711432