Fix sanitizer issues

[markus2330]

Hello newcomer

Since 702cd0c one can easily enable undefinied behavior sanitizer.

Simply use clang as toolchain and use debug/asan (e.g. by adding -DENABLE_DEBUG=ON -DENABLE_ASAN=ON -DCMAKE_C_COMPILER=clang-3.8 -DCMAKE_CXX_COMPILER=clang++-3.8 to your cmake invocation). It might work for gcc, too.

Then try to run all tests (see tree.libelektra.org/doc/TESTING.md) and fix some issues that are reported. Do not try to fix all at once, but create a PR as early as possible.

See also:
https://build.libelektra.org/jenkins/job/elektra-clang-asan/lastFailedBuild/console
https://build.libelektra.org/jenkins/job/elektra-gcc-asan/lastFailedBuild/console

[rautesamtr]

Should we open separate bugs per found undefined behavior or just add them here?

[markus2330]

I am not sure if it is useful to list them at all. There are quite many and its easy to trigger them. But you can add them here.

[rautesamtr]

This one in keyset.c
/src/libs/elektra/keyset.c:922:11: runtime error: unsigned integer overflow: 42 + 18446744073709551592 cannot be represented in type 'unsigned long'

the used nickel library seems to have quite some in hash.c that should probably just suppressed as it is not really part of elektra
/elektra/src/plugins/ni/nickel-1.1.0/src/hash.c:

and one in the resolver.c
/src/elektra/src/plugins/resolver/resolver.c:95:16: runtime error: member access within null pointer of type 'resolverHandles' (aka 'struct _resolverHandles')

[markus2330]

I updated the issue to use -DENABLE_ASAN=ON. ENABLE_DEBUG does not activate asan anymore.

[sanssecours]

Since the ASAN build jobs do not build all plugins, here are some other tests that currently also fail on my machine after I enable the address sanitizer:

• testjna_maven
• test_kdb.lua
• test_key.lua
• test_keyset.lua

.

[markus2330]

The error messages would be interesting. If lua/java cause the problems we might have to exclude the tests (at least for the version/OS where it does not work).

I get now a segmentation fault I had not earlier (only with ASAN):

[ 13%] Generating exported_symbols.h
Segmentation fault
src/libs/elektra/CMakeFiles/elektra-core.dir/build.make:53: recipe for target 'src/libs/elektra/exported_symbols.h' failed

Unfortunately, no core dump gets generated, even though they are activated. Have to investigate it later.

[sanssecours]

The error messages would be interesting.

Below are logs of the individual tests. Hope they are helpful.

• org.libelektra.KeySetTest.txt
• org.libelektra.KeyTest.txt
• test_kdb.lua.txt
• test_key.lua.txt
• test_keyset.lua.txt
• TEST-org.libelektra.KeySetTest.xml.txt
• TEST-org.libelektra.KeyTest.xml.txt
  testjna_maven.txt

It looks like the Lua tests fail cause the address sanitizer is not loaded correctly:

35: ==14551==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
35: DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.1.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
35: "interceptors not installed" && 0

.

[markus2330]

Yes, I have the same impression. And for Java it seems like libelektra is not found at all, do the tests work without ASAN? Disabling these tests for ASAN&&APPLE could be an option?

[sanssecours]

…do the tests work without ASAN?

Yes.

Disabling these tests for ASAN&&APPLE could be an option?

Do these tests work on Linux with enabled ASAN? Anyway, as long as I do not need any functionality of the Java or Lua bindings, disabling these tests if address sanitization is enabled is fine with me.

[markus2330]

René Schwaiger wrote:

Yes.

Strange, seems like we have to dig more into details what ASAN does. Why would it cause to not find libraries? For me now it yields some crashes when `LD_LIBRARY_PATH` is set (without any useful stack trace). It might also be related to not finding some libs (at runtime). Also the build process failed because of the segfaults, I needed to manually invoke: ``` build-64/src/include> ../../bin//elektra-export-errors /home/markus/Projekte/Elektra/current/src/error/specification /home/markus/Projekte/Elektra/build/src/include/kdberrors.h build-64/src/libs/elektra> ../../../bin//elektra-export-symbols resolver_fm_hpu_b dump augeas base64 blockresolver boolean c cachefilter ccode conditionals constants counter crypto_botan crypto_gcrypt crypto_openssl csvstorage curlget date dbus desktop doc dpkg dump enum error fcrypt file filecheck fstab glob hexcode hidden hosts iconv ini iterate jni journald keytometa line lineendings list logchange lua mathcheck mini mozprefs multifile network ni noresolver null passwd path profile python python2 range regexstore rename required resolver_fm_b_b resolver_fm_hb_b resolver_fm_hp_b resolver_fm_hpu_b resolver_fm_pb_b resolver_fm_ub_x resolver_fm_uhb_xb resolver_fm_xb_x resolver_fm_xhp_x resolver_fm_xp_x semlock shell simpleini simplespeclang spec struct sync syslog tcl template timeofday tracer type uname validation wresolver xerces xmltool yajl yaml ``` This problem was not here earlier, but its hard to find out which change causes this problem.

> Disabling these tests for ASAN&&APPLE could be an option? Do these tests work on Linux with enabled ASAN? Anyway, as long as I do not need any functionality of the Java or Lua bindings, disabling these tests if address sanitization is enabled is fine with me.

At least lua worked for me (with ASAN): ``` Start 38: test_keyset.lua 38/172 Test #38: test_keyset.lua ......................... Passed 0.01 sec Start 39: test_kdb.lua 39/172 Test #39: test_kdb.lua ............................ Passed 0.12 sec Start 40: test_key.lua 40/172 Test #40: test_key.lua ............................ Passed 0.01 sec ``` Maven seems to also not work with Linux (ASAN): ``` Start 32: testjna_maven 32/174 Test #32: testjna_maven ...........................***Failed 1.73 sec

…

------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.libelektra.KeySetTest ==503==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING. ``` So I would say you can disable the maven/java/jni tests with ASAN on any system and lua only for Mac OS X? Other tests fail though because of these segfault issues (mainly shell recorder tests). But this seems to be a completely different problem, maybe ASAN is not installed correctly on my system?

[sanssecours]

So I would say you can disable the maven/java/jni tests with ASAN on any
system and lua only for Mac OS X?

Sounds good to me. I included commits that disable the tests in pull request #1548.