test_framework thread sanitizer

[Oipo]

Data race + deadlock:

==================
WARNING: ThreadSanitizer: data race (pid=64112)
  Write of size 8 at 0x7b0c00000548 by main thread (mutexes: write M29408895206162704, write M34193969810243856, write M37853144507482384, write M41512319204720912, write M45452968878670096, write M48549193622487552, write M52771318273147392, write M58119342830649856, write M62622942458020352, write M70504241805918720, write M75007841433289216, write M78948491107238400, write M82607665804476928, write M89644540222243328, write M329):
    #0 arrayList_add /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:187 (libcelix_utils.so.2+0x92b9)
    #1 fw_addBundleListener /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:1490 (libcelix_framework.so.2+0x29f20)
    #2 celix_bundleContext_trackBundlesWithOptions /home/oipo-unencrypted/Programming/celix/libs/framework/src/bundle_context.c:567 (libcelix_framework.so.2+0x2055a)
    #3 celix_bundleContext_trackBundles /home/oipo-unencrypted/Programming/celix/libs/framework/src/bundle_context.c:597 (libcelix_framework.so.2+0x2078b)
    #4 CelixBundleContextBundlesTests_trackBundlesTest_Test::TestBody() /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/bundle_context_bundles_tests.cpp:309 (test_framework+0x4c6d7)
    #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (test_framework+0xa59a9)
    #6 main /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/run_tests.cpp:24 (test_framework+0x410d0)

  Previous read of size 8 at 0x7b0c00000548 by thread T12:
    #0 arrayList_size /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:104 (libcelix_utils.so.2+0x8cd8)
    #1 fw_eventDispatcher /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:2003 (libcelix_framework.so.2+0x2bb6e)
    #2 <null> <null> (libtsan.so.0+0x2b3c6)

  Location is heap block of size 48 at 0x7b0c00000540 allocated by main thread:
    #0 calloc <null> (libtsan.so.0+0x2cdbd)
    #1 celix_arrayList_createWithEquals /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:365 (libcelix_utils.so.2+0xa0f6)
    #2 celix_arrayList_create /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:361 (libcelix_utils.so.2+0xa0b8)
    #3 arrayList_createWithEquals /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:45 (libcelix_utils.so.2+0x8843)
    #4 arrayList_create /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:41 (libcelix_utils.so.2+0x8808)
    #5 fw_init /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:404 (libcelix_framework.so.2+0x2521e)
    #6 framework_start /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:527 (libcelix_framework.so.2+0x25d9d)
    #7 celix_frameworkFactory_createFramework /home/oipo-unencrypted/Programming/celix/libs/framework/src/celix_framework_factory.c:33 (libcelix_framework.so.2+0x4845d)
    #8 CelixBundleContextBundlesTests::CelixBundleContextBundlesTests() <null> (test_framework+0x51b3e)
    #9 CelixBundleContextBundlesTests_trackBundlesTest_Test::CelixBundleContextBundlesTests_trackBundlesTest_Test() /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/bundle_context_bundles_tests.cpp:284 (test_framework+0x522a8)
    #10 testing::internal::TestFactoryImpl<CelixBundleContextBundlesTests_trackBundlesTest_Test>::CreateTest() <null> (test_framework+0x56abb)
    #11 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) <null> (test_framework+0xa5b59)
    #12 main /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/run_tests.cpp:24 (test_framework+0x410d0)

  Mutex M29408895206162704 is already destroyed.

  Mutex M34193969810243856 is already destroyed.

  Mutex M37853144507482384 is already destroyed.

  Mutex M41512319204720912 is already destroyed.

  Mutex M45452968878670096 is already destroyed.

  Mutex M48549193622487552 is already destroyed.

  Mutex M52771318273147392 is already destroyed.

  Mutex M58119342830649856 is already destroyed.

  Mutex M62622942458020352 is already destroyed.

  Mutex M70504241805918720 is already destroyed.

  Mutex M75007841433289216 is already destroyed.

  Mutex M78948491107238400 is already destroyed.

  Mutex M82607665804476928 is already destroyed.

  Mutex M89644540222243328 is already destroyed.

  Mutex M329 (0x7b4c00000210) created at:
    #0 pthread_mutex_init <null> (libtsan.so.0+0x2e561)
    #1 celixThreadMutex_create /home/oipo-unencrypted/Programming/celix/libs/utils/src/celix_threads.c:102 (libcelix_utils.so.2+0x1020e)
    #2 framework_create /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:251 (libcelix_framework.so.2+0x244f5)
    #3 celix_frameworkFactory_createFramework /home/oipo-unencrypted/Programming/celix/libs/framework/src/celix_framework_factory.c:30 (libcelix_framework.so.2+0x48430)
    #4 CelixBundleContextBundlesTests::CelixBundleContextBundlesTests() <null> (test_framework+0x51b3e)
    #5 CelixBundleContextBundlesTests_trackBundlesTest_Test::CelixBundleContextBundlesTests_trackBundlesTest_Test() /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/bundle_context_bundles_tests.cpp:284 (test_framework+0x522a8)
    #6 testing::internal::TestFactoryImpl<CelixBundleContextBundlesTests_trackBundlesTest_Test>::CreateTest() <null> (test_framework+0x56abb)
    #7 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) <null> (test_framework+0xa5b59)
    #8 main /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/run_tests.cpp:24 (test_framework+0x410d0)

  Thread T12 (tid=64142, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2dc82)
    #1 celixThread_create /home/oipo-unencrypted/Programming/celix/libs/utils/src/celix_threads.c:37 (libcelix_utils.so.2+0xff3d)
    #2 fw_init /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:407 (libcelix_framework.so.2+0x252b4)
    #3 framework_start /home/oipo-unencrypted/Programming/celix/libs/framework/src/framework.c:527 (libcelix_framework.so.2+0x25d9d)
    #4 celix_frameworkFactory_createFramework /home/oipo-unencrypted/Programming/celix/libs/framework/src/celix_framework_factory.c:33 (libcelix_framework.so.2+0x4845d)
    #5 CelixBundleContextBundlesTests::CelixBundleContextBundlesTests() <null> (test_framework+0x51b3e)
    #6 CelixBundleContextBundlesTests_trackBundlesTest_Test::CelixBundleContextBundlesTests_trackBundlesTest_Test() /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/bundle_context_bundles_tests.cpp:284 (test_framework+0x522a8)
    #7 testing::internal::TestFactoryImpl<CelixBundleContextBundlesTests_trackBundlesTest_Test>::CreateTest() <null> (test_framework+0x56abb)
    #8 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) <null> (test_framework+0xa5b59)
    #9 main /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/run_tests.cpp:24 (test_framework+0x410d0)

SUMMARY: ThreadSanitizer: data race /home/oipo-unencrypted/Programming/celix/libs/utils/src/array_list.c:187 in arrayList_add
==================

FATAL: ThreadSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_deadlock_detector.h:67 "((n_all_locks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_contexts_)[0]))))" (0x40, 0x40)
    #0 <null> <null> (libtsan.so.0+0x891b4)
    #1 <null> <null> (libtsan.so.0+0xa74ae)
    #2 <null> <null> (libtsan.so.0+0x94125)
    #3 <null> <null> (libtsan.so.0+0x8790f)
    #4 pthread_rwlock_wrlock <null> (libtsan.so.0+0x2f58e)
    #5 celixThreadRwlock_writeLock /home/oipo-unencrypted/Programming/celix/libs/utils/src/celix_threads.c:200 (libcelix_utils.so.2+0x106ed)
    #6 serviceRegistration_release /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_registration.c:92 (libcelix_framework.so.2+0x369ca)
    #7 serviceReference_release /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_reference.c:84 (libcelix_framework.so.2+0x353f0)
    #8 serviceRegistry_ungetServiceReference /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_registry.c:514 (libcelix_framework.so.2+0x39650)
    #9 celix_serviceRegistry_serviceChanged /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_registry.c:1181 (libcelix_framework.so.2+0x3bee9)
    #10 serviceRegistry_unregisterService /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_registry.c:280 (libcelix_framework.so.2+0x3887d)
    #11 serviceRegistration_unregister /home/oipo-unencrypted/Programming/celix/libs/framework/src/service_registration.c:178 (libcelix_framework.so.2+0x36ea1)
    #12 celix_bundleContext_unregisterService /home/oipo-unencrypted/Programming/celix/libs/framework/src/bundle_context.c:503 (libcelix_framework.so.2+0x2000c)
    #13 CelixBundleContextServicesTests_servicesTrackerSetTest_Test::TestBody() /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/bundle_context_services_test.cpp:704 (test_framework+0x62d7b)
    #14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (test_framework+0xa59a9)
    #15 testing::Test::Run() <null> (test_framework+0x9c069)
    #16 testing::TestInfo::Run() <null> (test_framework+0x9c1b7)
    #17 testing::TestCase::Run() <null> (test_framework+0x9c294)
    #18 testing::internal::UnitTestImpl::RunAllTests() <null> (test_framework+0x9c7ab)
    #19 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) <null> (test_framework+0xa5eb9)
    #20 testing::UnitTest::Run() <null> (test_framework+0x9c8db)
    #21 RUN_ALL_TESTS() /usr/include/gtest/gtest.h:2341 (test_framework+0x411d8)
    #22 main /home/oipo-unencrypted/Programming/celix/libs/framework/gtest/src/run_tests.cpp:24 (test_framework+0x410d0)
    #23 __libc_start_main <null> (libc.so.6+0x271e2)
    #24 _start <null> (test_framework+0x40fdd)

[pnoltes]

This could be solved in #223. Retest when that PR is ready

[Oipo]

Datarace seems to be solved, dead lock is still there.

[Oipo]

The second finding is probably because of the many "destroy of a locked mutex" warnings in celix. See this issue for more info: google/sanitizers#950

[pnoltes]

@Oipo: How can I retest this. I want to check if #223 solved the issue.

[Oipo]

Use -DENABLE_THREAD_SANITIZER=ON with cmake and re-run the test. That should do it.

[PengZheng]

Event-loop based design eliminate most of such issues.