Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Coredump when running global_executor_test #1252

Open
dangleptr opened this issue Oct 31, 2019 · 14 comments
Open

Coredump when running global_executor_test #1252

dangleptr opened this issue Oct 31, 2019 · 14 comments

Comments

@dangleptr
Copy link 

[heng@host02-cluster _build]$ ./global_executor_test
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from GlobalExecutorTest
[ RUN      ] GlobalExecutorTest.GlobalCPUExecutor
[       OK ] GlobalExecutorTest.GlobalCPUExecutor (0 ms)
[ RUN      ] GlobalExecutorTest.GlobalIOExecutor
[       OK ] GlobalExecutorTest.GlobalIOExecutor (1 ms)
[----------] 2 tests from GlobalExecutorTest (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (2 ms total)
[  PASSED  ] 2 tests.
*** Aborted at 1572509134 (Unix time, try 'date -d @1572509134') ***
*** Signal 11 (SIGSEGV) (0xe0) received by PID 30858 (pthread TID 0x7f5b86094700) (linux TID 30860) (code: address not mapped to object), stack trace: ***
    @ 000000000086f5af _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
                       /home/heng/workspace/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
    @ 000000000086f654 _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /home/heng/workspace/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
    @ 00007f5b86e685ef (unknown)
    @ 00000000007a02e7 _ZNK5folly14ThreadLocalPtrINS_20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS5_11DefaultMakeIS4_EEvE7WrapperEvvE3getEv
                       /home/heng/workspace/folly/folly/ThreadLocal.h:159
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 000000000079a797 _ZNK5folly11ThreadLocalINS_20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS5_11DefaultMakeIS4_EEvE7WrapperEvvEdeEv
                       /home/heng/workspace/folly/folly/ThreadLocal.h:68
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00000000007980c0 _ZN5folly20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS4_11DefaultMakeIS3_EEvE10getWrapperEv
                       /home/heng/workspace/folly/folly/SingletonThreadLocal.h:148
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00000000007980d4 _ZN5folly20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS4_11DefaultMakeIS3_EEvE13LocalLifetimeD1Ev
                       /home/heng/workspace/folly/folly/SingletonThreadLocal.h:120
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00007f5b8761eeb5 _ZN12_GLOBAL__N_13runEPv
                       /home/heng/workspace/gcc-8.3.0/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
    @ 00007f5b86e60c61 __nptl_deallocate_tsd
    @ 00007f5b86e60e72 start_thread
    @ 00007f5b86b8988c clone
Segmentation fault (core dumped)

version: v2019.10.14.00
@dangleptr
Copy link
Author 

(gdb) bt
#0  0x000000000079cf5b in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:159
#1  0x000000000079740c in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:68
#2  folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::operator* (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:77
#3  0x0000000000794d35 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::getWrapper
    () at /home/heng/workspace/folly/folly/SingletonThreadLocal.h:148
#4  0x0000000000794d49 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::LocalLifetime::~LocalLifetime (this=0x7f4add6def78, __in_chrg=<optimized out>) at /home/heng/workspace/folly/folly/SingletonThreadLocal.h:120
#5  0x00007f4adec70eb6 in (anonymous namespace)::run (p=<optimized out>) at ../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f4ade4b2c62 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0
#7  0x00007f4ade4b2e73 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f4ade1db88d in clone () from /lib64/libc.so.6

@dangleptr
Copy link
Author

It seems that in IOThreadPool, the thread local variable has been released when accessing it.
Any thoughts?

@boringuy
Copy link

@dangleptr What OS are you running? I run into this similar crash with CentOS74 + devtoolset7. So, I tried the same with a Ubuntu 19.10 Docker and don't see the same crash.

@dangleptr
Copy link
Author

centos 7.5 + gcc 8.3.0
@boringuy

@tobz
Copy link

tobz commented Jan 9, 2020 •
edited
Loading

Anecdotally, observing this in mcrouter as well, which depends on folly: also on CentOS 7.5, GCC 5.x.

@boringuy
Copy link

I have not tried all the versions in between but confirmed the unit test cored with v2019.11.11.00 but does not core in v2019.12.30.00 and latest v2020.01.13.00

@tobz
Copy link

tobz commented Jan 17, 2020

Woohoo! I can confirm that v2019.12.30.00 also fixes my issue. I too did not try bisecting, but I can say that v2019.12.02.00 still segfaulted for me.

@dangleptr
Copy link
Author

dangleptr commented Apr 27, 2020 •
edited
Loading

thread_pool_executor_test SegFault on v2020.4.20 with the similar error.

#0  0x00000000009cac61 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::get (this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:160
#1  0x00000000009ca0dc in folly::ThreadLocal<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::get (this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::operator* (
    this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:78
#3  0x00000000009c9e89 in folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::getWrapper ()
    at /home/vesoft/data/chenheng/3rd/folly/folly/SingletonThreadLocal.h:149
#4  0x00000000009c9e9d in folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::LocalLifetime::~LocalLifetime (
    this=0x7f6e8effd508, __in_chrg=<optimized out>) at /home/vesoft/data/chenheng/3rd/folly/folly/SingletonThreadLocal.h:121
#5  0x00007f6ea01773b6 in (anonymous namespace)::run (p=<optimized out>) at ../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f6ea0474c62 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0
#7  0x00007f6ea0474e73 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f6e9f90188d in clone () from /lib64/libc.so.6

ThreadPoolExecutorTest.EDFBasic

@cangfengzhs cangfengzhs mentioned this issue Nov 29, 2021
Closed
apeters1971 pushed a commit to cern-eos/eos that referenced this issue Aug 16, 2022
@esindril
FST: Destroying the FmdConverter object destroys the underlying IOThr…
3c2f4d8 
…eadPoolExecutor

object and the local storage of the threads in this pool will lead to a segv due
to a bug in the folly library. Therefore, we keep the FmdConverter object around until
the FST daemon is shutdown.

Referenced folly bug: facebook/folly#1252

The ASAN trace of such a crash:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==8066==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000038 (pc 0x7ff5370a3e11 bp 0x7ff521df0bf0 sp 0x7ff521df0bb0 T60)
==8066==The signal is caused by a READ memory access.
==8066==Hint: address points to the zero page.
    #0 0x7ff5370a3e10 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly
/folly-2019.11.11.00/folly/ThreadLocal.h:159
    #1 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly/fo
lly-2019.11.11.00/folly/ThreadLocal.h:68
    #2 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::operator*() const /root/rpmbuild/BUILD/eos-fo
lly/folly-2019.11.11.00/folly/ThreadLocal.h:77
    #3 0x7ff53709c0fe in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::getWrapper() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/SingletonThreadL
ocal.h:148
    #4 0x7ff53709c112 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::LocalLifetime::~LocalLifetime() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/fol
ly/SingletonThreadLocal.h:120
    #5 0x7ff547b80b18 in run ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:64
    #6 0x7ff54803aca1 in __nptl_deallocate_tsd /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:155
    #7 0x7ff54803aeb2 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:314
    #8 0x7ff54733c8dc in __clone (/lib64/libc.so.6+0xfe8dc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/ThreadLocal.h:159 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::Reque
stContext> >, void>::Wrapper, void, void>::get() const
Thread T60 created by T40 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e5b1e50 in void AssistedThread::reset<void (eos::fst::ScanDir::*)(ThreadAssistant&) noexcept, eos::fst::ScanDir*>(void (eos::fst::ScanDir::*&&)(ThreadAssistant&) noexcept, eos::fst::ScanDir*&&) (/lib64/libXrdEosFst-4.so+0x41be50)
    #3 0x7ff53e587b13 in eos::fst::ScanDir::SetConfig(std::string const&, long long) ../fst/ScanDir.cc:136
    #4 0x7ff53e72dba2 in eos::fst::FileSystem::ConfigScanner(eos::fst::Load*, std::string const&, long long) ../fst/storage/FileSystem.cc:165
    #5 0x7ff53e710721 in eos::fst::Storage::ProcessFsConfigChange(eos::fst::FileSystem*, std::string const&, std::string const&, std::string const&) ../fst/storage/Communicator.cc:321
    #6 0x7ff53e711b67 in eos::fst::Storage::ProcessFsConfigChange(std::string const&, std::string const&) ../fst/storage/Communicator.cc:400
    #7 0x7ff53e713e13 in eos::fst::Storage::Communicator(ThreadAssistant&) ../fst/storage/Communicator.cc:513
    #8 0x7ff53e76f00f in void std::__invoke_impl<void, void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >(std::__invoke_memfun_deref, void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrap
per<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:73
    #9 0x7ff53e76a4cd in std::__invoke_result<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >::type std::__invoke<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant>
>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrapper<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
    #10 0x7ff53e7786c4 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::_M_invoke<0ul, 1ul, 2ul>(std
::_Index_tuple<0ul, 1ul, 2ul>) /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
    #11 0x7ff53e778663 in std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::operator()() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
    #12 0x7ff53e778647 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > > >::_M_run() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
    #13 0x7ff539339bce in execute_native_thread_routine (/usr/lib64/libEosCommon.so.4+0x8f2bce)

Thread T40 created by T0 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e766d3c in void AssistedThread::reset<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&) ../common/AssistedThread.hh:217
    #3 0x7ff53e758042 in eos::fst::Storage::Storage(char const*) ../fst/storage/Storage.cc:184
    #4 0x7ff53e7569be in eos::fst::Storage::Create(char const*) ../fst/storage/Storage.cc:56
    #5 0x7ff53e5e2f63 in eos::fst::XrdFstOfs::Configure(XrdSysError&, XrdOucEnv*) ../fst/XrdFstOfs.cc:843
    #6 0x7ff53e5dc668 in XrdSfsGetFileSystem2 ../fst/XrdFstOfs.cc:128
    #7 0x7ff54889633a in XrdXrootdloadFileSystem(XrdSysError*, XrdSfsFileSystem*, char*, int, char const*, XrdOucEnv*) /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdLoadLib.cc:71

==8066==ABORTING
apeters1971 pushed a commit to cern-eos/eos that referenced this issue Aug 17, 2022
@esindril
FST: Destroying the FmdConverter object destroys the underlying IOThr…
7622bef 
…eadPoolExecutor

object and the local storage of the threads in this pool will lead to a segv due
to a bug in the folly library. Therefore, we keep the FmdConverter object around until
the FST daemon is shutdown.

Referenced folly bug: facebook/folly#1252

The ASAN trace of such a crash:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==8066==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000038 (pc 0x7ff5370a3e11 bp 0x7ff521df0bf0 sp 0x7ff521df0bb0 T60)
==8066==The signal is caused by a READ memory access.
==8066==Hint: address points to the zero page.
    #0 0x7ff5370a3e10 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly
/folly-2019.11.11.00/folly/ThreadLocal.h:159
    #1 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly/fo
lly-2019.11.11.00/folly/ThreadLocal.h:68
    #2 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::operator*() const /root/rpmbuild/BUILD/eos-fo
lly/folly-2019.11.11.00/folly/ThreadLocal.h:77
    #3 0x7ff53709c0fe in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::getWrapper() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/SingletonThreadL
ocal.h:148
    #4 0x7ff53709c112 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::LocalLifetime::~LocalLifetime() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/fol
ly/SingletonThreadLocal.h:120
    #5 0x7ff547b80b18 in run ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:64
    #6 0x7ff54803aca1 in __nptl_deallocate_tsd /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:155
    #7 0x7ff54803aeb2 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:314
    #8 0x7ff54733c8dc in __clone (/lib64/libc.so.6+0xfe8dc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/ThreadLocal.h:159 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::Reque
stContext> >, void>::Wrapper, void, void>::get() const
Thread T60 created by T40 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e5b1e50 in void AssistedThread::reset<void (eos::fst::ScanDir::*)(ThreadAssistant&) noexcept, eos::fst::ScanDir*>(void (eos::fst::ScanDir::*&&)(ThreadAssistant&) noexcept, eos::fst::ScanDir*&&) (/lib64/libXrdEosFst-4.so+0x41be50)
    #3 0x7ff53e587b13 in eos::fst::ScanDir::SetConfig(std::string const&, long long) ../fst/ScanDir.cc:136
    #4 0x7ff53e72dba2 in eos::fst::FileSystem::ConfigScanner(eos::fst::Load*, std::string const&, long long) ../fst/storage/FileSystem.cc:165
    #5 0x7ff53e710721 in eos::fst::Storage::ProcessFsConfigChange(eos::fst::FileSystem*, std::string const&, std::string const&, std::string const&) ../fst/storage/Communicator.cc:321
    #6 0x7ff53e711b67 in eos::fst::Storage::ProcessFsConfigChange(std::string const&, std::string const&) ../fst/storage/Communicator.cc:400
    #7 0x7ff53e713e13 in eos::fst::Storage::Communicator(ThreadAssistant&) ../fst/storage/Communicator.cc:513
    #8 0x7ff53e76f00f in void std::__invoke_impl<void, void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >(std::__invoke_memfun_deref, void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrap
per<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:73
    #9 0x7ff53e76a4cd in std::__invoke_result<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >::type std::__invoke<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant>
>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrapper<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
    #10 0x7ff53e7786c4 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::_M_invoke<0ul, 1ul, 2ul>(std
::_Index_tuple<0ul, 1ul, 2ul>) /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
    #11 0x7ff53e778663 in std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::operator()() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
    #12 0x7ff53e778647 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > > >::_M_run() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
    #13 0x7ff539339bce in execute_native_thread_routine (/usr/lib64/libEosCommon.so.4+0x8f2bce)

Thread T40 created by T0 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e766d3c in void AssistedThread::reset<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&) ../common/AssistedThread.hh:217
    #3 0x7ff53e758042 in eos::fst::Storage::Storage(char const*) ../fst/storage/Storage.cc:184
    #4 0x7ff53e7569be in eos::fst::Storage::Create(char const*) ../fst/storage/Storage.cc:56
    #5 0x7ff53e5e2f63 in eos::fst::XrdFstOfs::Configure(XrdSysError&, XrdOucEnv*) ../fst/XrdFstOfs.cc:843
    #6 0x7ff53e5dc668 in XrdSfsGetFileSystem2 ../fst/XrdFstOfs.cc:128
    #7 0x7ff54889633a in XrdXrootdloadFileSystem(XrdSysError*, XrdSfsFileSystem*, char*, int, char const*, XrdOucEnv*) /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdLoadLib.cc:71

==8066==ABORTING
apeters1971 pushed a commit to cern-eos/eos that referenced this issue Aug 19, 2022
@esindril
FST: Destroying the FmdConverter object destroys the underlying IOThr…
cbe353d 
…eadPoolExecutor

object and the local storage of the threads in this pool will lead to a segv due
to a bug in the folly library. Therefore, we keep the FmdConverter object around until
the FST daemon is shutdown.

Referenced folly bug: facebook/folly#1252

The ASAN trace of such a crash:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==8066==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000038 (pc 0x7ff5370a3e11 bp 0x7ff521df0bf0 sp 0x7ff521df0bb0 T60)
==8066==The signal is caused by a READ memory access.
==8066==Hint: address points to the zero page.
    #0 0x7ff5370a3e10 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly
/folly-2019.11.11.00/folly/ThreadLocal.h:159
    #1 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get() const /root/rpmbuild/BUILD/eos-folly/fo
lly-2019.11.11.00/folly/ThreadLocal.h:68
    #2 0x7ff53709e9d9 in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::operator*() const /root/rpmbuild/BUILD/eos-fo
lly/folly-2019.11.11.00/folly/ThreadLocal.h:77
    #3 0x7ff53709c0fe in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::getWrapper() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/SingletonThreadL
ocal.h:148
    #4 0x7ff53709c112 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::LocalLifetime::~LocalLifetime() /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/fol
ly/SingletonThreadLocal.h:120
    #5 0x7ff547b80b18 in run ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:64
    #6 0x7ff54803aca1 in __nptl_deallocate_tsd /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:155
    #7 0x7ff54803aeb2 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:314
    #8 0x7ff54733c8dc in __clone (/lib64/libc.so.6+0xfe8dc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /root/rpmbuild/BUILD/eos-folly/folly-2019.11.11.00/folly/ThreadLocal.h:159 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::Reque
stContext> >, void>::Wrapper, void, void>::get() const
Thread T60 created by T40 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e5b1e50 in void AssistedThread::reset<void (eos::fst::ScanDir::*)(ThreadAssistant&) noexcept, eos::fst::ScanDir*>(void (eos::fst::ScanDir::*&&)(ThreadAssistant&) noexcept, eos::fst::ScanDir*&&) (/lib64/libXrdEosFst-4.so+0x41be50)
    #3 0x7ff53e587b13 in eos::fst::ScanDir::SetConfig(std::string const&, long long) ../fst/ScanDir.cc:136
    #4 0x7ff53e72dba2 in eos::fst::FileSystem::ConfigScanner(eos::fst::Load*, std::string const&, long long) ../fst/storage/FileSystem.cc:165
    #5 0x7ff53e710721 in eos::fst::Storage::ProcessFsConfigChange(eos::fst::FileSystem*, std::string const&, std::string const&, std::string const&) ../fst/storage/Communicator.cc:321
    #6 0x7ff53e711b67 in eos::fst::Storage::ProcessFsConfigChange(std::string const&, std::string const&) ../fst/storage/Communicator.cc:400
    #7 0x7ff53e713e13 in eos::fst::Storage::Communicator(ThreadAssistant&) ../fst/storage/Communicator.cc:513
    #8 0x7ff53e76f00f in void std::__invoke_impl<void, void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >(std::__invoke_memfun_deref, void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrap
per<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:73
    #9 0x7ff53e76a4cd in std::__invoke_result<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> >::type std::__invoke<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant>
>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&, std::reference_wrapper<ThreadAssistant>&&) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
    #10 0x7ff53e7786c4 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::_M_invoke<0ul, 1ul, 2ul>(std
::_Index_tuple<0ul, 1ul, 2ul>) /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
    #11 0x7ff53e778663 in std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > >::operator()() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
    #12 0x7ff53e778647 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*, std::reference_wrapper<ThreadAssistant> > > >::_M_run() /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
    #13 0x7ff539339bce in execute_native_thread_routine (/usr/lib64/libEosCommon.so.4+0x8f2bce)

Thread T40 created by T0 here:
    #0 0x7ff548ea7471 in pthread_create (/usr/lib64/libasan.so.5+0x3a471)
    #1 0x7ff539339c34 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib64/libEosCommon.so.4+0x8f2c34)
    #2 0x7ff53e766d3c in void AssistedThread::reset<void (eos::fst::Storage::*)(ThreadAssistant&), eos::fst::Storage*>(void (eos::fst::Storage::*&&)(ThreadAssistant&), eos::fst::Storage*&&) ../common/AssistedThread.hh:217
    #3 0x7ff53e758042 in eos::fst::Storage::Storage(char const*) ../fst/storage/Storage.cc:184
    #4 0x7ff53e7569be in eos::fst::Storage::Create(char const*) ../fst/storage/Storage.cc:56
    #5 0x7ff53e5e2f63 in eos::fst::XrdFstOfs::Configure(XrdSysError&, XrdOucEnv*) ../fst/XrdFstOfs.cc:843
    #6 0x7ff53e5dc668 in XrdSfsGetFileSystem2 ../fst/XrdFstOfs.cc:128
    #7 0x7ff54889633a in XrdXrootdloadFileSystem(XrdSysError*, XrdSfsFileSystem*, char*, int, char const*, XrdOucEnv*) /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdLoadLib.cc:71

==8066==ABORTING
apeters1971 pushed a commit to cern-eos/eos that referenced this issue Oct 3, 2022
@theanalyst
TESTS: Add a global fixture for folly::IOThreadPoolExecutor
160c12d 
Currently with centos7 we're having faults with thread_local when folly
executors are created and destroyed with
facebook/folly#1252 & possibly
facebook/folly#1135, these issues don't seem to be
reproducible in newer gcc's shipped with centos8+. Since in the EOS Code we
create the executors and keep them around until shutdown, use a similar pattern
instead of creating and destroying multiple folly threadpools

Signed-off-by: Abhishek Lekshmanan <[email protected]>
@lrita
Copy link
Contributor

lrita commented Feb 24, 2023 •
edited
Loading

folly-v2023.02.20.00 unit test ./synchronized_test crashed, compiled by devtoolset-8 gcc 8.3 and centos 7.8 and kernel 3.10.0-957.21.3.el7

#0  0x00000000005c3a99 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/detail/ThreadLocalDetail.h:322
#1  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::operator* (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:78
#3  folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::getWrapper(void) () at ../folly/SingletonThreadLocal.h:138
#4  0x00000000005c3af9 in folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::LocalLifetime::~LocalLifetime(void) (this=0x7fffc765f4e0, __in_chrg=<optimized out>)
    at ../folly/SingletonThreadLocal.h:128
#5  0x00000000006cc626 in (anonymous namespace)::run(void*) ()
#6  0x00007ffff74aaca2 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#7  0x00007ffff74aaeb3 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ffff71d39fd in clone () from /usr/lib64/libc.so.6

@lrita
Copy link
Contributor

lrita commented Mar 6, 2023

folly-v2023.02.20.00 unit test ./synchronized_test crashed, compiled by devtoolset-8 gcc 8.3 and centos 7.8 and kernel 3.10.0-957.21.3.el7

#0  0x00000000005c3a99 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/detail/ThreadLocalDetail.h:322
#1  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::operator* (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:78
#3  folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::getWrapper(void) () at ../folly/SingletonThreadLocal.h:138
#4  0x00000000005c3af9 in folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::LocalLifetime::~LocalLifetime(void) (this=0x7fffc765f4e0, __in_chrg=<optimized out>)
    at ../folly/SingletonThreadLocal.h:128
#5  0x00000000006cc626 in (anonymous namespace)::run(void*) ()
#6  0x00007ffff74aaca2 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#7  0x00007ffff74aaeb3 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ffff71d39fd in clone () from /usr/lib64/libc.so.6

CentOS 7.4 using glibc2.17，which has no __cxa_thread_atexit and __cxa_thread_atexit_impl support，then libstdc++ using TSD(pthread_key_create) implements thread_local variables' dtor invoking(which is shown as (anonymous namespace)::run(void*) () in the backtrace).

Unfortunately glibc TSD delete callbacks are not invoked in reverse order in which they were added. So this make ~LocalLifetime is invoked later than StaticMetaBase::onThreadExit.

lrita added a commit to lrita/folly that referenced this issue Mar 6, 2023
@lrita
Invoking thread_local dtor register early to fix folly::ThreadLocal c…
dab84c8 
…rash

issue with glibc < 2.18. Details see facebook#1252
@lrita lrita mentioned this issue Mar 6, 2023
Closed
@ehds ehds mentioned this issue Apr 10, 2023
Closed
facebook-github-bot pushed a commit that referenced this issue May 8, 2023
@lrita @facebook-github-bot
Invoking thread_local dtor register early to fix folly::ThreadLocal c…
da55354 
…rash issue with glibc < 2.18 (#1948)

Summary:
Invoking thread_local dtor register early to fix folly::ThreadLocal crash issue with glibc < 2.18
Details see #1252

Signed-off-by: lrita [email protected]

Pull Request resolved: #1948

Reviewed By: Gownta

Differential Revision: D43881397

Pulled By: Orvid

fbshipit-source-id: 782a1557241b0ddc0a731b5c02fa0b6b8ddb32ec
facebook-github-bot pushed a commit to facebook/hhvm that referenced this issue May 8, 2023
@lrita @facebook-github-bot
Invoking thread_local dtor register early to fix folly::ThreadLocal c…
2469b03 
…rash issue with glibc < 2.18

Summary:
Invoking thread_local dtor register early to fix folly::ThreadLocal crash issue with glibc < 2.18
Details see facebook/folly#1252

Signed-off-by: lrita [email protected]

X-link: facebook/folly#1948

Reviewed By: Gownta

Differential Revision: D43881397

Pulled By: Orvid

fbshipit-source-id: 782a1557241b0ddc0a731b5c02fa0b6b8ddb32ec
@HolyLow
Copy link

HolyLow commented May 27, 2024

Seems this bug still exists. I ran into the same coredump when using the folly-2024.04.01. My environment is AliOS-7 (based on centos I believe) with libc 2.32 :

$ uname -a
Linux 2a24bff1ef3d 4.9.168-019.ali3000.alios7.x86_64 #1 SMP Thu Dec 31 20:20:01 CST 2020 x86_64 x86_64 x86_64 GNU/Linux
$ /lib64/libc.so.6
GNU C Library (GNU libc) release release version 2.32.
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 10.2.1 20200825 (Alibaba 10.2.1-3 2.17).
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

And the stacktrace is

*** Signal 11 (SIGSEGV) (0x7f59dde2a6b9) received by PID 3634169 (pthread TID 0x7f5e2cf06640) (linux TID 3634170) (code: address not mapped to object), stack trace: ***
    @ 000000000058190c folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /code/thirdparty/src/folly-2024.04.01.00/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 00000000000140ff (unknown)
    @ 000000000050b96f folly::SingletonThreadLocal<folly::RequestContext::StaticContext, folly::RequestContext, folly::detail::DefaultMake<folly::RequestContext::StaticContext>, folly::RequestContext>::getWrapper()
                       /code/thirdparty/src/folly-2024.04.01.00/folly/ThreadLocal.h:151
                       -> /code/thirdparty/src/folly-2024.04.01.00/folly/io/async/Request.cpp
    @ 000000000050bac8 folly::SingletonThreadLocal<folly::RequestContext::StaticContext, folly::RequestContext, folly::detail::DefaultMake<folly::RequestContext::StaticContext>, folly::RequestContext>::LocalLifetime::~LocalLifetime()
                       /code/thirdparty/src/folly-2024.04.01.00/folly/SingletonThreadLocal.h:127
                       -> /code/thirdparty/src/folly-2024.04.01.00/folly/io/async/Request.cpp
    @ 00000000000a4a35 (unknown)
    @ 0000000000009210 __nptl_deallocate_tsd
    @ 000000000000940b start_thread
    @ 0000000000100302 clone

@lrita do you have any idea how to fix this? Thanks a lot.

@HolyLow
Copy link

HolyLow commented May 27, 2024

Following the instructions in README.md, I tried to compile and run all the test cases of folly on my dev machine, and it appears that a large amount of testcases failed because of segfault.

The following tests FAILED:
        336 - atomic_shared_ptr_test.AtomicSharedPtr.DeterministicTest (SEGFAULT
)
        341 - cache_locality_test.CacheLocality.LinuxActual (Failed)
        354 - core_cached_shared_ptr_test.CoreCachedSharedPtr.AtomicCoreCachedSh
aredPtr (SEGFAULT)
        403 - dynamic_bounded_queue_test.DynamicBoundedQueue.enqDeq (SEGFAULT)
        420 - unbounded_queue_test.UnboundedQueue.enqDeq (SEGFAULT)
        474 - executor_test.ManualExecutor.getViaDoesNotDeadlock (SEGFAULT)
        484 - fiber_io_executor_test.FiberIOExecutorTest.event_base (SEGFAULT)
        487 - global_executor_test.GlobalExecutorTest.GlobalImmutableIOExecutor
(SEGFAULT)
        489 - global_executor_test.GlobalExecutorTest.GlobalIOExecutor (SEGFAULT
)
        503 - threaded_executor_test.ThreadedExecutorTest.exception (SEGFAULT)
        516 - priority_unbounded_blocking_queue_test.PriorityUnboundedBlockingQu
eueTest.concurrent_push_pop (SEGFAULT)
        522 - unbounded_blocking_queue_test.UnboundedBlockingQueue.concurrentPus
hPop (SEGFAULT)
        581 - lock_free_ring_buffer_test.LockFreeRingBuffer.writesNeverFail (SEG
FAULT)
        775 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsValue (SEGF
AULT)
        776 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsValueThrows
 (SEGFAULT)
        777 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsFuture (SEG
FAULT)
        778 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsFutureThrow
s (SEGFAULT)
        779 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueMatch (SEGFAULT)
        780 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueMatchThrows (SEGFAULT)
        781 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueWrong (SEGFAULT)
        782 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueWrongThrows (SEGFAULT)
        783 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureMatch (SEGFAULT)
        784 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureMatchThrows (SEGFAULT)
        785 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureWrong (SEGFAULT)
        786 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureWrongThrows (SEGFAULT)
        787 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsValue (SEGFAULT)
        788 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsValueThrows (SEGFAULT)
        789 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsFuture (SEGFAULT)
        790 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsFutureThrows (SEGFAULT)
        904 - interrupt_test.Interrupt.futureWithinTimedOut (SEGFAULT)
        905 - interrupt_test.Interrupt.semiFutureWithinTimedOut (SEGFAULT)
        967 - retrying_test.RetryingTest.policyCappedJitteredExponentialBackoff
(SEGFAULT)
        968 - retrying_test.RetryingTest.policyCappedJitteredExponentialBackoffU
nsafe (SEGFAULT)
        971 - retrying_test.RetryingTest.policySleepDefaults (SEGFAULT)
        972 - retrying_test.RetryingTest.largeRetries (SEGFAULT)
        1031 - via_test.ViaFixture.threadHops (SEGFAULT)
        1032 - via_test.ViaFixture.chainVias (SEGFAULT)
        1034 - via_test.ViaFixture.viaAssignment (SEGFAULT)
        1038 - via_test.Via.viaThenGetWasRacy (SEGFAULT)
        1045 - via_test.Via.viaRaces (SEGFAULT)
        1063 - wait_test.Wait.waitWithDuration (SEGFAULT)
        1064 - wait_test.Wait.multipleWait (SEGFAULT)
        1065 - wait_test.Wait.WaitPlusThen (SEGFAULT)
        1066 - wait_test.Wait.cancelAfterWait (SEGFAULT)
        1080 - window_test.Window.parallel (SEGFAULT)
        1081 - window_test.Window.parallelWithError (SEGFAULT)
        1082 - window_test.Window.allParallelWithError (SEGFAULT)
        1283 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPong (SEGFAULT)
        1284 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongNotify (SEG
FAULT)
        1285 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongNotifyMmsg
(SEGFAULT)
        1286 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongRecvTosDisa
bled (SEGFAULT)
        1287 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongRecvTos (SE
GFAULT)
        1288 - AsyncUDPSocketTest.*/ConnectedAsyncSocketIntegrationTest.Connecte
dPingPong/* (SEGFAULT)
        1289 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongPauseResume
Listening (SEGFAULT)
        1292 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestErrToNonExistentServer
(SEGFAULT)
        1293 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestUnsetErrCallback (SEGFA
ULT)
        1294 - AsyncUDPSocketTest.AsyncUDPSocketTest.CloseInErrorCallback (Timeo
ut)
        1295 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestNonExistentServerNoErrC
b (SEGFAULT)
        1300 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestDetachAttach (SEGFAULT)
        1301 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteCmsg (SEGFAULT)
        1302 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteDynamicCmsg (SEGFA
ULT)
        1304 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemCmsg (SEGFAULT)
        1305 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemDynamicCmsg (SEGF
AULT)
        1307 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemNontrivialCmsgs (
SEGFAULT)
        1321 - EventBaseLocalTest.EventBaseLocalTest.DestructorStressTest (SEGFA
ULT)
        1341 - NotificationQueueTest.NotificationQueueTest.ConsumeUntilDrained (
SEGFAULT)
        1342 - NotificationQueueTest.NotificationQueueTest.ConsumeUntilDrainedSt
ress (SEGFAULT)
        1343 - NotificationQueueTest.NotificationQueueTest.SendOneEventFD (SEGFA
ULT)
        1344 - NotificationQueueTest.NotificationQueueTest.PutMessagesEventFD (S
EGFAULT)
        1345 - NotificationQueueTest.NotificationQueueTest.MultiConsumerEventFD
(SEGFAULT)
        1349 - NotificationQueueTest.NotificationQueueTest.SendOnePipe (SEGFAULT
)
        1350 - NotificationQueueTest.NotificationQueueTest.PutMessagesPipe (SEGF
AULT)
        1351 - NotificationQueueTest.NotificationQueueTest.MultiConsumerPipe (SE
GFAULT)
        1355 - NotificationQueueTest.NotificationQueueTest.UseAfterFork (SEGFAUL
T)
        1374 - RequestContextTest.RequestContextTest.AccessAllThreadsDestruction
Guard (SEGFAULT)
        1376 - RequestContextTest.RequestContextTryGetTest.TryGetTest (SEGFAULT)
        1379 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.example (SEGF
AULT)
        1380 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.named_example
 (SEGFAULT)
        1381 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.default_manag
er (SEGFAULT)
        1382 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.custom_manage
r (SEGFAULT)
        1383 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.eb_dtor_in_io
_thread (SEGFAULT)
        1384 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.keepalive (SE
GFAULT)
        1387 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleIov
 (SEGFAULT)
        1388 - writechain_test.WriteChainAsyncTransportWrapperTest.TestChainedIo
v (SEGFAULT)
        1389 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleBuf
 (SEGFAULT)
        1696 - baton_test.Baton.pingpongBlocking (SEGFAULT)
        1697 - baton_test.Baton.pingpongNonblocking (SEGFAULT)
        1700 - baton_test.Baton.timedWaitTimeoutSystemClockBlocking (SEGFAULT)
        1701 - baton_test.Baton.timedWaitTimeoutSystemClockNonblocking (SEGFAULT
)
        1702 - baton_test.Baton.timedWaitSystemClockBlocking (SEGFAULT)
        1703 - baton_test.Baton.timedWaitSystemClockNonblocking (SEGFAULT)
        1706 - baton_test.Baton.timedWaitTimeoutSteadyClockBlocking (SEGFAULT)
        1707 - baton_test.Baton.timedWaitTimeoutSteadyClockNonblocking (SEGFAULT
)
        1708 - baton_test.Baton.timedWaitSteadyClockBlocking (SEGFAULT)
        1709 - baton_test.Baton.timedWaitSteadyClockNonblocking (SEGFAULT)
        1721 - lifo_sem_test.LifoSemTest.pingpong (SEGFAULT)
        1722 - lifo_sem_test.LifoSemTest.mutex (SEGFAULT)
        1726 - lifo_sem_test.LifoSemTest.shutdown_multi (SEGFAULT)
        1822 - small_locks_test.SmallLocks.SpinLockCorrectness (SEGFAULT)
        2008 - deterministic_schedule_test.DeterministicSchedule.buggyAdd (SEGFA
ULT)
        2009 - deterministic_schedule_test.DeterministicSchedule.globalInvariant
s (SEGFAULT)
        2178 - file_util_test.WriteFileAtomic.directoryPermissions (Failed)
        2268 - futex_test.Futex.basicDeterministic (SEGFAULT)
        2301 - locks_test.SpinLock.Correctness (SEGFAULT)
        2346 - memory_idler_test.MemoryIdler.futexWaitValueChangedEarly (Timeout
)
        2347 - memory_idler_test.MemoryIdler.futexWaitValueChangedLate (Timeout)
        2348 - memory_idler_test.MemoryIdler.futexWaitAwokenEarly (Timeout)
        2349 - memory_idler_test.MemoryIdler.futexWaitAwokenLate (Timeout)
        2350 - memory_idler_test.MemoryIdler.futexWaitImmediateFlush (Timeout)
        2351 - memory_idler_test.MemoryIdler.futexWaitNeverFlush (Timeout)
        2443 - random_test.Random.MultiThreaded (SEGFAULT)
        2722 - synchronized_test.SynchronizedTimedTest/*.Timed (SEGFAULT)
        2723 - synchronized_test.SynchronizedTimedWithConstTest/*.TimedShared (S
EGFAULT)

And the error msgs are basicly like "signal 11 (SIGSEGV) (0x7fa7220d8e18) received by PID 3988029 (pthread TID 0x
7fa0e901c640) (linux TID 3988056) (code: address not mapped to object)".

I don't know what's wrong and how to fix this...

@HolyLow
Copy link

HolyLow commented May 27, 2024

I tried to modify the above patch facebook/hhvm@2469b03 to make it turned on for glibc-2.32, and seems many test cases have passed. The failure list shrinks to

The following tests FAILED:
        341 - cache_locality_test.CacheLocality.LinuxActual (Failed)
        1292 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestErrToNonExistentServer (SEGFAULT)
        1293 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestUnsetErrCallback (SEGFAULT)
        1294 - AsyncUDPSocketTest.AsyncUDPSocketTest.CloseInErrorCallback (Timeout)
        1295 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestNonExistentServerNoErrCb (SEGFAULT)
        1300 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestDetachAttach (SEGFAULT)
        1301 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteCmsg (SEGFAULT)
        1302 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteDynamicCmsg (SEGFAULT)
        1304 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemCmsg (SEGFAULT)
        1305 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemDynamicCmsg (SEGFAULT)
        1307 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemNontrivialCmsgs (SEGFAULT)
        1387 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleIov (SEGFAULT)
        1388 - writechain_test.WriteChainAsyncTransportWrapperTest.TestChainedIov (SEGFAULT)
        1389 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleBuf (SEGFAULT)
        2178 - file_util_test.WriteFileAtomic.directoryPermissions (Failed)
        2346 - memory_idler_test.MemoryIdler.futexWaitValueChangedEarly (Timeout)
        2347 - memory_idler_test.MemoryIdler.futexWaitValueChangedLate (Timeout)
        2348 - memory_idler_test.MemoryIdler.futexWaitAwokenEarly (Timeout)
        2349 - memory_idler_test.MemoryIdler.futexWaitAwokenLate (Timeout)
        2350 - memory_idler_test.MemoryIdler.futexWaitImmediateFlush (Timeout)
        2351 - memory_idler_test.MemoryIdler.futexWaitNeverFlush (Timeout)

@HolyLow
Copy link

HolyLow commented May 29, 2024 •
edited
Loading

UPDATE: The latest main branch of folly has enabled the patch for all GLIBC cases, so the problem should be solved. The patch is f141935#diff-be0ee3867901cddc207db866334f0820d4d77e333c583471f55ba517866c84cc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@tobz @boringuy @lrita @HolyLow @dangleptr