Deadlock in blob_client_wrapper::upload_file_to_blob

[AlexanderYukhanov]

blob_client_wrapper::upload_file_to_blob MUST wait for all threads before exiting.
If it doesn't wait, cv is destroyed first, after that destructor of task_list is called and destructors of std::future are called, ~future will wait for the corresponding thread to finish execution. But this thread will block on calling cv.notify_once (because cv is already destroyed).

So, here is callstack of blob_client_wrapper::upload_file_to_blob:
#0 0x00007fb8cda4c98d in pthread_join (threadid=140430591125248, thread_return=0x0) at pthread_join.c:90
#1 0x00007fb8cd29cb17 in __gthread_join (__value_ptr=0x0, __threadid=)
at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/x86_64-conda_cos6-linux-gnu/bits/gthr-default.h:681
#2 std::thread::join (this=0x7fb8b44a4290) at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/src/gcc-7.2.0/libstdc++-v3/src/c++11/thread.cc:136
#3 0x00000000005fb0ca in std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void>::_Async_state_impl(void) (this=0x7fb8b44a4270, __in_chrg=)
at /usr/include/c++/5/future:1673
#4 0x00000000005fb518 in __gnu_cxx::new_allocator<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> >::destroy<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> >(std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> *) (this=0x7fb8b44a4270, __p=0x7fb8b44a4270)
at /usr/include/c++/5/ext/new_allocator.h:124
#5 0x00000000005fb33d in std::allocator_traits<std::allocator<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> > >::destroy<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> >(std::allocator_traits<std::allocator<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> > >::allocator_type &, std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> *) (__a=..., __p=0x7fb8b44a4270) at /usr/include/c++/5/bits/alloc_traits.h:542
#6 0x00000000005fada5 in std::_Sp_counted_ptr_inplace<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void>, std::allocator<std::__future_base::_Async_state_impl<std::_Bind_simple<microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob(const string&, const string&, std::__cxx11::string, const std::vector<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >&, size_t)::<lambda()>()>, void> >, (__gnu_cxx::_Lock_policy)2u>::_M_dispose(void) (this=0x7fb8b44a4260) at /usr/include/c++/5/bits/shared_ptr_base.h:531
#7 0x0000000000579150 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fb8b44a4260) at /usr/include/c++/5/bits/shared_ptr_base.h:150
#8 0x0000000000578d11 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (this=0x7fb8b458c808, __in_chrg=) at /usr/include/c++/5/bits/shared_ptr_base.h:659
#9 0x00000000005bda12 in std::__shared_ptr<std::__future_base::_State_baseV2, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fb8b458c800, __in_chrg=)
at /usr/include/c++/5/bits/shared_ptr_base.h:925
#10 0x00000000005bda2e in std::shared_ptrstd::__future_base::_State_baseV2::shared_ptr (this=0x7fb8b458c800, __in_chrg=) at /usr/include/c++/5/bits/shared_ptr.h:93
#11 0x00000000005fb694 in std::__basic_future::__basic_future (this=0x7fb8b458c800, __in_chrg=) at /usr/include/c++/5/future:635
#12 0x00000000005fb716 in std::future::~future (this=0x7fb8b458c800, __in_chrg=) at /usr/include/c++/5/future:811
#13 0x0000000000602cab in std::_Destroy<std::future > (__pointer=0x7fb8b458c800) at /usr/include/c++/5/bits/stl_construct.h:93
#14 0x0000000000601841 in std::_Destroy_aux::__destroy<std::future> (__first=0x7fb8b458c800, __last=0x7fb8b458c860) at /usr/include/c++/5/bits/stl_construct.h:103
#15 0x000000000060032e in std::_Destroy<std::future> (__first=0x7fb8b458c780, __last=0x7fb8b458c860) at /usr/include/c++/5/bits/stl_construct.h:126
#16 0x00000000005fed5f in std::_Destroy<std::future*, std::future > (__first=0x7fb8b458c780, __last=0x7fb8b458c860) at /usr/include/c++/5/bits/stl_construct.h:151
#17 0x00000000005fd759 in std::vector<std::future, std::allocator<std::future > >::~vector (this=0x7fb8c8836620, __in_chrg=) at /usr/include/c++/5/bits/stl_vector.h:424
#18 0x00000000005f306b in microsoft_azure::storage::blob_client_wrapper::upload_file_to_blob (this=0x7fb8c40d7410, sourcePath=..., container=..., blob=..., metadata=..., parallel=4)
at /home/ayukhanov/azure-storage-fuse/azure-storage-cpp-lite/src/blob/blob_client_wrapper.cpp:466

and here is the call stack of a worker thread:

#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007fb8cda51a7d in pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:140
#2 0x00007fb8cd298a3b in __gthread_cond_signal (__cond=)
at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/x86_64-conda_cos6-linux-gnu/bits/gthr-default.h:871
#3 std::condition_variable::notify_one (this=) at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/src/gcc-7.2.0/libstdc++-v3/src/c++11/condition_variable.cc:62
#4 0x00000000005f2717 in microsoft_azure::storage::blob_client_wrapper::<lambda()>::operator()(void) const (__closure=0x7fb8b44a42a8)
at /home/ayukhanov/azure-storage-fuse/azure-storage-cpp-lite/src/blob/blob_client_wrapper.cpp:540

[seguler]

This is now fixed in the dev branch with PR #68

[seguler]

@AlexanderYukhanov could you confirm if this is also fixed ?

[AlexanderYukhanov]

yes, fixed