Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

column_family_test: EnvCounter::num_new_writable_file_ to be atomic #2331

Closed
wants to merge 1 commit into from
Closed

column_family_test: EnvCounter::num_new_writable_file_ to be atomic #2331

wants to merge 1 commit into from

Conversation

siying
Copy link
Contributor

@siying siying commented May 18, 2017

Summary: TSAN shows warning of data race of EnvCounter::num_new_writable_file_. Make it atomic.

Test Plan: Run all tests.

@siying
column_family_test: EnvCounter::num_new_writable_file_ to be atomic
66010b4 
Summary: TSAN shows warning of data race of EnvCounter::num_new_writable_file_. Make it atomic.

Test Plan: Run all tests.
@siying siying requested a review from sagar0 May 18, 2017 19:34
@facebook-github-bot
Copy link
Contributor

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@siying
Copy link
Contributor Author

siying commented May 18, 2017

The TSAN warning:

[ RUN ] ColumnFamilyTest.SameCFManualAutomaticConflict

WARNING: ThreadSanitizer: data race (pid=47784)
Read of size 4 at 0x7d0800076398 by thread T18:
#0 rocksdb::EnvCounter::NewWritableFile(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<rocksdb::WritableFile, std::default_deleterocksdb::WritableFile >, rocksdb::EnvOptions const&) db/column_family_test.cc:53 (column_family_test+0x000000576156)
#1 rocksdb::NewWritableFile(rocksdb::Env, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<rocksdb::WritableFile, std::default_deleterocksdb::WritableFile >, rocksdb::EnvOptions const&) util/file_reader_writer.cc:581 (column_family_test+0x0000009149b1)
#2 rocksdb::CompactionJob::OpenCompactionOutputFile(rocksdb::CompactionJob::SubcompactionState) db/compaction_job.cc:1223 (column_family_test+0x00000062a345)
#3 rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) db/compaction_job.cc:800 (column_family_test+0x000000630bd5)
#4 rocksdb::CompactionJob::Run() db/compaction_job.cc:537 (column_family_test+0x0000006324d4)
#5 rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*) db/db_impl_compaction_flush.cc:1575 (column_family_test+0x000000697a52)
#6 rocksdb::DBImpl::BackgroundCallCompaction(void*) db/db_impl_compaction_flush.cc:1267 (column_family_test+0x00000069b0e8)
#7 rocksdb::DBImpl::BGWorkCompaction(void*) db/db_impl_compaction_flush.cc:1106 (column_family_test+0x00000069b8e3)
#8 operator() util/threadpool_imp.cc:416 (column_family_test+0x000000930e3d)
#9 _M_invoke /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1871 (column_family_test+0x000000930e3d)
#10 std::function<void ()>::operator()() const /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:2267 (column_family_test+0x00000057f093)
#11 rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) util/threadpool_imp.cc:238 (column_family_test+0x0000009322e1)
#12 rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) util/threadpool_imp.cc:262 (column_family_test+0x00000093261d)
#13 void* std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1531 (column_family_test+0x00000093428f)
#14 std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)>::operator()() /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1520 (column_family_test+0x00000093428f)
#15 std::thread::_Impl<std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)> >::_M_run() /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/thread:115 (column_family_test+0x00000093428f)
#16 execute_native_thread_routine (libstdc++.so.6+0x0000000c4fe0)

Previous write of size 4 at 0x7d0800076398 by thread T19:
#0 rocksdb::EnvCounter::NewWritableFile(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<rocksdb::WritableFile, std::default_deleterocksdb::WritableFile >, rocksdb::EnvOptions const&) db/column_family_test.cc:53 (column_family_test+0x000000576165)
#1 rocksdb::NewWritableFile(rocksdb::Env, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<rocksdb::WritableFile, std::default_deleterocksdb::WritableFile >, rocksdb::EnvOptions const&) util/file_reader_writer.cc:581 (column_family_test+0x0000009149b1)
#2 rocksdb::CompactionJob::OpenCompactionOutputFile(rocksdb::CompactionJob::SubcompactionState) db/compaction_job.cc:1223 (column_family_test+0x00000062a345)
#3 rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) db/compaction_job.cc:800 (column_family_test+0x000000630bd5)
#4 rocksdb::CompactionJob::Run() db/compaction_job.cc:537 (column_family_test+0x0000006324d4)
#5 rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*) db/db_impl_compaction_flush.cc:1575 (column_family_test+0x000000697a52)
#6 rocksdb::DBImpl::BackgroundCallCompaction(void*) db/db_impl_compaction_flush.cc:1267 (column_family_test+0x00000069b0e8)
#7 rocksdb::DBImpl::BGWorkCompaction(void*) db/db_impl_compaction_flush.cc:1106 (column_family_test+0x00000069b8e3)
#8 operator() util/threadpool_imp.cc:416 (column_family_test+0x000000930e3d)
#9 _M_invoke /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1871 (column_family_test+0x000000930e3d)
#10 std::function<void ()>::operator()() const /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:2267 (column_family_test+0x00000057f093)
#11 rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) util/threadpool_imp.cc:238 (column_family_test+0x0000009322e1)
#12 rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) util/threadpool_imp.cc:262 (column_family_test+0x00000093261d)
#13 void* std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1531 (column_family_test+0x00000093428f)
#14 std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)>::operator()() /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/functional:1520 (column_family_test+0x00000093428f)
#15 std::thread::_Impl<std::_Bind_simple<void* ((rocksdb::BGThreadMetadata))(void*)> >::_M_run() /mnt/gvfs/third-party2/gcc/2928bb3ed95bf64f5b388ee88c30dc74710c3b35/5.x/centos6-native/f4950a1/include/c++/5.4.1/thread:115 (column_family_test+0x00000093428f)
#16 execute_native_thread_routine (libstdc++.so.6+0x0000000c4fe0)

Location is heap block of size 32 at 0x7d0800076380 allocated by main thread:
#0 operator new(unsigned long) (libtsan.so.0+0x00000004097d)
#1 rocksdb::ColumnFamilyTest::ColumnFamilyTest() db/column_family_test.cc:64 (column_family_test+0x00000058c372)
#2 rocksdb::ColumnFamilyTest_SameCFManualAutomaticConflict_Test::ColumnFamilyTest_SameCFManualAutomaticConflict_Test() db/column_family_test.cc:1847 (column_family_test+0x00000058e6f4)
#3 testing::internal::TestFactoryImplrocksdb::ColumnFamilyTest_SameCFManualAutomaticConflict_Test::CreateTest() third-party/gtest-1.7.0/fused-src/gtest/gtest.h:7991 (column_family_test+0x00000058e6f4)
#4 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c42d)
#5 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c42d)
#6 testing::TestInfo::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4065 (column_family_test+0x000000b20822)
#7 testing::TestCase::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4190 (column_family_test+0x000000b20a31)
#8 testing::internal::UnitTestImpl::RunAllTests() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:6062 (column_family_test+0x000000b20f04)
#9 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c86d)
#10 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c86d)
#11 testing::UnitTest::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:5680 (column_family_test+0x000000b21345)
#12 RUN_ALL_TESTS() third-party/gtest-1.7.0/fused-src/gtest/gtest.h:20722 (column_family_test+0x000000410740)
#13 main db/column_family_test.cc:3215 (column_family_test+0x000000410740)

Thread T18 'rocksdb:bg1' (tid=49212, running) created by main thread at:
#0 pthread_create (libtsan.so.0+0x000000027dfa)
#1 std::thread::_M_start_thread(std::shared_ptrstd::thread::_Impl_base, void ()()) (libstdc++.so.6+0x0000000c5129)
#2 rocksdb::ThreadPoolImpl::Impl::SetBackgroundThreadsInternal(int, bool) util/threadpool_imp.cc:280 (column_family_test+0x000000932bb2)
#3 rocksdb::ThreadPoolImpl::IncBackgroundThreadsIfNeeded(int) util/threadpool_imp.cc:400 (column_family_test+0x000000932d2c)
#4 IncBackgroundThreadsIfNeeded env/env_posix.cc:769 (column_family_test+0x0000007efd63)
#5 rocksdb::EnvWrapper::IncBackgroundThreadsIfNeeded(int, rocksdb::Env::Priority) include/rocksdb/env.h:1022 (column_family_test+0x0000006b0831)
#6 rocksdb::SanitizeOptions(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::DBOptions const&) db/db_impl_open.cc:65 (column_family_test+0x0000006b0831)
#7 rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) db/db_impl.cc:190 (column_family_test+0x00000067243a)
#8 rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB**) db/db_impl_open.cc:917 (column_family_test+0x0000006ba494)
#9 rocksdb::ColumnFamilyTest::TryOpen(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:171 (column_family_test+0x000000590f62)
#10 rocksdb::ColumnFamilyTest::Open(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:197 (column_family_test+0x0000005917db)
#11 rocksdb::ColumnFamilyTest::Reopen(std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:277 (column_family_test+0x000000592fdd)
#12 rocksdb::ColumnFamilyTest_MultipleManualCompactions_Test::TestBody() db/column_family_test.cc:1308 (column_family_test+0x00000053b6e5)
#13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c20d)
#14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c20d)
#15 testing::Test::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3897 (column_family_test+0x000000b20693)
#16 testing::TestInfo::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4072 (column_family_test+0x000000b208f7)
#17 testing::TestCase::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4190 (column_family_test+0x000000b20a31)
#18 testing::internal::UnitTestImpl::RunAllTests() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:6062 (column_family_test+0x000000b20f04)
#19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c86d)
#20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c86d)
#21 testing::UnitTest::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:5680 (column_family_test+0x000000b21345)
#22 RUN_ALL_TESTS() third-party/gtest-1.7.0/fused-src/gtest/gtest.h:20722 (column_family_test+0x000000410740)
#23 main db/column_family_test.cc:3215 (column_family_test+0x000000410740)

Thread T19 'rocksdb:bg2' (tid=49213, running) created by main thread at:
#0 pthread_create (libtsan.so.0+0x000000027dfa)
#1 std::thread::_M_start_thread(std::shared_ptrstd::thread::_Impl_base, void ()()) (libstdc++.so.6+0x0000000c5129)
#2 rocksdb::ThreadPoolImpl::Impl::SetBackgroundThreadsInternal(int, bool) util/threadpool_imp.cc:280 (column_family_test+0x000000932bb2)
#3 rocksdb::ThreadPoolImpl::IncBackgroundThreadsIfNeeded(int) util/threadpool_imp.cc:400 (column_family_test+0x000000932d2c)
#4 IncBackgroundThreadsIfNeeded env/env_posix.cc:769 (column_family_test+0x0000007efd63)
#5 rocksdb::EnvWrapper::IncBackgroundThreadsIfNeeded(int, rocksdb::Env::Priority) include/rocksdb/env.h:1022 (column_family_test+0x0000006b0831)
#6 rocksdb::SanitizeOptions(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::DBOptions const&) db/db_impl_open.cc:65 (column_family_test+0x0000006b0831)
#7 rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) db/db_impl.cc:190 (column_family_test+0x00000067243a)
#8 rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB**) db/db_impl_open.cc:917 (column_family_test+0x0000006ba494)
#9 rocksdb::ColumnFamilyTest::TryOpen(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:171 (column_family_test+0x000000590f62)
#10 rocksdb::ColumnFamilyTest::Open(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:197 (column_family_test+0x0000005917db)
#11 rocksdb::ColumnFamilyTest::Reopen(std::vector<rocksdb::ColumnFamilyOptions, std::allocatorrocksdb::ColumnFamilyOptions >) db/column_family_test.cc:277 (column_family_test+0x000000592fdd)
#12 rocksdb::ColumnFamilyTest_MultipleManualCompactions_Test::TestBody() db/column_family_test.cc:1308 (column_family_test+0x00000053b6e5)
#13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c20d)
#14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c20d)
#15 testing::Test::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3897 (column_family_test+0x000000b20693)
#16 testing::TestInfo::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4072 (column_family_test+0x000000b208f7)
#17 testing::TestCase::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:4190 (column_family_test+0x000000b20a31)
#18 testing::internal::UnitTestImpl::RunAllTests() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:6062 (column_family_test+0x000000b20f04)
#19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3824 (column_family_test+0x000000b2c86d)
#20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3860 (column_family_test+0x000000b2c86d)
#21 testing::UnitTest::Run() third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:5680 (column_family_test+0x000000b21345)
#22 RUN_ALL_TESTS() third-party/gtest-1.7.0/fused-src/gtest/gtest.h:20722 (column_family_test+0x000000410740)
#23 main db/column_family_test.cc:3215 (column_family_test+0x000000410740)

SUMMARY: ThreadSanitizer: data race db/column_family_test.cc:53 rocksdb::EnvCounter::NewWritableFile(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<rocksdb::WritableFile, std::default_deleterocksdb::WritableFile >*, rocksdb::EnvOptions const&)

yiwu-arbug
yiwu-arbug approved these changes May 18, 2017
View reviewed changes
Copy link
Contributor

@yiwu-arbug yiwu-arbug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if build passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiwu-arbug yiwu-arbug yiwu-arbug approved these changes

@sagar0 sagar0 Awaiting requested review from sagar0

Assignees
No one assigned
Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@siying @facebook-github-bot @yiwu-arbug