Fix race conditions in GenericRateLimiter #10374
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ajkr
changed the title
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ajkr
commented
Jul 16, 2022
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ajkr
added a commit
to ajkr/rocksdb
that referenced
this pull request
Jul 16, 2022
`GetOptionsFromString()` is included in `set_options_one_in` because its current behavior actually involves invoking `PrepareOptions()` on existing shared option objects that may be in-use by a DB. While this has known problems like facebook#10374, it is actually used in production and users are finding these problems before us. To prevent accidental coasting we should turn on the feature in stress/crash test to ensure we discover these problems before our users do. Test Plan: - Build with TSAN: `$ COMPILE_WITH_TSAN=1 make -j24 db_stress` - Repro some `GenericRateLimiter` race conditions: ``` $ ./db_stress -rate_limiter_bytes_per_sec=1048576 --set_options_one_in=100 --max_key=1000000 ... WARNING: ThreadSanitizer: data race (pid=3077932) [14/168] Read of size 4 at 0x7b5800000378 by thread T19: #0 rocksdb::SerializeSingleOptionHelper(void const*, rocksdb::OptionType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) options/options_helper.cc:480 (db_stress+0x9a858e) #1 rocksdb::OptionTypeInfo::Serialize(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const options/options_helper.cc:1089 (db_stress+0x9a8f94) #2 rocksdb::ConfigurableHelper::SerializeOptions(rocksdb::ConfigOptions const&, rocksdb::Configurable const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) options/configurable.cc:575 (db_stress+0x977c13) facebook#3 rocksdb::Configurable::SerializeOptions(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const options/configurable.cc:519 (db_stress+0x978aff) facebook#4 rocksdb::Customizable::SerializeOptions(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const options/customizable.cc:56 (db_stress+0x98066f) facebook#5 rocksdb::Configurable::ToString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const options/configurable.cc:507 (db_stress+0x9767c4) facebook#6 rocksdb::Configurable::ToString[abi:cxx11](rocksdb::ConfigOptions const&) const include/rocksdb/configurable.h:181 (db_stress+0x9a94cb) facebook#7 rocksdb::OptionTypeInfo::Serialize(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const options/options_helper.cc:1070 (db_stress+0x9a94cb) facebook#8 rocksdb::ConfigurableHelper::SerializeOptions(rocksdb::ConfigOptions const&, rocksdb::Configurable const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) options/configurable.cc:575 (db_stress+0x977c13) facebook#9 rocksdb::Configurable::GetOptionString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const options/configurable.cc:497 (db_stress+0x978a1c) facebook#10 rocksdb::GetStringFromDBOptions(rocksdb::ConfigOptions const&, rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) options/options_helper.cc:631 (db_stress+0x9a1833) facebook#11 rocksdb::PersistRocksDBOptions(rocksdb::ConfigOptions const&, rocksdb::DBOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<rocksdb::ColumnFamilyOptions, std::allocator<rocksdb::ColumnFamilyOptions> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > co nst&, rocksdb::FileSystem*) options/options_parser.cc:99 (db_stress+0x9cbc20) facebook#12 rocksdb::PersistRocksDBOptions(rocksdb::DBOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<rocksdb::ColumnFamilyOptions, std::allocator<rocksdb::ColumnFamilyOptions> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::FileSystem*) opt ions/options_parser.cc:53 (db_stress+0x9ce616) facebook#13 rocksdb::DBImpl::WriteOptionsFile(bool, bool) db/db_impl/db_impl.cc:4506 (db_stress+0x5c98e1) facebook#14 rocksdb::DBImpl::SetOptions(rocksdb::ColumnFamilyHandle*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11:: basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) db/db_impl/db_impl.cc:1152 (db_stress+0x5ca8d8) facebook#15 rocksdb::StressTest::SetOptions(rocksdb::ThreadState*) db_stress_tool/db_stress_test_base.cc:577 (db_stress+0x52769c) facebook#16 rocksdb::StressTest::OperateDb(rocksdb::ThreadState*) db_stress_tool/db_stress_test_base.cc:714 (db_stress+0x534a7e) facebook#17 rocksdb::ThreadBody(void*) db_stress_tool/db_stress_driver.cc:33 (db_stress+0x500b02) facebook#18 StartThreadWrapper env/env_posix.cc:461 (db_stress+0x8a0d3f) Previous write of size 4 at 0x7b5800000378 by thread T37: #0 rocksdb::GenericRateLimiter::Initialize() util/rate_limiter.cc:106 (db_stress+0xb028a0) #1 rocksdb::GenericRateLimiter::PrepareOptions(rocksdb::ConfigOptions const&) util/rate_limiter.cc:146 (db_stress+0xb02bb6) #2 rocksdb::OptionTypeInfo::Prepare(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*) const options/options_helper.cc:1408 (db_stress+0x9a0756) facebook#3 rocksdb::Configurable::PrepareOptions(rocksdb::ConfigOptions const&) options/configurable.cc:50 (db_stress+0x9757e6) facebook#4 rocksdb::DBOptionsConfigurable::ConfigureOptions(rocksdb::ConfigOptions const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std ::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to< std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*) options/db_options.cc:645 (db_stress+0x98d178) facebook#5 rocksdb::Configurable::ConfigureFromMap(rocksdb::ConfigOptions const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<st d::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cx x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*) options/configurable.cc:143 (db_stress+0x9766f7) facebook#6 rocksdb::GetOptionsFromString(rocksdb::ConfigOptions const&, rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options*) options/options_helper.cc:799 (db_stress+0x9b19cf) facebook#7 rocksdb::GetOptionsFromString(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options*) options/options_helper.cc:782 (db_stress+0x9b2611) facebook#8 rocksdb::StressTest::SetOptions(rocksdb::ThreadState*) db_stress_tool/db_stress_test_base.cc:556 (db_stress+0x527802) facebook#9 rocksdb::StressTest::OperateDb(rocksdb::ThreadState*) db_stress_tool/db_stress_test_base.cc:714 (db_stress+0x534a7e) facebook#10 rocksdb::ThreadBody(void*) db_stress_tool/db_stress_driver.cc:33 (db_stress+0x500b02) facebook#11 StartThreadWrapper env/env_posix.cc:461 (db_stress+0x8a0d3f) ```
This was referenced Jul 16, 2022
ajkr
added a commit
to ajkr/rocksdb
that referenced
this pull request
Jul 18, 2022
Summary: (PR created for informational/testing purposes only.) - Fixes lost dynamic updates to GenericRateLimiter bandwidth using `SetBytesPerSecond()` - Benefit over facebook#10374 is eliminating race conditions with Configurable framework. Pull Request resolved: facebook#10378 Differential Revision: D37914865 fbshipit-source-id: e2b455eb30058f9eea5fb2af870eaa7c77da6969
pdillinger
reviewed
Jul 18, 2022
util/rate_limiter_test.cc
Outdated
| request_mutex->Lock(); | ||
| }); | ||
|
|
||
| Arg arg(target, Env::IO_HIGH, limiter); | ||
| // TODO(lightmark): more test cases are welcome. | ||
| env->StartThread(writer, &arg); |
There was a problem hiding this comment.
Weird to immediately create and join a single thread
facebook-github-bot
pushed a commit
that referenced
this pull request
Jul 18, 2022
Summary: (PR created for informational/testing purposes only.) - Fixes lost dynamic updates to GenericRateLimiter bandwidth using `SetBytesPerSecond()` - Benefit over #10374 is eliminating race conditions with Configurable framework. Pull Request resolved: #10378 Reviewed By: pdillinger Differential Revision: D37914865 fbshipit-source-id: d4f566d60ec9726d26932388c61671adf0ee0f30
ajkr
force-pushed
the
fix-rate-limiter-bandwidth-revert
branch
from
9feca58 to
c645a74
Compare
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
ajkr
changed the title
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
siying
pushed a commit
that referenced
this pull request
Jul 18, 2022
Summary: (PR created for informational/testing purposes only.) - Fixes lost dynamic updates to GenericRateLimiter bandwidth using `SetBytesPerSecond()` - Benefit over #10374 is eliminating race conditions with Configurable framework. Pull Request resolved: #10378 Reviewed By: pdillinger Differential Revision: D37914865 fbshipit-source-id: d4f566d60ec9726d26932388c61671adf0ee0f30
|
@ajkr has updated the pull request. You must reimport the pull request before landing. |
|
@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ajkr
added a commit
that referenced
this pull request
Jul 19, 2022
Summary: (PR created for informational/testing purposes only.) - Fixes lost dynamic updates to GenericRateLimiter bandwidth using `SetBytesPerSecond()` - Benefit over #10374 is eliminating race conditions with Configurable framework. Pull Request resolved: #10378 Reviewed By: pdillinger Differential Revision: D37914865 fbshipit-source-id: d4f566d60ec9726d26932388c61671adf0ee0f30
|
Thanks for the review! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Made locking strict for all accesses of
GenericRateLimiterinternal state.SetBytesPerSecond()was the main problem since it had no locking, while the two updates it makes need to be done as one atomic operation.The test case, "ConfigOptionsTest.ConfiguringOptionsDoesNotRevertRateLimiterBandwidth", is for the issue fixed in #10378, but I forgot to include the test there.