-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance of some of the Configurable methods #9756
base: main
Are you sure you want to change the base?
Conversation
Change allows the serialization of only values that have changed (rather than all values). Only applies if the ConfigOptions::only_changed_options=true and if there is a default for the specific value.
These methods use a default Option. Because of that, they do not need to do as much work for parse/restore.
@mrambacher has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@mrambacher has updated the pull request. You must reimport the pull request before landing. |
./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=0 -stats_dump_period_sec=60 ./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=1 -stats_dump_period_sec=60
|
This change was motivated to speed up the addition/removal of ColumnFamilies.
The change consists of the following main components:
With these changes, the time to add/drop 200 ColumnFamilies dropped from 93s to 75s. 10000 calls to the new GetDBOptionsFromMap/GetColumnFamilyOptionsFromMap functions take a 1700 and 2800 ms, respectively vs 2300/3900 ms for the old functions and 3000/6400 in the original code.
Note that this PR does NOT change the options file to only print out the changed options (those that do not match the default). Doing so would potentially speed up the options file parsing even further but could lead to semantic changes in the future. A quick test shows the time for Create/DropColumnFamilies dropping to 69s.
There are other performance improvements possible. For example, not writing an Options file if options were not changed via SetDBOptions could also potentially save some time. The StringToMap function is also fairly expensive and could potentially be optimized.