Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of some of the Configurable methods #9756

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

mrambacher
Copy link
Contributor

This change was motivated to speed up the addition/removal of ColumnFamilies.

The change consists of the following main components:

  1. Some of the internal ConfigurableHelper methods were changed to take/return a vector of string pairs, rather than a map. Vectors are faster for traversal and removal.
  2. The ability to add default values was added to the Configurable and OptionTypeInfo registration. An option to not print unchanged values was added to the ConfigOptions class. By not writing the default values, we spent less time in StringToMap and in configuring options that were not changed.
  3. New GetXXOptionsFromYY were added that take no default DB/ColumnFamilyOption. This signature allows the code to know that it does not need to worry about possibly restoring the options to their initial state on failure conditions. An option was added to ConfigOptions for this instruction as well.
  4. If paranoid_checks is off, when an Options file is written, it is not read back in for validation purposes.

With these changes, the time to add/drop 200 ColumnFamilies dropped from 93s to 75s. 10000 calls to the new GetDBOptionsFromMap/GetColumnFamilyOptionsFromMap functions take a 1700 and 2800 ms, respectively vs 2300/3900 ms for the old functions and 3000/6400 in the original code.

Note that this PR does NOT change the options file to only print out the changed options (those that do not match the default). Doing so would potentially speed up the options file parsing even further but could lead to semantic changes in the future. A quick test shows the time for Create/DropColumnFamilies dropping to 69s.

There are other performance improvements possible. For example, not writing an Options file if options were not changed via SetDBOptions could also potentially save some time. The StringToMap function is also fairly expensive and could potentially be optimized.

Change allows the serialization of only values that have changed (rather than all values).  Only applies if the ConfigOptions::only_changed_options=true and if there is a default for the specific value.
These methods use a default Option.  Because of that, they do not need to do as much work for parse/restore.
@facebook-github-bot
Copy link
Contributor

@mrambacher has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@mrambacher has updated the pull request. You must reimport the pull request before landing.

@mrambacher
Copy link
Contributor Author


./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=0 -stats_dump_period_sec=60
DB path: [/tmp/bench]
readrandom : 95.518 micros/op 10469 ops/sec 300.022 seconds 3140999 operations; 1.7 MB/s (1986514 of 3140999 found)


./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=1 -stats_dump_period_sec=60
DB path: [/tmp/bench]
readrandom : 87.854 micros/op 11382 ops/sec 300.023 seconds 3414999 operations; 1.9 MB/s (2160348 of 3414999 found)

./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=1 -stats_dump_period_sec=60 --pinning_policy="id=speedb_scoped_pinning_policy;capacity=3221225472"
9.23BG memory usage
DB path: [/tmp/bench]
readrandom : 87.218 micros/op 11465 ops/sec 300.030 seconds 3439999 operations; 1.9 MB/s (2174623 of 3439999 found)

./db_bench --benchmarks=readrandom --threads=1 -report_interval_seconds=1 --num=100000000 --value_size=256 --max_background_jobs=12 --compression_type=NONE --duration=300 --db=/tmp/bench --use_existing_db -disable_auto_compactions --cache_size=$((1024 * 1024 * 1024 * 8)) --disable_auto_compactions --duration=300 --statistics --report_interval_seconds=1 --cache_index_and_filter_blocks=1 -stats_dump_period_sec=60 --pinning_policy="id=speedb_scoped_pinning_policy;capacity=6442450944"
9.23BG memory usage
DB path: [/tmp/bench]
readrandom : 99.365 micros/op 10063 ops/sec 300.083 seconds 3019999 operations; 1.6 MB/s (1908000 of 3019999 found)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants