release-19.1: storage: RocksDB visibility improvements #36506
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Set up a debug page at "/_status/enginestats/<node_id>" that contains values of all RocksDB ticker stats and histograms. It is also included in debug zip. A portion of the debug page (including both tickers and histograms) looks like this:  The debug zip file contains a JSON file named "debug/nodes/<node_id>/enginestats" with all the tickers/histograms, and looks like this: ``` { "stats": [ { "storeId": 1, "tickersAndHistograms": { "tickers": { "rocksdb.blobdb.blob.file.bytes.read": "0", ... "rocksdb.write.wal": "296" }, "histograms": { "rocksdb.blobdb.blob.file.read.micros": { "mean": 0, "p50": 0, "p95": 0, "p99": 0, "max": 0, "count": "0", "sum": "0" }, ... "rocksdb.write.raw.block.micros": { "mean": 1.8235294117647058, "p50": 0.6071428571428571, "p95": 10.749999999999993, "p99": 12, "max": 12, "count": "17", "sum": "31" } } } } ] } ``` Release note: None
Previously we enabled RocksDB logging only when glog verbosity was set to three. Additionally, the RocksDB logs were printed at Cockroach's info severity, regardless of the severity reported by RocksDB. With this PR, RocksDB warnings, errors, and fatals are logged by default. Info messages still require verbosity set to three to be sent to Cockroach's log. Also, we try to set the severity in the Cockroach log to match the severity reported by RocksDB. Test Plan: set stall limits low and watch the log messages flow: ``` $ ./cockroach start --logtostderr --insecure ... W190402 04:40:00.579900 138 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:799] [default] Stalling writes because we have 3 level-0 files rate 3044057 W190402 04:40:00.666972 17 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:754] [default] Stopping writes because we have 4 level-0 files W190402 04:40:00.985046 482 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:799] [default] Stalling writes because we have 3 level-0 files rate 3044057 W190402 04:40:00.986236 138 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:799] [default] Stalling writes because we have 3 level-0 files rate 1826434 W190402 04:40:01.054917 482 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:754] [default] Stopping writes because we have 4 level-0 files W190402 04:40:01.422071 17 storage/engine/rocksdb.go:97 [rocksdb] [/home/andrew/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/column_family.cc:799] [default] Stalling writes because we have 3 level-0 files rate 1826434 ... ``` Release note: None
|
This change is |
|
LGTM |
|
bors r=tbg |
Merge conflict |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport:
Please see individual PRs for details.
/cc @cockroachdb/release