Better bloom filter metrics #3612
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
src/bucket/BucketIndex.cpp
Outdated
| "Underestimated bloom filter size. Estimated entry " | ||
| "count: {}, Actual: {}", | ||
| estimatedNumElems, count); | ||
| CLOG_DEBUG(Bucket, |
There was a problem hiding this comment.
I think the reason we had this at WARNING by default was that
- we expect this to not happen that often
- when it does, I thought you had done research that showed that underestimating could have fairly dramatic (bad) perf impact, and should probably be surfaced
SirTyson
force-pushed
the
bloom-filter-estimation
branch
2 times, most recently
from
ee5ceb0
to
3fe913c
Compare
SirTyson
force-pushed
the
bloom-filter-estimation
branch
from
3fe913c
to
ecae078
Compare
|
r+ ecae078 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Resolves #3604
This PR provides better metrics and log messages for BucketListDB bloom filters. Previously, it appeared that the bloom filter false positive rate was too high due to element count underestimation. However, this was due to faulty metrics. In practice, the false positive rate is significantly better than our target and underestimation is not an issue.
To calculate false positive rates previously, we used the number of ledger entries queried. However, one ledger entry query may lead to several bloom filter queries since we have to check multiple levels of the BucketList for an entry. This PR adds an explicit bloom filter query metric to provide more accurate false positive rates and also logs bloom filter setup parameters.
Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)