Optimisations for LoadNextMsgMulti
#6448
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neilalexander
force-pushed
the
neil/firstmatchingmulti
branch
2 times, most recently
from
46d73c5 to
2a93e6c
Compare
This avoids a potentially expensive linear walk if it is obvious that the block FSS contains less subjects than sequences that we would otherwise have to scan. Signed-off-by: Neil Twigg <[email protected]>
…alls Signed-off-by: Neil Twigg <[email protected]>
neilalexander
force-pushed
the
neil/firstmatchingmulti
branch
from
2a93e6c to
5499017
Compare
derekcollison
added a commit
that referenced
this pull request
Feb 4, 2025
This updates `generatePerSubjectInfo`, `NumPending` and `NumPendingMulti` to avoid updating the last load timestamp in a tight-loop while skipping over a potentially large number of interior deletes, as this becomes noticeable in the CPU profile. Similar to one of the changes in #6448. Signed-off-by: Neil Twigg <[email protected]>
neilalexander
added a commit
that referenced
this pull request
Feb 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR optimises
LoadNextMsgMultiby doing two things:firstMatchingMultiby using subject tree intersection when it looks like the FSS size is smaller than the scan range, this should be less expensive (although the threshold for choosing strategy may need further attention);dmapbefore callingmb.cacheLookup()in either the FSS search or the linear scan, such that we can avoid callingtime.Now()a lot when we are tight-looped scanning over a large number of interior deletes.This PR provides a 2x improvement on the 1000-filter
LoadNextMsgMultiunit test and provides a 10x improvement when tested against a store that has extremely sparse messages and 8 consumer filters.Signed-off-by: Neil Twigg [email protected]