Improve startup time of inmem index #9488
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit improves the startup time when using the `inmem` index by ensuring that the series are created in the index and series file in batches of 10000, rather than individually. Fixes #9486.
ghost
assigned 
|
Not re-using the buffers doesn't make any real difference to the startup time, it just means more garbage. |
jwilder
approved these changes
Feb 27, 2018
e-dard
changed the title
inmem index
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves the startup time when using the
inmemindex byensuring that the series are created in the index and series file in
batches of 10000, rather than individually.
Fixes #9486.
Using the same dataset as #9486 with 4M series:
So that's an improvement of
~25%on1.3.9. However @jwilder @stuartcarnie I'm not set on this PR yet as we may not be able to re-use thekeysbuffer as I'm currently doing. I'm going to dig around a bit and see if it needs to be changed to allocate a new[][]bytefor each batch rather than re-using the same one.