analyzer/block: configurable batch size #417
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mitjat
approved these changes
May 16, 2023
analyzer/block/block.go
Outdated
| // Number of blocks to be processed in a batch. | ||
| blocksBatchSize = 100 | ||
| // Default number of blocks to be processed in a batch. | ||
| defaultBatchSize = 100 |
There was a problem hiding this comment.
Should we change this to 1000 or even 5000? Now that we have the cache in place (and it's prewarmed and we can copy it around with some ops busywork), it's more likely that we'll read from the cache than from the node. It's also fine if the batch is too large (for the batch timeout); it just means we'll lock some blocks too many, but the lock will auto-release soon enough.
config/config.go
Outdated
Comment on lines
254
to
255
| // BatchSize determines the number of blocks the block analyzer | ||
| // processes per batch. |
There was a problem hiding this comment.
Suggested change
| // BatchSize determines the number of blocks the block analyzer | |
| // processes per batch. | |
| // BatchSize determines the maximum number of blocks the block analyzer | |
| // processes per batch. This is relevant only when the analyzer is still catching | |
| // up to the latest block. |
ptrus
force-pushed
the
ptrus/feature/block-batch_size-param
branch
from
cc9f71e to
3ee0373
Compare
ptrus
force-pushed
the
ptrus/feature/block-batch_size-param
branch
from
3ee0373 to
d870f41
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a config parameter for configuring the batch size of the block analyzer.
The default value of 100 is likely unreasonably small for production (especially if reading blocks from cache).
I think a reasonable value for production is such that the batch processing takes 30s-1min in the expected/average case (the batch processing and lock timeouts are both 5 min). Otherwise, the "mega-query" is run unnecessarily often.