-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
interop: parallelized receipt fetching #13044
Merged
Merged
+117
−31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
protolambda
reviewed
Nov 26, 2024
tynes
reviewed
Nov 27, 2024
protolambda
approved these changes
Dec 4, 2024
Loading status checks…
…-receipt-fetching
Merged in develop so that the CI job this is waiting on can pass. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #13044 +/- ##
===========================================
- Coverage 47.68% 43.08% -4.61%
===========================================
Files 923 754 -169
Lines 77580 67886 -9694
Branches 849 0 -849
===========================================
- Hits 36996 29246 -7750
+ Misses 37849 36177 -1672
+ Partials 2735 2463 -272
Flags with carried forward coverage won't be shown. Click here to find out more.
|
sigma
pushed a commit
that referenced
this pull request
Dec 19, 2024
* interop: parallelized receipt fetching * fix test * remove elastic thread count * Add Debug Message for Range Fetching * rename end to last * Remove Println --------- Co-authored-by: Matthew Slipper <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Allows for
ChainProcessor
to initiate up to 10 blocks worth of receipts at a time by creating a worker pool during fetching.Why
#12903
I have tried solving previously by adding BatchFetchReceipt to the eth client, which was pretty heavy. @protolambda suggests that parallelism on the client is roughly as effective as batch calls when operating over a web-socket, so I am proposing this solution too.
The receipt-focused PR can be found here: #12992
It's not clear if one of these solutions is more effective in production than another, but this PR is more to-the-point and it appears to help.
How
ChainProcessor
now features the following:maxFetcherThreads
represents a maximum number of threads that can be used. On creation, set to 10.update
is replaced withrangeUpdate
. Rather than takingnext
and only processing that block-height,rangeUpdate
takes no arguments and instead generates its own range of blocks to fetch, up tomaxFetcherThread
of them.Testing
Tested this using the
local-devnet
debug messages which are now part of this PR:"Fetching block" chain=900200 next=20 end=20 count=1
during normal operation"Fetching block" chain=900201 next=27 end=45 count=10
after pausing the supervisor"Fetching block" chain=900201 next=37 end=45 count=9
when there's less work than threads once caught up