op-node: generalize and improve alt-sync

[protolambda]

Description

This PR iterates on top of the RPC-sync implementation to prepare for P2P-sync functionality as second form of alt-sync method, by encapsulating the RPC sync and updating its interface to make p2p sync fit in as well. Also fixes an off-by-one bug in the range to sync.

Changes:

• Add trust-RPC flag to L2SyncEndpointConfig. Previously it was hardcoded to true, yet also called blockhash verification. We can just allow the user to set it like they do with the L1 RPC.
• Change the sync-config check to be checked during the other config checks
• Clarify GetUnsafeQueueGap start is incl., end is excl., and fix off by one issue: we don't want to include the first block of the queue, but we do want to include the expected block number if queue is empty.
• Move RPC sync init code to its own function, separating it more from the l2 init
• Don't Close the RPC sync by reaching into attributes of the driver, but instead maintain the sync as separate lifetime thing in the node to close; there will be other sync methods.
• Put RPC sync requests behind an AltSync interface: the node itself implements it, and then routes it to the active & preferred sync methods that each also implement it, to allow more to be added.
  • The interface requests a full range of blocks, to avoid splitting the range into tinier tasks upfront, to allow the sync-method implementation to do batch / divide as desired.
  • The RPC SyncClient now consumes these range requests and does the fetching, instead of getting individual block numbers. Once new range information arrives, we now also drop previously scheduled information that may be stale.
• The driver now triggers sync sooner, when it's not changing the unsafe head as frequently as expected (thus not requesting syncing while it's deriving or has gossip coming in as expected). Also don't trigger sync when resetting the engine state (since we can't determine the range of L2 blocks accurately until the engine state is consistent again)

Tests

• Change RPC op-e2e test to disable batcher completely. When the sync runs around the same time as it syncs the data from L2, not all the blocks may be synced through the RPC. By disabling the batcher, we ensure it's really syncing through the RPC, and avoid potential flakiness.
• Change the same RPC test to use stringified block IDs, for more readable debug messages of what is and is not included in the received/published arrays
• Rename L2SyncRPCConfig to PreparedL2SyncEndpoint to match the other config utils of L1 and L2 endpoints that insert a prepared RPC client directly.

Metadata

Fix CLI-3643

[changeset-bot]

⚠️ No Changeset found

Latest commit: 46c639e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[netlify]

✅ Deploy Preview for opstack-docs ready!

Name	Link
🔨 Latest commit	46c639e
🔍 Latest deploy log	https://app.netlify.com/sites/opstack-docs/deploys/641b6571ef90800008aa4996
😎 Deploy Preview	https://deploy-preview-5179--opstack-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[mergify]

Hey @protolambda! This PR has merge conflicts. Please fix them before continuing review.

[ajsutton]

Looks like a good improvement. Just a couple of questions that I could be pretty easily persuaded to not worry about. :)

[protolambda]

Rebased and addressed PR review. Ready for final review

[codecov]

Codecov Report

Merging #5179 (163d072) into develop (c12366c) will decrease coverage by 2.42%.
The diff coverage is 14.28%.

❗ Current head 163d072 differs from pull request most recent head 46c639e. Consider uploading reports for the commit 46c639e to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #5179      +/-   ##
===========================================
- Coverage    40.09%   37.68%   -2.42%     
===========================================
  Files          371      215     -156     
  Lines        23453    18237    -5216     
  Branches       832        0     -832     
===========================================
- Hits          9404     6873    -2531     
+ Misses       13345    10690    -2655     
+ Partials       704      674      -30     

Flag	Coverage Δ
bedrock-go-tests	37.68% <14.28%> (+1.45%)	⬆️
common-ts-tests	?
contracts-bedrock-tests	?
contracts-tests	?
core-utils-tests	?
dtl-tests	?
fault-detector-tests	?
sdk-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
op-node/flags/flags.go	28.57% <ø> (ø)
op-node/node/client.go	0.00% <0.00%> (ø)
op-node/node/config.go	0.00% <0.00%> (ø)
op-node/node/node.go	0.00% <0.00%> (ø)
op-node/rollup/driver/driver.go	0.00% <0.00%> (ø)
op-node/rollup/driver/state.go	0.00% <0.00%> (ø)
op-node/service.go	0.00% <0.00%> (ø)
op-node/sources/sync_client.go	0.00% <0.00%> (ø)
op-node/rollup/derive/engine_queue.go	43.92% <72.72%> (-0.69%)	⬇️
op-node/rollup/derive/payloads_queue.go	100.00% <100.00%> (ø)

... and 161 files with indirect coverage changes

[protolambda]

Hmm, found an issue with the sync-changes: the payload queue is removing duplicates based on block number, not block hash: so on reorgs it is more likely to drop the new canonical blocks.

[trianglesphere]

Need to fix lint + tests, but after that should be good to go.

[mergify]

This PR has been added to the merge queue, and will be merged soon.

[mergify]

This PR is next in line to be merged, and will be merged as soon as checks pass.

[mergify]

This PR is next in line to be merged, and will be merged as soon as checks pass.