Move syncing code from `sc-network-common` to `sc-network-sync` #1912

dmitry-markin · 2023-10-17T12:42:05Z

This PR moves syncing-related code from sc-network-common to sc-network-sync.

Unfortunately, some parts are tightly integrated with networking, so they were left in sc-network-common for now:

SyncMode in common/src/sync.rs (used in NetworkConfiguration).
BlockAnnouncesHandshake, BlockRequest, BlockResponse, etc. in common/src/sync/message.rs (used in src/protocol.rs and src/protocol/message.rs).

More substantial refactoring is needed to decouple syncing and networking completely, including getting rid of the hardcoded sync protocol.

Release notes

Move syncing-related code from sc-network-common to sc-network-sync. Delete ChainSync trait as it's never used (the only implementation is accessed directly from SyncingEngine and exposes a lot of public methods that are not part of the trait). Some new trait(s) for syncing will likely be introduced as part of Sync 2.0 refactoring to represent syncing strategies.

Builds upon #1736.

altonen

I don't think this should be silent. There are at least two external teams that are working on their own syncing implementations so this will likely affect them. At least the removal of ChainSync is something that should be mentioned, with a note that there will be some kind of new ChainSync trait in the near future.

bkchr

Looking good. Just a little bit confused why some types are being made public?

bkchr · 2023-10-29T18:04:45Z

substrate/client/network/sync/src/lib.rs

@@ -206,24 +206,65 @@ struct GapSync<B: BlockT> {

 /// Action that [`engine::SyncingEngine`] should perform after reporting imported blocks with
 /// [`ChainSync::on_blocks_processed`].
-enum BlockRequestAction<B: BlockT> {
+pub enum BlockRequestAction<B: BlockT> {


Why do we need to make this public?

It's consumed by SyncingEngine, that owns ChainSync.

But isn't this in the same crate?

Technically, SyncingEngine is in the same root module as ChainSync, because ChainSync is defined in lib.rs. So pub is not needed at all (even as pub(crate)). Semantically, these types are public for SyncingEngine, so I feel inclined to highlight this with either pub or pub(crate).

pub(crate) is probably a better option, but may be it's also worth moving ChainSync to a separate chain_sync.rs file.

@bkchr would it be better to move ChainSync and its types to a separate file and use pub(crate) for types consumed by SyncingEngine?

@bkchr would it be better to move ChainSync and its types to a separate file and use pub(crate) for types consumed by SyncingEngine?

Done. PR looks scary now due to moved code, but overall situation with ChainSync should have improved.

@bkchr @altonen Could you have a look if moving ChainSync to a separate file worth it: 2511cfc?

pub(crate) is probably a better option, but may be it's also worth moving ChainSync to a separate chain_sync.rs file.

Yeah sorry for these dumb questions :P I should have realized that the module is private to the crate. Pub is fine by me here.

I was thinking this reordering could've been done in another PR but whatever. SyncingEngine should be moved to sync/src/lib.rs but that can be done in another PR.

Why SyncingEngine should be moved to the root module and not kept in it's own module, may be reexporting it in the root module?

bkchr · 2023-10-29T18:07:18Z

with a note that there will be some kind of new ChainSync trait in the near future.

What is the status of completely getting rid off sync in the networking crate? Because ideally there should not exist a chain sync trait at all.

dmitry-markin · 2023-10-30T08:32:56Z

with a note that there will be some kind of new ChainSync trait in the near future.

What is the status of completely getting rid off sync in the networking crate? Because ideally there should not exist a chain sync trait at all.

There won't be ChainSync trait in networking for sure. There may be some new traits for syncing in sc-network-sync though.

bkchr · 2023-10-30T08:40:24Z

There may be some new traits for syncing in sc-network-sync though.

Do we really need to make sync itself generic? I don't think so? As this is some self contained implementation.

altonen · 2023-10-30T08:44:46Z

There may be some new traits for syncing in sc-network-sync though.

Do we really need to make sync itself generic? I don't think so? As this is some self contained implementation.

That is not the current plan at least. We are most likely going to introduce some traits for the strategies that we discussed earlier, simply to make the code easier to maintain and plug additional syncing implementations without major headache.

bkchr · 2023-10-30T09:10:58Z

Okay :)

bkchr

Moving to its own file sounds reasonable!

bkchr · 2023-10-30T16:11:15Z

substrate/client/network/sync/src/chain_sync.rs

+}
+
+#[cfg(test)]
+mod test {


If you are at it, why not also move them into their own file?

Change introduced here: paritytech/polkadot-sdk#1912

* Get `entropy-shared` compiling with updated deps * Fix `entropy-shared` build for `wasm-no-std` * Make `pallet-staking-extension` compile Starts introducing some of the changes brought in by paritytech/polkadot-sdk#1484. * Introduce staking changes from paritytech/substrate#12970 * Add RuntimeFreezeReason from paritytech/polkadot-sdk#1900 * Re-map errors in staking extension pallet * Bump `pallet-programs` * Missed a crate dep for staking extension * Remove commented out code * Get `pallet-relayer` compiling * Fix relayer test compilation * Fix `pallet-free-tx`'s tests * Reorder config types * Bump `pallet-propagation` dependencies * Bump `pallet-slashing` dependencies * Bump `pallet-transaction-pause` This includes the deprecation of `Balances::transfer` from paritytech/polkadot-sdk#1226. * TaploFmt * Use `StakingAccount` instead of `.into()` * Import `vec` macro in `pallet-slashing` * Import `vec` in other pallets * Bump TOML dependencies for runtime Haven't fixed the Rust code yet though * Self define types instead of using `node-primitives` * Update staking related configs Values were mostly taken from the Polkadot runtime config * Update `pallet-preimage` config Changes introduced in paritytech/polkadot-sdk#1363 * Add `MaxNominators` to `pallet_babe` Introduced in paritytech/substrate#14471 * Add `MaxNominators` to `pallet_grandpa` Also introduced in paritytech/substrate#14471 * Add `RuntimeFreezeReason` to `pallet_nomination_pools` * Add `MaxTipAmount` to `pallet_tips` Introduced in paritytech/polkadot-sdk#1709 * Add `IdentityInformation` to `pallet_identity` Introduced in paritytech/polkadot-sdk#1661 * finish runtime * Use `crates.io` packages in `entropy` TOML This doesn't compile yet * Remove reference to `node_primitives` * Move imports under correct dependency section These shouldn't be built-deps, my bad * Change `WarpSyncParams` import to come from `sc_network_sync` Change introduced here: paritytech/polkadot-sdk#1912 * Add missing `sc-consensus-babe` package * Remove second generic from `DefaultImportQueue` This was removed in paritytech/substrate#14612 * Add temporary cursed RPC bounds * Switch `sp-consensus-grandpa` to be from Crates * Add Grandpa justification period to service * Rename `justification_period to` `justification_generation_period` * Add missing `block_relay` param Introduced in paritytech/polkadot-sdk#1524 * Try to clean up runtime types * Add `opaque` types to runtime These match what was in `node-primitives`. In particular we're interested in the `Block` type which we were using in a lot of places, and I incorrectly changed to be `Unchechecked` instead of `Opaque`. * Use opaque block type in service * Add full session key impls * Remove cursed trait bounds for RPC * Convert some tabs to spaces Sorry, I had to * Import `Vec` from `sp_std` in benchmarks Turns out the issue responsible for this change is this: paritytech/polkadot-sdk#172 * Pull `TrackedStorageKey` from `sp_storage` From here: paritytech/substrate#14787 * Add `BenchmarkHelper` to treasury pallet * TaploFmt * change max freezes * Sort TOML imports in runtime manifest * Move `version` field to be first in runtime manifest * Organize imports in pallet manifests * Sort and organize imports for client * Address TODOs in `entropy-shared` * Remove TODOs related to `node-primitives` * Pull `substrate-wasm-builder` from crates.io * TaploFmt * Fix benchmarking import post-merge * Fix another program import port-merge * Import `Vec` to programs benches * Wrong vec import 🙃 * Poke CI * Bump node metadata --------- Co-authored-by: Jesse Abramowitz <[email protected]>

…tytech#1912) This PR moves syncing-related code from `sc-network-common` to `sc-network-sync`. Unfortunately, some parts are tightly integrated with networking, so they were left in `sc-network-common` for now: 1. `SyncMode` in `common/src/sync.rs` (used in `NetworkConfiguration`). 2. `BlockAnnouncesHandshake`, `BlockRequest`, `BlockResponse`, etc. in `common/src/sync/message.rs` (used in `src/protocol.rs` and `src/protocol/message.rs`). More substantial refactoring is needed to decouple syncing and networking completely, including getting rid of the hardcoded sync protocol. ## Release notes Move syncing-related code from `sc-network-common` to `sc-network-sync`. Delete `ChainSync` trait as it's never used (the only implementation is accessed directly from `SyncingEngine` and exposes a lot of public methods that are not part of the trait). Some new trait(s) for syncing will likely be introduced as part of Sync 2.0 refactoring to represent syncing strategies.

Try to move syncing stuff from sc-network-common to sc-network-sync

b93fb1a

dmitry-markin added R0-silent Changes should not be mentioned in any release notes T0-node This PR/Issue is related to the topic “node”. labels Oct 17, 2023

dmitry-markin requested a review from altonen October 17, 2023 12:46

dmitry-markin added 6 commits October 18, 2023 15:19

Get rid of ChainSync trait

cad9895

Update visibility and rustdocs

a4fa0d7

Merge remote-tracking branch 'origin/master' into dm-get-rid-common-sync

cf62530

minor: comment

6097fee

minor: grammar

888f35c

minor: rustdoc

080883e

dmitry-markin changed the title ~~[draft] Move syncing code from sc-network-common to sc-network-sync~~ Move syncing code from sc-network-common to sc-network-sync Oct 18, 2023

dmitry-markin marked this pull request as ready for review October 18, 2023 14:26

dmitry-markin requested a review from andresilva as a code owner October 18, 2023 14:26

altonen approved these changes Oct 20, 2023

View reviewed changes

dmitry-markin removed the R0-silent Changes should not be mentioned in any release notes label Oct 20, 2023

dmitry-markin requested a review from bkchr October 23, 2023 14:29

bkchr approved these changes Oct 29, 2023

View reviewed changes

dmitry-markin added 2 commits October 30, 2023 15:45

Merge remote-tracking branch 'origin/master' into dm-get-rid-common-sync

250020b

Move ChainSync to separate file, update visibility

2511cfc

bkchr approved these changes Oct 30, 2023

View reviewed changes

dmitry-markin added 4 commits October 31, 2023 10:57

Make ChainSync public

ecdeb2c

Move ChainSync tests to separate file

d3b291a

Merge remote-tracking branch 'origin/master' into dm-get-rid-common-sync

8bfe0fa

Merge remote-tracking branch 'origin/master' into dm-get-rid-common-sync

62561fa

dmitry-markin merged commit 1cd6acd into master Nov 1, 2023
110 of 111 checks passed

dmitry-markin deleted the dm-get-rid-common-sync branch November 1, 2023 13:10

HCastano added a commit to entropyxyz/entropy-core that referenced this pull request Jan 20, 2024

Change WarpSyncParams import to come from sc_network_sync

c0ce1b5

Change introduced here: paritytech/polkadot-sdk#1912

github-actions bot mentioned this pull request Feb 19, 2024

Update substrate/polkadot/cumulus from v1.3.0 to v1.6.0 moondance-labs/tanssi#419

Closed

Ali-Usama mentioned this pull request Mar 4, 2024

Why is ExtendedPeerInfo private in release-polkadot-v1.6.0? #3556

Closed

github-actions bot mentioned this pull request Mar 13, 2024

Update polkadot-sdk from v1.3.0 to v1.7.2 moonbeam-foundation/moonbeam#2703

Closed

dmitry-markin mentioned this pull request Nov 27, 2024

Convert SyncingEngine::run to use tokio::select! instead of polling #2132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move syncing code from `sc-network-common` to `sc-network-sync` #1912

Move syncing code from `sc-network-common` to `sc-network-sync` #1912

dmitry-markin commented Oct 17, 2023 •

edited

Loading

altonen left a comment

bkchr left a comment

bkchr Oct 29, 2023

dmitry-markin Oct 30, 2023

bkchr Oct 30, 2023

dmitry-markin Oct 30, 2023

dmitry-markin Oct 30, 2023

dmitry-markin Oct 30, 2023 •

edited

Loading

bkchr Oct 30, 2023

dmitry-markin Oct 31, 2023

altonen Oct 31, 2023

dmitry-markin Oct 31, 2023

bkchr commented Oct 29, 2023

dmitry-markin commented Oct 30, 2023

bkchr commented Oct 30, 2023

altonen commented Oct 30, 2023

bkchr commented Oct 30, 2023

bkchr left a comment

bkchr Oct 30, 2023

dmitry-markin Oct 31, 2023

Move syncing code from sc-network-common to sc-network-sync #1912

Move syncing code from sc-network-common to sc-network-sync #1912

Conversation

dmitry-markin commented Oct 17, 2023 • edited Loading

Release notes

altonen left a comment

Choose a reason for hiding this comment

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitry-markin Oct 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkchr commented Oct 29, 2023

dmitry-markin commented Oct 30, 2023

bkchr commented Oct 30, 2023

altonen commented Oct 30, 2023

bkchr commented Oct 30, 2023

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Move syncing code from `sc-network-common` to `sc-network-sync` #1912

Move syncing code from `sc-network-common` to `sc-network-sync` #1912

dmitry-markin commented Oct 17, 2023 •

edited

Loading

dmitry-markin Oct 30, 2023 •

edited

Loading