`parachain-omni-node` binary #3597

kianenigma · 2024-03-06T12:36:36Z

Related to #62, #5, #1984 and possibly more.

This node now should be able to take any runtime that has no consensus related expectations (ie. data being in header digests etc), and run it with its own manual consensus. It should be ideal for a scenario like a light tutorial.

But moreover, if the approach here is good, it can already be used to build solo-chain-omni-node and parachain-omni-node etc.

kianenigma · 2024-03-06T12:38:52Z

templates/minimal/node/src/command.rs

+			properties.insert("tokenDecimals".to_string(), 0.into());
+			properties.insert("tokenSymbol".to_string(), "MINI".into());
+
+			// `.with_genesis_config(Default::default)` won't work, but should.


@michalkucharczyk this was my main chain-spec related issue so far, I would have expected it to work without this. Or perhaps the Default::default should do this for me.

This API is under some development. The idea is here: #1984 and some PoC: #2714.

This would help here (fetching default is trivial to add).

What I propose is to move predefined genesis configs into runtime (and call them presets), and extend GenesisBuilder runtime API to fetch them.
This would allow for functionality mentioned in #2714 (comment).

The reasoning for this is that hand-crafting chain-spec files maybe difficult as the serialization of types can be tricky to guess. Also default RuntimeGenesisConfig is typically useless (e.g. does not contains keys).

Actually, .with_genesis_config_patch(json!({})) should do the job. (which means empty patch for default config).

kianenigma · 2024-03-06T12:39:57Z

templates/minimal/node/src/impl_fake_runtime_apis.rs

+pub struct FakeRuntime;
+use crate::types::{AccountId, Header, Nonce, OpaqueBlock as Block};
+
+sp_api::impl_runtime_apis! {


We can also manually implement this stuff without this macro, but this macro already does the job easily.

…force-remove-that-native-runtime-sir

bkchr · 2024-03-24T00:52:26Z

parachain-omni-node

polkadot-parachains should be this binary. No need to have another special binary for this IMO.

kianenigma · 2024-04-01T08:40:46Z

parachain-omni-node

polkadot-parachains should be this binary. No need to have another special binary for this IMO.

Agreed, I didn't know it is already a node capable of this.

…force-remove-that-native-runtime-sir

Polkadot-Forum · 2024-04-04T08:02:51Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/parity-tech-update-for-march/7226/1

Polkadot-Forum · 2024-04-27T16:27:28Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/rfc-should-we-launch-a-thousand-cores-program/7604/1

Polkadot-Forum · 2024-05-01T07:32:09Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/paritytech-update-for-april/7646/1

Polkadot-Forum · 2024-05-02T23:35:27Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/stabilizing-polkadot/7175/20

xlc · 2024-05-08T02:09:53Z

What is blocking this (assuming it is blocked)?

Upgrade the client to the new polkadot-sdk is always a big overhead and error-prune process and for no real benefits. Omni node is the solution to allow parachain teams only focus on the runtime. This is a super critical work and I would like to see it released ASAP.

kianenigma · 2024-05-08T15:56:07Z

cumulus/service/src/lib.rs

+// same anyways. Possibly requires a big refactor to move Block to an associated type instead of
+// generic in a few places.
+
+pub type ParachainClient<Block, RuntimeApi, HostFns> =


What I am trying with this crate is as follows:

this is the "cumulus service builder" helpers. There should be one single source of truth for how to create eg. an aura with async backing import queue. At the moment we copy paste this in multiple places even in our repo.

Almost everything in this crate is generic over 3 things:

Block

RuntimeApi

HostFunctions

So a user of this crate should provide these 3 things, and then they can use the helper functions for

building the import queue

building consensus

For each of:

relay consensus

aura consensus

async backing

relay -> aura (we should get rid of this one :/)

If we learn we need customization for database (for EVM we 99% need it), we add it here in a similar fashion.

The first milestone I see is as follows:

Finish this crate, and make sure the same underlying code is being used in polkadot-parachian and /template/parachain. As noted, for example, one function to build aura consensus with async backing, not two instances copy-pasted.

Then we see where go from there :)

kianenigma · 2024-05-08T15:59:07Z

omni-nodes/common/src/fake_runtime.rs

+}
+use whatever::*;
+
+sp_api::impl_runtime_apis! {


This should ideally move to cumulus_service now. I think one set of fake runtime impls should be enough.

michalkucharczyk · 2024-05-08T19:08:09Z

omni-nodes/parachain/src/chain_spec.rs

+	.with_name("Development")
+	.with_id("dev")
+	.with_chain_type(ChainType::Development)
+	.with_genesis_config_patch(testnet_genesis(


Don't know if this is final or temporary proposal. I'd suggest fetching "local_testnet"/"development" genesis from runtime here as omni-node shall make no assumptions about runtime's internals.

If omni-node is to provide two hardcoded chainspecs ("development", "local_testnet") then we could require runtime's author to provided corresponding presets. If runtime does not provide those presets we could fail.

oh, it is not actually used...

michalkucharczyk · 2024-05-08T19:36:15Z

omni-nodes/parachain/src/command.rs

+			let extensions = Extensions { relay_chain: "rococo-local".into(), para_id: 2000 };
+
+			let tmp = sc_chain_spec::GenesisConfigBuilderRuntimeCaller::<'_, ()>::new(&code);
+			let genesis = tmp.get_default_config()?;


Note: default runtime config may not be runnable - some crucial components may not be properly initialized in default runtime genesis.

kianenigma · 2024-05-09T09:27:41Z

What is blocking this (assuming it is blocked)?

Upgrade the client to the new polkadot-sdk is always a big overhead and error-prune process and for no real benefits. Omni node is the solution to allow parachain teams only focus on the runtime. This is a super critical work and I would like to see it released ASAP.

There should be no blocker for syncing normal parachains as far as I can say. I have synced hydra, interlay and frequency at 40fbf47 and it all works. Although, as additional tests, we should also test:

If the DApp of the corresponding parachain can work with the omni-node (missing RPCs)
If the node can collate blocks (important, else this is not useful)

What seems like a blocker to me is syncing system chains that do relay -> aura. Tbh I wish to not need to keep the specialized code for this use case around. Maybe it is a reasonable compromise to drop support for this type of consensus in the omni-node. If you want to sync the old versions of eg. AH, you should use a non-omni-legacy-node or something.

The code in this commit already provides some soft degree of configurability through a builder pattern as well. This can either remain a builder or become CLI flags.

polkadot-sdk/omni-nodes/parachain/src/main.rs

Lines 14 to 20 in 40fbf47

    
           fn main() -> sc_cli::Result<()> { 
        
           	let node = Builder::default() 
        
           		// .parachain_consensus(builder::ParachainConsensus::Relay(12)) 
        
           		.parachain_consensus(builder::ParachainConsensus::Aura(12)) 
        
           		.build()?; 
        
           	node.run() 
        
           }

xlc · 2024-05-09T12:27:57Z

let’s get a simple version out that only supports the most common scenario and then we can iteratively make it compatible with more chains

Polkadot-Forum · 2024-05-10T14:52:50Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-parachain-omni-node-gathering-ideas-and-feedback/7823/1

kianenigma · 2024-05-21T14:08:18Z

#4369 is a pretty similar PR to what is being done here, and can be reviewed and merged as a first step.

paritytech-cicd-pr · 2024-05-23T06:40:08Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6265821

kianenigma · 2024-06-10T00:13:42Z

This branch will likely be closed in favor of s slimmer version brought to you by @serban300. I will leave open for visibility for now, but feel free to close when a replacemebt comes by.

prototype seems done

69bfb80

kianenigma added the T0-node This PR/Issue is related to the topic “node”. label Mar 6, 2024

kianenigma commented Mar 6, 2024

View reviewed changes

rzadp mentioned this pull request Mar 15, 2024

[EngAut] Automate Sync/Release the polakdot-sdk templates paritytech/eng-automation#6

Closed

kianenigma added 6 commits March 21, 2024 09:28

revive minimal-template, make a new omni-node crate

3aed1f6

Merge branch 'master' of github.com:paritytech/polkadot-sdk into kiz-…

1b7e4c0

…force-remove-that-native-runtime-sir

cleanup a bit

c9383ba

wrap up minimal, next is solochain

8955471

add solochain omni

aac4673

refactor fake runtime apis into separate file

a3eb65c

kianenigma requested a review from a team as a code owner March 22, 2024 15:09

make it build

80bc7bb

Merge branch 'master' of github.com:paritytech/polkadot-sdk into kiz-…

2ed72f9

…force-remove-that-native-runtime-sir

kianenigma mentioned this pull request Apr 4, 2024

Fix polkadot-parachain to work just with chain-spec #3944

Closed

half finished builder, now moving to revise polkoadt-parachain

daa85ee

dummy checkpoint

94cf40f

builds

7d3aed4

kianenigma commented May 8, 2024

View reviewed changes

michalkucharczyk reviewed May 8, 2024

View reviewed changes

fix up RPC stuff a bit, and make it configurable

3dd444a

kianenigma added 2 commits May 9, 2024 10:03

fix up RPC stuff a bit, and make it configurable

2bf9080

can sync a couple of parachains, and is now configurable :)

40fbf47

kianenigma changed the title ~~WIP/DNM: Turn minimal-template-node into omni-node~~ parachain-omni-node binary May 9, 2024

kianenigma mentioned this pull request May 10, 2024

release chain-spec-builder #4352

Closed

kianenigma added 2 commits May 10, 2024 12:12

remove a lot of stuff that is not needed

67cfd1b

more specs

b74e39f

add a bunch more specs

88fd32b

kianenigma mentioned this pull request May 21, 2024

Split polkadot parachains service file into multiple modules #4369

Closed

Master.into()

e2142d6

serban300 mentioned this pull request Jun 12, 2024

polkadot-parachain-bin: small cosmetics and improvements #4666

Merged

kianenigma closed this Jul 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`parachain-omni-node` binary #3597

`parachain-omni-node` binary #3597

kianenigma commented Mar 6, 2024

kianenigma Mar 6, 2024

michalkucharczyk Mar 6, 2024 •

edited

Loading

michalkucharczyk Mar 6, 2024 •

edited

Loading

kianenigma Mar 6, 2024

bkchr commented Mar 24, 2024

kianenigma commented Apr 1, 2024

Polkadot-Forum commented Apr 4, 2024

Polkadot-Forum commented Apr 27, 2024

Polkadot-Forum commented May 1, 2024

Polkadot-Forum commented May 2, 2024

xlc commented May 8, 2024

kianenigma May 8, 2024 •

edited

Loading

kianenigma May 8, 2024

michalkucharczyk May 8, 2024 •

edited

Loading

michalkucharczyk May 8, 2024

michalkucharczyk May 8, 2024

kianenigma commented May 9, 2024 •

edited

Loading

xlc commented May 9, 2024

Polkadot-Forum commented May 10, 2024

kianenigma commented May 21, 2024

paritytech-cicd-pr commented May 23, 2024

kianenigma commented Jun 10, 2024

parachain-omni-node binary #3597

parachain-omni-node binary #3597

Conversation

kianenigma commented Mar 6, 2024

kianenigma Mar 6, 2024

Choose a reason for hiding this comment

michalkucharczyk Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

michalkucharczyk Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

kianenigma Mar 6, 2024

Choose a reason for hiding this comment

bkchr commented Mar 24, 2024

kianenigma commented Apr 1, 2024

Polkadot-Forum commented Apr 4, 2024

Polkadot-Forum commented Apr 27, 2024

Polkadot-Forum commented May 1, 2024

Polkadot-Forum commented May 2, 2024

xlc commented May 8, 2024

kianenigma May 8, 2024 • edited Loading

Choose a reason for hiding this comment

kianenigma May 8, 2024

Choose a reason for hiding this comment

michalkucharczyk May 8, 2024 • edited Loading

Choose a reason for hiding this comment

michalkucharczyk May 8, 2024

Choose a reason for hiding this comment

michalkucharczyk May 8, 2024

Choose a reason for hiding this comment

kianenigma commented May 9, 2024 • edited Loading

xlc commented May 9, 2024

Polkadot-Forum commented May 10, 2024

kianenigma commented May 21, 2024

paritytech-cicd-pr commented May 23, 2024

kianenigma commented Jun 10, 2024

`parachain-omni-node` binary #3597

`parachain-omni-node` binary #3597

michalkucharczyk Mar 6, 2024 •

edited

Loading

michalkucharczyk Mar 6, 2024 •

edited

Loading

kianenigma May 8, 2024 •

edited

Loading

michalkucharczyk May 8, 2024 •

edited

Loading

kianenigma commented May 9, 2024 •

edited

Loading