Merge pull request #11 from j75689/separate-node

Add separate node tutorial for BSC
bnb-chain · Jul 28, 2022 · e48fbee · e48fbee
2 parents 58075d2 + 6a2c423
commit e48fbee
Show file tree

Hide file tree

Showing 5 changed files with 373 additions and 0 deletions.
diff --git a/docs/BSC-fast-node.md b/docs/BSC-fast-node.md
@@ -0,0 +1,155 @@
+---
+sidebar_label: BNB Smart Chain Fast Node
+hide_table_of_contents: false
+sidebar_position: 2
+---
+# How to Run A Fast Node on BNB Smart Chain
+
+## Fast Node Functions
+
+* Stores the full blockchain history on disk and can answer the data request from the network.
+* Receives and validates the new blocks and transactions.
+* Verifies the states of every accounts.
+
+## Suggested Requirements
+
+### Fast Node
+- VPS running recent versions of Mac OS X or Linux.
+- **IMPORTANT** 1T GB of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 250MB/S throughput, read latency <1ms. (if start with snap/fast sync, it will need NVMe SSD)
+- 16 cores of CPU and 64 gigabytes of memory (RAM).
+- Suggest m5zn.3xlarge instance type on AWS, c2-standard-16 on Google cloud.
+- A broadband Internet connection with upload/download speeds of 5 megabyte per second
+
+## Settings
+
+## Steps to Run a Fast Node
+
+### Sync From Snapshot (Recommended)
+
+1. Download the pre-build binaries from [release page](https://github.com/bnb-chain/bsc/releases/latest) or follow the instructions below:
+
+```bash
+# Linux
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_linux |cut -d\" -f4)
+# MacOS
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_mac |cut -d\" -f4)
+```
+
+2. Download the config files
+
+Download `genesis.json` and `config.toml` by:
+
+```bash
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep mainnet |cut -d\" -f4)
+unzip mainnet.zip
+```
+
+3. Download snapshot
+
+Download latest snapshot from [Download site](https://github.com/bnb-chain/bsc-snapshots)
+Follow the guide to structure the files.
+
+4. Prune all trie data
+
+Fast node does not need trie data anymore, prune the trie data by the following command.
+```
+geth snapshot insecure-prune-all --datadir ./node ./config.toml
+```
+
+### Sync From Genesis Block (Not Recommended)
+
+1.Build from source code
+
+Make sure that you have installed [Go 1.13+](https://golang.org/doc/install) and have added `GOPATH` to `PATH` environment variable
+
+```bash
+git clone https://github.com/bnb-chain/bsc
+# Enter the folder bsc was cloned into
+cd bsc
+# Compile and install bsc
+make geth
+```
+
+or you can download the pre-build binaries from [release page](https://github.com/bnb-chain/bsc/releases/latest) or follow the instructions below:
+
+```bash
+# Linux
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_linux |cut -d\" -f4)
+# MacOS
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_mac |cut -d\" -f4)
+```
+
+2.Download the config files
+
+Download `genesis.json` and `config.toml` by:
+
+```bash
+## mainet
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep mainnet |cut -d\" -f4)
+unzip mainnet.zip
+
+## testnet
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep testnet |cut -d\" -f4)
+unzip testnet.zip
+```
+
+3.Write genesis state locally
+
+```bash
+geth --datadir node init genesis.json
+```
+
+You could see the following output:
+
+```
+INFO [05-19|14:53:17.468] Allocated cache and file handles         database=/Users/huangsuyu/Downloads/bsc/node/geth/chaindata cache=16.00MiB handles=16
+INFO [05-19|14:53:17.498] Writing custom genesis block
+INFO [05-19|14:53:17.501] Persisted trie from memory database      nodes=21 size=56.84KiB time=357.915µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-574.00B
+INFO [05-19|14:53:17.502] Successfully wrote genesis state         database=chaindata hash=7d79cc…fb0d1e
+INFO [05-19|14:53:17.503] Allocated cache and file handles         database=/Users/huangsuyu/Downloads/bsc/node/geth/lightchaindata cache=16.00MiB handles=16
+INFO [05-19|14:53:17.524] Writing custom genesis block
+INFO [05-19|14:53:17.525] Persisted trie from memory database      nodes=21 size=56.84KiB time=638.396µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-574.00B
+INFO [05-19|14:53:17.528] Successfully wrote genesis state         database=lightchaindata hash=7d79cc…fb0d1e
+```
+
+## Start Fast Node Without Snapshot Verification
+1. Start your fast node without snapshot verification by verify nodes.
+
+```bash
+## start a fast node
+geth --tries-verify-mode none --config ./config.toml --datadir ./node  --cache 8000 --rpc.allow-unprotected-txs --txlookuplimit 0
+```
+
+## Start Fast Node With Snapshot Verification
+1. Add verifyNodes peers in config.toml.
+
+```
+[Node.P2P]
+MaxPeers = 1350
+NoDiscovery = false
+BootstrapNodes = ["enode://...", "enode://...", ...]
+VerifyNodes = ["enode://...", "enode://...", ...]
+StaticNodes = ["enode://...", "enode://...", ...]
+ListenAddr = ":30311"
+EnableMsgEvents = false
+```
+
+2. Start your fast node with snapshot verification by verify nodes.
+
+```bash
+## start a fast node
+geth --tries-verify-mode full --config ./config.toml --datadir ./node  --cache 8000 --rpc.allow-unprotected-txs --txlookuplimit 0
+```
+
+## Node Maintainence
+
+### Peer Discovery
+The bootstrap nodes will be enhanced in the short future. So far, a discovery http service will provide some stable public p2p peers for syncing. Please visit https://api.binance.org/v1/discovery/peers to get dynamic peer info. You can append the peer info to the `StaticNodes` in the config.toml to enhance the networking of the full nodes. To avoid crowded networking, the discovery service will change the peer info from time to time, try fetch new ones if the connected peers of full node are too few.
+
+### Binary
+All the clients are suggested to upgrade to the latest release. The [latest version](https://github.com/bnb-chain/bsc/releases/latest) is supposed to be more stable and get better performance.
+
+## Upgrade Geth
+
+Please read [this guide](./upgrade-fullnode.md)
+
diff --git a/docs/BSC-separate-node.md b/docs/BSC-separate-node.md
@@ -0,0 +1,49 @@
+---
+sidebar_label: BNB Smart Chain Separate Node
+hide_table_of_contents: false
+sidebar_position: 2
+---
+# Introduce
+
+![img](../static/img/separate-node-architecture.png)
+
+Currently, a BSC node has two kinds of state world: MPT and snapshot. MPT(Merkle Patricia Tries) is a tree-structured state world. The key function of MPT is to generate the state root to ensure state consistency, while the query/commit on MPT is quite slow. Snapshot is a flattened key-value-based state world. Snapshot provides fast queries and commits. The storage size of the snapshot increases slowly even with a large transaction volume. Snapshot is usually used for block processing, while MPT is used for state verification.
+
+In order to lower the hardware requirement and keep security, we introduce two types of nodes to make full use of different storage, one is named fast node, and the other is named verify node. The fast node will do block processing with snapshot, it will do all verification against blocks except state root. The verify node receives diffhash from the fast node and then responds MPT root to the fast node.
+
+The fast node doesn’t need to store MPT, so the storage and computation requirement will be much lower.
+
+## Roles
+
+- Fast Node
+  It does full sync using only Snapshot and generates difflayer. It needs the confirm message from the verify node before freezing the blocks, it has to wait until it receives a confirm message from the verify node for the ancestor block before inserting new blocks.
+- Verify Node
+  It is a normal BSC full node that does full sync using Snapshot and MPT, and generates difflayer. It receives diffhash from fast nodes, finds the corresponding difflayer whose diffhash is matched, then responds MPT root message to the fast node.
+
+
+## Related Commands
+
+#### Fast Node
+Introduce a new **tries-verify-mode** setting, there are four modes:
+- **local**: a normal full node with complete state world(both MPT and snapshot), merkle state root will be verified against the block header.
+- **full**: a fast node with only snapshot state world. Merkle state root is verified by the trustworthy remote verify node by comparing the diffhash(an identify of difflayer generated by the block) and state root.
+- **insecure**: same as full mode, except that it can tolerate without verifying the diffhash when verify node does not have it.
+- **none**: no merkle state root verification at all, there is no need to setup or connect remote verify node at all, it is more light comparing to full and insecure mode, but get a very little chance that the state is not consistent with other peers.
+
+If the fast node runs in not local mode, the node will disable diff protocol by default, If the fast node runs in full or light mode, the node will enable trust protocol by default.
+
+` ./geth --config ./config.toml --datadir ./node --syncmode full --cache 5000 --tries-verify-mode none`
+
+#### Verify node
+When a full node has enabled the trust protocol, it can serve as a verify node, at the same time, we will recommend you to enable persist diff, disable snap protocol and diff protocol when running a verify node.
+
+` ./geth --config ./config.toml --datadir ./node --syncmode full --cache 5000 --persistdiff --enabletrustprotocol --disablesnapprotocol --disablediffprotocol`
+
+#### Prune tries node
+Prune the tires node:  `./geth snapshot insecure-prune-all --datadir ./node  ./genesis.json`
+
+## Run Verify Node
+Please read [this guide](./BSC-verify-node.md)
+
+## Run Fast Node
+Please read [this guide](./BSC-fast-node.md)
diff --git a/docs/BSC-verify-node.md b/docs/BSC-verify-node.md
@@ -0,0 +1,163 @@
+---
+sidebar_label: BNB Smart Chain Verify Node
+hide_table_of_contents: false
+sidebar_position: 2
+---
+# How to Run A Verify Node on BNB Smart Chain
+
+## Verify Node Functions
+
+* Stores the full blockchain history on disk and can answer the data request from the network.
+* Receives and validates the new blocks and transactions.
+* Verifies the states of every accounts.
+* Verifies the stated of Fast Node.
+
+## Suggested Requirements
+
+### Verify Node
+- VPS running recent versions of Mac OS X or Linux.
+- **IMPORTANT** 2T GB of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 250MB/S throughput, read latency <1ms
+- 16 cores of CPU and 64 gigabytes of memory (RAM)
+- Suggest m5zn.3xlarge instance type on AWS, or c2-standard-16 on Google cloud.
+- A broadband Internet connection with upload/download speeds of 10 megabyte per second
+
+## Settings
+
+## Chaindata Snapshot
+
+Please download the chain data [snapshot](./snapshot) and extract to your home folder to speed up
+
+```
+## Extract the data
+unzip geth.zip -d /NAME_OF_YOUR_HOME/node &
+```
+## Steps to Run a Verify Node
+
+### Sync From Snapshot (Recommended)
+
+1. Download the pre-build binaries from [release page](https://github.com/bnb-chain/bsc/releases/latest) or follow the instructions below:
+
+```bash
+# Linux
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_linux |cut -d\" -f4)
+# MacOS
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_mac |cut -d\" -f4)
+```
+
+2. Download the config files
+
+Download `genesis.json` and `config.toml` by:
+
+```bash
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep mainnet |cut -d\" -f4)
+unzip mainnet.zip
+```
+
+3. Download snapshot
+
+Download latest snapshot from [Download site](https://github.com/bnb-chain/bsc-snapshots)
+Follow the guide to structure the files.
+
+4. Start a full node
+```
+geth --config ./config.toml --datadir ./node --diffsync --cache 8000 --rpc.allow-unprotected-txs --txlookuplimit 0
+```
+
+### Sync From Genesis Block (Not Recommended)
+
+1.Build from source code
+
+Make sure that you have installed [Go 1.13+](https://golang.org/doc/install) and have added `GOPATH` to `PATH` environment variable
+
+```bash
+git clone https://github.com/bnb-chain/bsc
+# Enter the folder bsc was cloned into
+cd bsc
+# Compile and install bsc
+make geth
+```
+
+or you can download the pre-build binaries from [release page](https://github.com/bnb-chain/bsc/releases/latest) or follow the instructions below:
+
+```bash
+# Linux
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_linux |cut -d\" -f4)
+# MacOS
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep geth_mac |cut -d\" -f4)
+```
+
+2.Download the config files
+
+Download `genesis.json` and `config.toml` by:
+
+```bash
+## mainet
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep mainnet |cut -d\" -f4)
+unzip mainnet.zip
+
+## testnet
+wget   $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep testnet |cut -d\" -f4)
+unzip testnet.zip
+```
+
+3.Write genesis state locally
+
+```bash
+geth --datadir node init genesis.json
+```
+
+You could see the following output:
+
+```
+INFO [05-19|14:53:17.468] Allocated cache and file handles         database=/Users/huangsuyu/Downloads/bsc/node/geth/chaindata cache=16.00MiB handles=16
+INFO [05-19|14:53:17.498] Writing custom genesis block
+INFO [05-19|14:53:17.501] Persisted trie from memory database      nodes=21 size=56.84KiB time=357.915µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-574.00B
+INFO [05-19|14:53:17.502] Successfully wrote genesis state         database=chaindata hash=7d79cc…fb0d1e
+INFO [05-19|14:53:17.503] Allocated cache and file handles         database=/Users/huangsuyu/Downloads/bsc/node/geth/lightchaindata cache=16.00MiB handles=16
+INFO [05-19|14:53:17.524] Writing custom genesis block
+INFO [05-19|14:53:17.525] Persisted trie from memory database      nodes=21 size=56.84KiB time=638.396µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-574.00B
+INFO [05-19|14:53:17.528] Successfully wrote genesis state         database=lightchaindata hash=7d79cc…fb0d1e
+```
+
+4.Start your verify node
+
+
+!!! Note
+    BREAKING CHANGE: Non-EIP155 transactions (i.e. transactions which are not replay-protected) are now rejected by the RPC API. You can disable this restriction using the --rpc.allow-unprotected-txs command-line flag.
+
+```bash
+## start a verify node
+geth --enabletrustprotocol --persistdiff --diffblock 1000000000 --config ./config.toml --datadir ./node --cache 8000 --rpc.allow-unprotected-txs --txlookuplimit 0
+```
+
+## Node Maintainence
+
+### Peer Discovery
+The bootstrap nodes will be enhanced in the short future. So far, a discovery http service will provide some stable public p2p peers for syncing. Please visit https://api.binance.org/v1/discovery/peers to get dynamic peer info. You can append the peer info to the `StaticNodes` in the config.toml to enhance the networking of the full nodes. To avoid crowded networking, the discovery service will change the peer info from time to time, try fetch new ones if the connected peers of full node are too few.
+
+### Binary
+All the clients are suggested to upgrade to the latest release. The [latest version](https://github.com/bnb-chain/bsc/releases/latest) is supposed to be more stable and get better performance.
+
+### Storage
+According to the test, the performance of a verifynode will degrade when the storage size exceeds 1.5T. We suggest the verifynode always keep light storage by pruning the storage. 
+
+How to prune:
+
+1. Stop the BSC node.
+2. Run `nohup geth snapshot prune-state --datadir {the data dir of your bsc node} &`. It will take 3-5 hours to finish.
+3. Start the node once it is done.
+
+The maintainers should always have a few backup nodes.
+
+The hardware is also important, **make sure the SSD meets: 2T GB of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 250MB/S throughput, read latency <1ms**.
+
+### Diff Sync
+The diffsync protocol rolled out as a stable feature in release v1.1.5. Diff sync improves the syncing speed by 60%～70% approximately according to the test. All full nodes are suggested to enable it by adding `--diffsync` in the starting command.  
+
+### Light Storage
+When the node crashes or been force killed, the node will sync from a block that was a few minutes or a few hours ago. This is because the state in memory is not persisted into the database in real time, and the node needs to replay blocks from the last checkpoint once it start. The replaying time dependents on the configuration `TrieTimeout` in the config.toml.  We suggest you raise it if you can tolerate with long replaying time, so the node can keep light storage.
+
+## Upgrade Geth
+
+Please read [this guide](./upgrade-fullnode.md)
+
diff --git a/sidebars.js b/sidebars.js
@@ -62,6 +62,12 @@ const sidebars = {
                         label: 'Running Fullnode',
                         items: ['validator/fullnode','validator/snapshot','local', 'validator/upgrade-fullnode']
                       },
+                      {
+                        type: 'category',
+                        collapsed: true,
+                        label: 'Separate Node',
+                        items: ['BSC-separate-node','BSC-verify-node', 'BSC-fast-node']
+                      },
                       {
                         type: 'category',
                         collapsed: true,

diff --git a/static/img/separate-node-architecture.png b/static/img/separate-node-architecture.png