fix: enable to join existing network with State Sync #514

ulbqb · 2022-11-29T05:15:17Z

Description

I fix to enable a new node to join existing network with State Sync.

Problem

If validators field in genesis doc is empty, it occur panic in following line and can not process State Sync.
https://github.com/line/ostracon/blob/71df846cb3016861098bc617beebd6ba9aef2470/consensus/state.go#L730

Repro

You can reproduce this issue using lbm-sdk@ffb28cd.

STEP 1: Clone

git clone [email protected]:line/lbm-sdk.git && \
cd lbm-sdk && \
git checkout ffb28cdb86cae69766dabf70d45dfc67c90f5202

STEP 2: Setup

zsh init_node.sh sim 2 && \

sed -i 's/pruning = "nothing"/pruning = "custom"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-keep-recent = "0"/pruning-keep-recent = "500"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-keep-every = "0"/pruning-keep-every = "100"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-interval = "0"/pruning-interval = "10"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/snapshot-interval = 0/snapshot-interval = 100/g' ~/.simapp/simapp0/config/app.toml && \

sed -i 's/pruning = "nothing"/pruning = "custom"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-keep-recent = "0"/pruning-keep-recent = "500"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-keep-every = "0"/pruning-keep-every = "100"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-interval = "0"/pruning-interval = "10"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/snapshot-interval = 0/snapshot-interval = 100/g' ~/.simapp/simapp1/config/app.toml && \

simd init node2 --home ~/.simapp/simapp2 --chain-id sim > /dev/null 2>&1 && \

cp ~/.simapp/simapp0/config/genesis.json ~/.simapp/simapp2/config/genesis.json && \

sed -i 's#"tcp://127.0.0.1:26657"#"tcp://127.0.0.1:26661"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#"tcp://0.0.0.0:26656"#"tcp://0.0.0.0:26660"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#persistent_peers = ""'"#$(cat ~/.simapp/simapp0/config/config.toml | grep "persistent_peers =")#g" ~/.simapp/simapp2/config/config.toml && \
sed -i 's/"localhost:6060"/"localhost:6062"/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#rpc_servers = ""#rpc_servers = "tcp://0.0.0.0:26657,tcp://0.0.0.0:26659"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/enable = false/enable = true/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/trust_height = 0/trust_height = 100/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/trust_hash = ""/trust_hash = "DCE490B407D7339BC59867FCEF698A6E82C7319A390CFFDB65F2143DD6F29CF1"/g' ~/.simapp/simapp2/config/config.toml && \

sed -i 's/"0.0.0.0:9091"/"0.0.0.0:9095"/g' ~/.simapp/simapp2/config/app.toml && \
sed -i 's/"0.0.0.0:9090"/"0.0.0.0:9094"/g' ~/.simapp/simapp2/config/app.toml

STEP 3: Run

simd start --home ~/.simapp/simapp0 > ~/.simapp/simapp0/log.out 2>&1 & \
simd start --home ~/.simapp/simapp1 > ~/.simapp/simapp1/log.out 2>&1 & \
simd start --home ~/.simapp/simapp2

target error:

2022/12/01-03:46:55.466 INF starting ABCI with Ostracon
2022/12/01-03:46:55.481 INF starting node with ABCI Ostracon in-process
2022/12/01-03:46:55.5 INF Starting multiAppConn service impl=multiAppConn module=proxy
2022/12/01-03:46:55.5 INF Starting localClient service connection=query impl=localClient module=abci-client
2022/12/01-03:46:55.5 INF Starting localClient service connection=snapshot impl=localClient module=abci-client
2022/12/01-03:46:55.5 INF Starting localClient service connection=mempool impl=localClient module=abci-client
2022/12/01-03:46:55.5 INF Starting localClient service connection=consensus impl=localClient module=abci-client
2022/12/01-03:46:55.5 INF Starting EventBus service impl=EventBus module=events
2022/12/01-03:46:55.5 INF Starting PubSub service impl=PubSub module=pubsub
2022/12/01-03:46:55.509 INF Starting IndexerService service impl=IndexerService module=txindex
2022/12/01-03:46:55.509 INF Version info block=11 p2p=8 software=
2022/12/01-03:46:55.515 INF This node is not a validator addr=3245155EB2FBE104D8C89C118BC502D809611057 module=consensus pubKey=QnaCJY2utKQKb6yslsYdMxfEef5wkl1So/Iiv3S7nKM=
panic: empty validator set

goroutine 1 [running]:
github.com/line/ostracon/types.(*ValidatorSet).SelectProposer(0x40001275c0?, {0x4000fb2980?, 0x0?, 0x36959a8?}, 0x400165b4b8?, 0x4354b8?)
	github.com/line/[email protected]/types/validator_set.go:655 +0x22c
github.com/line/ostracon/consensus.(*State).updateToState(_, {{{0xb, 0x0}, {0x0, 0x0}}, {0x4001128da8, 0x3}, 0x1, 0x4001128fb8, 0x0, ...})
	github.com/line/[email protected]/consensus/state.go:730 +0x6b8
github.com/line/ostracon/consensus.NewState(_, {{{0xb, 0x0}, {0x0, 0x0}}, {0x4001128da8, 0x3}, 0x1, 0x4001128fb8, 0x0, ...}, ...)
	github.com/line/[email protected]/consensus/state.go:236 +0x55c
github.com/line/ostracon/node.createConsensusReactor(_, {{{0xb, 0x0}, {0x0, 0x0}}, {0x4001128da8, 0x3}, 0x1, 0x4001128fb8, 0x0, ...}, ...)
	github.com/line/[email protected]/node/node.go:471 +0x124
github.com/line/ostracon/node.NewNode(0x4000172c80, {0x26e1f98, 0x4000c48000}, 0x4001a97df0, {0x26c5090, 0x4000b895d8}, 0x26c5410?, 0x4000dacb60?, 0x4000c06010, {0x26e27b0, ...}, ...)
	github.com/line/[email protected]/node/node.go:831 +0x800
github.com/line/lbm-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x26f71c0, 0x4000d6ba10}, {0x0, 0x0}, {0x26e63b8, 0x4000dbdbb0}, ...}, ...)
	github.com/line/lbm-sdk/server/start.go:312 +0x634
github.com/line/lbm-sdk/server.StartCmd.func2(0x40000b4600?, {0x4000e94ca0?, 0x0?, 0x2?})
	github.com/line/lbm-sdk/server/start.go:139 +0x120
github.com/spf13/cobra.(*Command).execute(0x40000b4600, {0x4000e94c80, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:916 +0x60c
github.com/spf13/cobra.(*Command).ExecuteC(0x4000b70600)
	github.com/spf13/[email protected]/command.go:1044 +0x368
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:961
github.com/line/lbm-sdk/server/cmd.Execute(0x400006e768?, {0x4000b9f0a0, 0xd})
	github.com/line/lbm-sdk/server/cmd/execute.go:40 +0x1a4
main.main()
	github.com/line/lbm-sdk/simapp/simd/main.go:15 +0x3c

Fix

I removed setting proposer in consensus state.

Test

I test this change by following test. I plan to add e2e test like this test with another PR.

STEP 1: Setup

zsh init_node.sh sim 2 && \

sed -i 's/pruning = "nothing"/pruning = "custom"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-keep-recent = "0"/pruning-keep-recent = "500"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-keep-every = "0"/pruning-keep-every = "100"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/pruning-interval = "0"/pruning-interval = "10"/g' ~/.simapp/simapp0/config/app.toml && \
sed -i 's/snapshot-interval = 0/snapshot-interval = 100/g' ~/.simapp/simapp0/config/app.toml && \

sed -i 's/pruning = "nothing"/pruning = "custom"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-keep-recent = "0"/pruning-keep-recent = "500"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-keep-every = "0"/pruning-keep-every = "100"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/pruning-interval = "0"/pruning-interval = "10"/g' ~/.simapp/simapp1/config/app.toml && \
sed -i 's/snapshot-interval = 0/snapshot-interval = 100/g' ~/.simapp/simapp1/config/app.toml && \

simd init node2 --home ~/.simapp/simapp2 --chain-id sim > /dev/null 2>&1 && \

cp ~/.simapp/simapp0/config/genesis.json ~/.simapp/simapp2/config/genesis.json && \

sed -i 's#"tcp://127.0.0.1:26657"#"tcp://127.0.0.1:26661"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#"tcp://0.0.0.0:26656"#"tcp://0.0.0.0:26660"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#persistent_peers = ""'"#$(cat ~/.simapp/simapp0/config/config.toml | grep "persistent_peers =")#g" ~/.simapp/simapp2/config/config.toml && \
sed -i 's/"localhost:6060"/"localhost:6062"/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's#rpc_servers = ""#rpc_servers = "tcp://0.0.0.0:26657,tcp://0.0.0.0:26659"#g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/enable = false/enable = true/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/trust_height = 0/trust_height = 100/g' ~/.simapp/simapp2/config/config.toml && \
sed -i 's/trust_hash = ""/trust_hash = "DCE490B407D7339BC59867FCEF698A6E82C7319A390CFFDB65F2143DD6F29CF1"/g' ~/.simapp/simapp2/config/config.toml && \

sed -i 's/"0.0.0.0:9091"/"0.0.0.0:9095"/g' ~/.simapp/simapp2/config/app.toml && \
sed -i 's/"0.0.0.0:9090"/"0.0.0.0:9094"/g' ~/.simapp/simapp2/config/app.toml

STEP 2: Run

simd start --home ~/.simapp/simapp0 > ~/.simapp/simapp0/log.out 2>&1 & \
simd start --home ~/.simapp/simapp1 > ~/.simapp/simapp1/log.out 2>&1 & \
sleep 150 && \
simd start --home ~/.simapp/simapp2

STEP 3: Fix config.toml

You need to change trust_hash.

2022/12/01-03:28:09.104 INF Downloading trusted light block using options module=light
Error: failed to start state sync: failed to set up light client state provider: expected header's hash DCE490B407D7339BC59867FCEF698A6E82C7319A390CFFDB65F2143DD6F29CF1, but got 8941A2CA0B433BD6E7112E24DCB16D8C179FDF0467071977E91D11349A6DE166

sed -i 's/trust_hash = "DCE490B407D7339BC59867FCEF698A6E82C7319A390CFFDB65F2143DD6F29CF1"/trust_hash = "8941A2CA0B433BD6E7112E24DCB16D8C179FDF0467071977E91D11349A6DE166"/g'  ~/.simapp/simapp2/config/config.toml && \
simd start --home ~/.simapp/simapp2

codecov · 2022-11-29T05:35:22Z

Codecov Report

Merging #514 (02a0eb8) into main (71df846) will decrease coverage by 0.02%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #514      +/-   ##
==========================================
- Coverage   65.62%   65.60%   -0.03%     
==========================================
  Files         279      279              
  Lines       38015    38013       -2     
==========================================
- Hits        24948    24937      -11     
- Misses      11268    11271       +3     
- Partials     1799     1805       +6

Impacted Files	Coverage Δ
consensus/state.go	`73.29% <ø> (-0.79%)`	⬇️
privval/socket_listeners.go	`79.72% <0.00%> (-4.06%)`	⬇️
blockchain/v0/pool.go	`79.06% <0.00%> (-1.81%)`	⬇️
abci/client/socket_client.go	`72.09% <0.00%> (-0.88%)`	⬇️
consensus/reactor.go	`74.84% <0.00%> (-0.80%)`	⬇️
p2p/conn/connection.go	`79.80% <0.00%> (-0.58%)`	⬇️
privval/signer_listener_endpoint.go	`88.88% <0.00%> (ø)`
blockchain/v0/reactor.go	`69.49% <0.00%> (+1.01%)`	⬆️
mempool/reactor.go	`79.67% <0.00%> (+1.09%)`	⬆️
... and 3 more

ulbqb added 2 commits November 29, 2022 14:14

Remove SelectProposer

e5bb43b

Remove validators

50b9637

fix maverick

02a0eb8

ulbqb changed the title ~~WIP~~ Enable to join existing network with State Sync Nov 29, 2022

ulbqb self-assigned this Nov 29, 2022

ulbqb marked this pull request as ready for review November 29, 2022 07:29

ulbqb requested review from Kynea0b, torao and tnasu as code owners November 29, 2022 07:29

ulbqb requested a review from Mdaiki0730 November 29, 2022 07:40

ulbqb marked this pull request as draft November 29, 2022 08:50

ulbqb changed the title ~~Enable to join existing network with State Sync~~ fix: enable to join existing network with State Sync Nov 30, 2022

ulbqb mentioned this pull request Dec 1, 2022

We need State Sync test #516

Open

4 tasks

ulbqb marked this pull request as ready for review December 1, 2022 04:04

torao approved these changes Dec 1, 2022

View reviewed changes

tnasu approved these changes Dec 1, 2022

View reviewed changes

tnasu added the C: bug Classification: Something isn't working label Dec 1, 2022

ulbqb merged commit 357b3d7 into Finschia:main Dec 1, 2022

ulbqb deleted the fix/state_sync branch December 1, 2022 05:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: enable to join existing network with State Sync #514

fix: enable to join existing network with State Sync #514

ulbqb commented Nov 29, 2022 •

edited

Loading

codecov bot commented Nov 29, 2022 •

edited

Loading

fix: enable to join existing network with State Sync #514

fix: enable to join existing network with State Sync #514

Conversation

ulbqb commented Nov 29, 2022 • edited Loading

Description

Problem

Repro

STEP 1: Clone

STEP 2: Setup

STEP 3: Run

Fix

Test

STEP 1: Setup

STEP 2: Run

STEP 3: Fix config.toml

codecov bot commented Nov 29, 2022 • edited Loading

Codecov Report

ulbqb commented Nov 29, 2022 •

edited

Loading

codecov bot commented Nov 29, 2022 •

edited

Loading