Skip to content

Troubleshooting

jchappelow edited this page Oct 8, 2024 · 9 revisions

This page details commonly encountered errors and possible resolutions.

duplicate node ID

{"level":"info","ts":1728330119.1452067,"logger":"kwild.cometbft","caller":"p2p/switch.go:714","msg":"Ignoring inbound connection: error while adding peer","module":"p2p","err":"duplicate ID<02b2dd22472c7b8f539bf2bee472ef03d76d0aa2>","id":"02b2dd22472c7b8f539bf2bee472ef03d76d0aa2"}

There are two node instances running with the same private key.

Stop the duplicate instance. Make a new one with a different private key. If may be necessary to either change the IP address of the new node, or on the other nodes that were rejecting inbound, delete abci/addrbook.json

apphash mismatch

Possible bug. The node made change to state that were different to what the validators who approved the block got.

Check your kwil version. Stop, drop the postgres data, restart to being replaying blocks.

no new blocks being produced

  • check node's peers
  • check other nodes' block counts with RPC
  • ensure more than 2/3 of validators are online.

timeout waiting for broadcast

timed out waiting for tx to be included in a block

Usually this is with the broadcast mode that waits for it to be included in a block.

Either keep checking with utils query-tx, or blocks are not being mined.

statesync trust period

When attempting to use statesync to sync a new node:

"msg":"failed to fetch and verify app hash","module":"statesync","err":"verify from #20401 to #20402 failed: old header has expired at 2024-10-03 14:19:34.658537651 +0000 UTC (now: 2024-10-07 07:32:29.50737627 +0000 UTC m=+24.376991152)"

Edit config.toml with chain.p2p.statesync.trust_period = 10000h

conflicting vote / duplicate vote

found conflicting vote from ourselves; did you unsafe_reset a validator?

"msg_type":"*consensus.VoteMessage","err":"conflicting votes from validator B04C9C60030FB70E55C706605A727AA5AC2700DD""

Two possible causes:

  • You have two validators running with the same private key.
  • If this is halting the entire chain, the majority of validators committed to a block, then deployed a change

app block height higher than core

error during handshake: error on replay: app block height (22496) is higher than core (22495)

higher up:

"msg":"Preparing ABCI application at height 22496, appHash 4a4a9e97147db6ced1bdd783f31f7d38b8528b3a143437ad35e303702c888dcd"}

The appHeight is what is reported from the postgresql (app state) tables, while the other two are from the cometbft data that was rolled back.

When postgresql database is deleted/re-created empty, it will start appHeight at 0, then reapply all blocks fairly quickly to rebuild state from the blockchain data.