Amoy testnet heimdall 1.0.10/1.1.0-beta broke before jorvik-hardfork height

[jun0tpyrc]

consensus layer (heimdall) height=5768654
rolled back to try , showing
47FE8530728F28365D236037AC55243299866FBABF2D937C871D3F4221FC8861 from 1.0.10
A25F11AAE04A089E9DCE70CA8827FF54BDAF398F22A26DF5F58259AACD47DE2D from 1.1.0-beta
expected FAE7FDF9DA93BCEB584E8DC34F7498FD5AD7CD0206D346560D5B77B53068CBCA

[DaraDoyoung]

having same issue. did you solve it?

[DaraDoyoung]

@jun0tpyrc the chain is already hardforked. it happened 5h ago.

[jajaislanina]

Once we hit this error we had to download the snapshot for heimdall and start the node again.
We had disabled bor_ namespace on Bor so some methods failed from heimdall and resulted in corrupt state.

[jun0tpyrc]

Once we hit this error we had to download the snapshot for heimdall and start the node again. We had disabled bor_ namespace on Bor so some methods failed from heimdall and resulted in corrupt state.

not very understanding this , so you succeeded with recover?!
even on 1.1.0-beta reworked with https://snapshots.publicnode.com/polygon-amoy-heimdall-5767815.tar.lz4 (before problem height) seems hitting same problem
i do have bor namespace on bor's side from old times

[jajaislanina]

well, kind of. recovered for a few thousand blocks but then hit the issue again.

[web3-nodeops]

We're seeing the same, If we d/l the snapshot above, the node works while running 1.0.10, but after switching to 1.1.0-beta it eventually fails with the same error

We've restarted a node already running 1.1.0-beta (post the hardfork block height) to take a backup of the data dir and that also now fails with the same issue

[matthewdarwin]

Here is the error we got.

panic: Failed to process committed block (5768655:A25F11AAE04A089E9DCE70CA8827FF54BDAF398F22A26DF5F58259AACD47DE2D): Wrong Block.Header.AppHash.  Expected DD5B3758C3C058AB988523275DD1025591EDF11334DE31DEA578F91C59239164, got FAE7FDF9DA93BCEB584E8DC34F7498FD5AD7CD0206D346560D5B77B53068CBCA
goroutine 117 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc001503180)
#011/home/runner/go/pkg/mod/github.com/maticnetwork/[email protected]/blockchain/v0/reactor.go:344 +0xfa5
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart in goroutine 1
#011/home/runner/go/pkg/mod/github.com/maticnetwork/[email protected]/blockchain/v0/reactor.go:118 +0x6e

[jajaislanina]

Got it running. Switched Heimdall to external bor bor_rpc_url = "https://rpc-amoy.polygon.technology"
Node synced to HEAD and now switched to our Bor.
Works again fine.

Make sure you enable bor_ namespace on Bor otherwise you will get corrupt db again

 [jsonrpc.http]
 api = ["eth","net","web3","admin","debug","personal","txpool","bor"]

 [jsonrpc.ws]
 api = ["eth","net","web3","admin","debug","personal","txpool","bor"]

[jun0tpyrc]

updating

bor_rpc_url = "https://rpc-amoy.polygon.technology"

in config/heimdall-config.toml

^
looks only with this line it becomes working
We have never unplugged any of ws/http's bor namespace API for weeks, but we still hit this in different combinations before switching to this bor_rpc_url

[matthewdarwin]

@jun0tpyrc I confirm your solution works.

[calinah]

Also experiencing the same error:

heimdall ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 47FE8530728F28365D236037AC55243299866FBABF2D937C871D3F4221FC8861, got FAE7FDF9DA93BCEB584E8DC34F7498FD5AD7CD0206D346560D5B77B53068CBCA

with heimdall v1.0.10 and v1.1.0-beta.

updating heimdall to use bor_rpc_url = "https://rpc-amoy.polygon.technology" does not make a difference on it's own. And I cannot add bor as a namespace as I'm using Erigon as execution layer. Am I missing a step or does anyone have any other suggestions?

[avalkov]

@calinah Please make sure your using v1.1.0-beta and take snapshot from here https://publicnode.com/snapshots#polygon
You need your execution to be in sync if its not you can use https://rpc-amoy.polygon.technology/ until it catches up.

Regarding erigon, in your config you must have http.api array, just add "bor" to it.

[calinah]

thank you @avalkov you suggestion fixed my issues

[icculp]

I'm a bit confused by this, doesn't bor need heimdall to sync? But now heimdall needs bor synced, before heimdall can sync, before bor can sync from heimdall?

[marcello33]

Hi @icculp,

When syncing from genesis without using a snapshot, the blocks in the [Jorvik, HFn] range require bor to be in sync. HFn refers to the upcoming hardfork on the Amoy network, which is scheduled for early January 2025 and will resolve this issue.
This problem will not occur on the mainnet, as a hardfork is expected there, likely in early February 2025.
Additionally, with the comprehensive refactoring of Heimdall-v2 currently underway (targeted for release in Q1 2025), this issue will no longer affect any network.

Thank you