Errror when restarting a node to sync blocks after setting halt-height in app.toml

[jaybxyz]

Summary of Bug

Gracefully halt the node at certain block height works fine, but I am not too sure if this is normal behavior that the node can't be restarted to sync blocks after specifying halt-height=0.

Dec  2 09:13:21 ip-172-31-18-43 gaiad[701]: I[2019-12-02|09:13:21.921] starting ABCI with Tendermint                module=main
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: panic: state.AppHash does not match AppHash after replay. Got
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: 9A9117C8C3859FB7D63D968F7E757840AAA0C077A1942CA4AC7F65A711C1860F, expected .
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: State: {{{10 0} 0.32.7} gaia-13006 10 0 A6B0CD6774A61043F3CE7AAF3C9BBEAF462DEE5ACEAE8758FBA9CE8E12EAE9A3:1:1684DAD04B89 2019-08-20 18:13:22.889296243 +0000 UTC ValidatorSet{
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]:   Proposer: Validator{4A34D913697AF5ED8D61C120E20178DFE234D70A PubKeyEd25519{FFE0064A296FED20269821373E78B6330C3915BCB4C0872F26CA9DA4D488B109} VP:210015 A:-953619}
................................................................................................
................................................................................................
................................................................................................
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: Did you reset Tendermint without resetting your application's data?
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: goroutine 1 [running]:
Dec  2 09:13:22 ip-172-31-18-43 gaiad[701]: github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState(0xc000b547c0, 0x20, 0x20, 0xa, 0x0, 0xc0004a5c36, 0x6, 0xc0004a5c50, 0xa, 0xa, ...)

Version

Gaia v2.0.3 - Cosmos SDK v0.37.4

Steps to Reproduce

Gaia 13006 Testnet

1. Set halt-height = 100 in app.toml
2. Start syncing to see if the node gracefully halts.
3. Set halt-height = 0 in app.toml
4. Restart to sync
5. Failed with Wrong Block Header App Hash error message.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[fedekunze]

thanks @kogisin. Indeed, we shouldn't allow halt-height = 0 in the configuration (i.e the app should read the config and check it has a valid halt-height before starting)

[jaybxyz]

@fedekunze Thanks for your quick response.
How about this case? This also fails to restart.
I assume halt-height is one-time use?

1. Set halt-height = 10 in app.toml
2. Start syncing to see if the node gracefully halts.
3. Set halt-height = 20 in app.toml
4. Restart to sync
5. Failed to restart

[alexanderbez]

Can confirm this issue exists even with non-zero halt-height configuration. This idea was that this should typically be used once, although it's not out of the question I think to use it subsequently. i.e. we should support it.

I imagine it would work if you roll back by 1 block. In any case, I've opened a PR that fixes the problem.

[alessio]

Ah! Nice catch @kogisin 👍

[jaybxyz]

@alexanderbez
That is awesome ! I naturally thought that halt-height works more than once.
Btw, what command do you use to roll back by 1 block?

[alexanderbez]

Btw, what command do you use to roll back by 1 block?

This doesn't exist yet in the SDK, but the code is out there somewhere. Waiting on the folks from the Regen team to upstream.

/cc @aaronc @ethanfrey

[aaronc]

@alexanderbez see regen-network#6