[BUG] terrad cannot start after Columbus 5 Upgrade

[appop]

Describe the bug
terrad 0.5.5 0.5.7 cannot start

To Reproduce
/opt/terra/terrad start --home /data/luna
5:55PM INF starting ABCI with Tendermint
panic: proto: Int64Value: wiretype end group for non-group

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/rootmulti.getLatestVersion(0x2d07fe0, 0xc000010010, 0x10)
github.com/cosmos/[email protected]/store/rootmulti/store.go:955 +0x11c
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).LoadLatestVersion(0xc000034640, 0x10, 0x7f66039455b8)
github.com/cosmos/[email protected]/store/rootmulti/store.go:162 +0x37
github.com/cosmos/cosmos-sdk/baseapp.DefaultStoreLoader(0x2d11238, 0xc000034640, 0x0, 0xc004f14998)
github.com/cosmos/[email protected]/baseapp/baseapp.go:273 +0x34
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).LoadLatestVersion(0xc000c749c0, 0xc004e9e460, 0x7f65dc83edd8)
github.com/cosmos/[email protected]/baseapp/baseapp.go:263 +0x42
github.com/terra-money/core/app.NewTerraApp(0x2ceef40, 0xc001224d80, 0x2d07fe0, 0xc000010010, 0x0, 0x0, 0x1, 0xc004050b70, 0x7ffeae9fa7f7, 0xa, ...)
github.com/terra-money/core/app/app.go:571 +0x9d05
main.appCreator.newApp(0x2cf89b8, 0xc00105c5d0, 0x2d0c0b8, 0xc00104b8a0, 0x2d014f8, 0xc000de5840, 0xc00013db10, 0x2ceef40, 0xc001224d80, 0x2d07fe0, ...)
github.com/terra-money/core/cmd/terrad/root.go:214 +0x9d5
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc0010a9f00, 0x0, 0x0, 0x0, 0x2d139d8, 0xc000cd4c60, 0x0, 0x0, 0x2cf8970, 0xc00104b8a0, ...)
github.com/cosmos/[email protected]/server/start.go:255 +0x2df
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0010bec80, 0xc001172040, 0x0, 0x2, 0x0, 0x0)
github.com/cosmos/[email protected]/server/start.go:129 +0x209
github.com/spf13/cobra.(*Command).execute(0xc0010bec80, 0xc001172020, 0x2, 0x2, 0xc0010bec80, 0xc001172020)
github.com/spf13/[email protected]/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d43b80, 0x0, 0x0, 0xc00105a1a0)
github.com/spf13/[email protected]/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/[email protected]/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000d43b80, 0xc00105a1a0, 0x10, 0x2d0c0b8, 0xc00104b8a0)
github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x285
main.main()
github.com/terra-money/core/cmd/terrad/main.go:15 +0x45

Context & versions
/opt/terra/terrad version --long
name: terra
server_name: terrad
version: 0.5.7
commit: 1581f2f
build_tags: netgo muslc,
go: go version go1.16.5 linux/amd64
build_deps:

[vlads123]

I have the same issue.
The same terrad versions and the same error text.

All the previous steps from the upgrade instructions were successful, and genesis hashes matched.

Any help is very appreciated.

[yun-yeo]

Where is your terrad home path?

seems this error occurs when the genesis wrong. Did you install the genesis and did unsafe-reset-all before starting?

[appop]

terrad program path
/opt/terra/terrad
data path
/data/luna/
did unsafe-reset-all will remove all the data?

[appop]

Finally we start from a empty folder
but now we have problem below

1:43PM INF Starting PubSub service impl=PubSub module=pubsub
1:43PM INF Starting IndexerService service impl=IndexerService module=txindex
1:43PM INF ABCI Handshake App Info hash="��t��y��8-�p�TA4ͽg\x18�E����`hq�\u007f�" height=4871535 module=consensus protocol-version=0 software-version=0.5.7
1:43PM INF ABCI Replay Blocks appHeight=4871535 module=consensus stateHeight=4871535 storeHeight=4871536
1:43PM INF Replay last block using real app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.AppHash. Expected AFB074B4BB79A9C6382DB670C8544134CDBD6718EF45ABCAEBDE6068718E7FF1, got 505ABB49E09EDE0DF0C8E45402ED845ACFA55563EB08F8F7B90F40C10E1C94FC

[vlads123]

in my case cmd to start terra is terrad start --home /data
so configs are located in /data/config and data in /data/data

This configuration worked fine on before upgrade - only terrad version was changed

I had to wipe the data and start from scratch (with the same config folder just removed data folder and copied one created with terrad init ... command)

Now the node syncs and looks good

If that was expected that the data cannot be migrated from columbus-4 to new one then could you state this in the Instructions.

[yun-yeo]

before this issue resolved, if you met app hash diff crash, you need to resync from the scratch.

or try to use https://terra.quicksync.io

[kenorb]

Finally we start from a empty folder but now we have problem below

1:43PM INF Starting PubSub service impl=PubSub module=pubsub 1:43PM INF Starting IndexerService service impl=IndexerService module=txindex 1:43PM INF ABCI Handshake App Info hash="��t��y��8-�p�TA4ͽg\x18�E����`hq�\u007f�" height=4871535 module=consensus protocol-version=0 software-version=0.5.7 1:43PM INF ABCI Replay Blocks appHeight=4871535 module=consensus stateHeight=4871535 storeHeight=4871536 1:43PM INF Replay last block using real app module=consensus Error: error during handshake: error on replay: wrong Block.Header.AppHash. Expected AFB074B4BB79A9C6382DB670C8544134CDBD6718EF45ABCAEBDE6068718E7FF1, got 505ABB49E09EDE0DF0C8E45402ED845ACFA55563EB08F8F7B90F40C10E1C94FC

See: cosmos/cosmos-sdk#10281