devnet - panic: runtime error

[sophoah]

Describe the bug

panic: runtime error at node start.

harmony.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
harmony.service: Failed with result 'exit-code'.
harmony.service: Scheduled restart job, restart counter is at 74.
Stopped harmony service.
Started harmony service.
Staking mode; node key 94e951bf8eda09aefcbc00a48932dfc4568d1f9e8973c81086e4f3152201bd38622d43041c67c0b470b98d2231ef6d98;e987759a67b47cb746efcdf4a2546d1d51221f135dfffaae8b2dad75e3f0378e5163cd1fb57cb06eb7c33610828fe500;01e990feaca2ab2dc09d272d8c52758267c8004b15a324c45b1bb0c16dbef23a0dcc4df03854f532141f522a8b6f360c; -> shard 0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1264690]
goroutine 1 [running]:
github.com/harmony-one/harmony/core.(*StateProcessor).Process(_, _, _, {0x0, {0x0, 0x0}, 0x0, 0x0, {{0x0, 0x0, ...}, ...}, ...}, ...)
        /root/go/src/github.com/harmony-one/harmony/core/state_processor.go:111 +0xd0
github.com/harmony-one/harmony/core.(*BlockChainImpl).insertChain(0xc0001c8000, {0xc0009d1548?, 0x1, 0x1}, 0x1)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1835 +0xcb9
github.com/harmony-one/harmony/core.(*BlockChainImpl).repairRecreateStateTries(0xc0001c8000?, 0xc0009d19d8)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:789 +0x7b
github.com/harmony-one/harmony/core.(*BlockChainImpl).repair(0xc0007b0c60?, 0x29ca02d?)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:809 +0xa6
github.com/harmony-one/harmony/core.(*BlockChainImpl).loadLastState(0xc0001c8000)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:598 +0x120b
github.com/harmony-one/harmony/core.newBlockChainWithOptions({_, _}, {_, _}, {_, _}, _, _, {0x2f14538, 0xc000130790}, ...)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:327 +0x8dd
github.com/harmony-one/harmony/core.NewBlockChainWithOptions(...)
        /root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:233
github.com/harmony-one/harmony/internal/shardchain.(*CollectionImpl).ShardChain(0xc000135b00, 0x0, {0x0, 0x0, 0x0?})
        /root/go/src/github.com/harmony-one/harmony/internal/shardchain/shardchains.go:143 +0x5c5
main.setupChain({{0xc00039d460, 0x5}, {{0xc00039d9e0, 0x9}, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
        /root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:773 +0x807
main.setupConsensusAndNode({{0xc00039d460, 0x5}, {{0xc00039d9e0, 0x9}, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
        /root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:812 +0x347
main.setupNodeAndRun({{0xc00039d460, 0x5}, {{0xc00039d9e0, 0x9}, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...})
        /root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:338 +0x4b4
main.runHarmonyNode(0x410eda0?, {0x2816684?, 0x2?, 0x2?})
        /root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:151 +0x1b5
github.com/spf13/cobra.(*Command).execute(0x410eda0, {0xc000052190, 0x2, 0x2})
        /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0x410eda0)
        /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main()
        /root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:125 +0x25
harmony.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
harmony.service: Failed with result 'exit-code'.

To Reproduce
Steps to reproduce the behavior:

1. run harmony latest dev commit 17a2522 on a shard 0 validator (s1 explorer and s1 validator didn't face the issue)
2. force crashed the node with kill - 9 <harmony_process>

Expected behavior
no panic

Screenshots
If applicable, add screenshots to help explain your problem.

node logs keep looping on:

{"level":"info","self":"0.0.0.0:9000","PeerID":"QmVAHbuww5WG2YiRCaryzJNhdC4arm6V2qKbft9wR4khvZ","PubKey":"94e951bf8eda09aefcbc00a48932dfc4568d1f9e8973c81086e4f3152201bd38622d43041c67c0b470b98d2231ef6d98","caller":"/root/go/src/github.com/harmony-one/harmony/p2p/host.go:279","time":"2024-01-22T08:00:57.644542351+01:00","message":"libp2p host ready"}
{"level":"warn","caller":"/root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:801","time":"2024-01-22T08:00:57.64518752+01:00","message":"Blacklist setup error: open ./.hmy/blacklist.txt: no such file or directory"}
{"level":"warn","caller":"/root/go/src/github.com/harmony-one/harmony/cmd/harmony/main.go:805","time":"2024-01-22T08:00:57.645223737+01:00","message":"AllowedTxs setup error: open ./.hmy/allowedtxs.txt: no such file or directory"}
{"level":"warn","number":"155207","hash":"0x8d6a97214852626b36966ef88e7ff21ef4189830cba849d2e5b5f5f4c680dc43","caller":"/root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:597","time":"2024-01-22T08:00:57.665722669+01:00","message":"Head state missing, repairing chain"}
{"level":"info","number":"154814","hash":"0xdb9be169d90b4a0592a217865f22d8df58751ea4d851e9c365ada0e8ff054d5d","caller":"/root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:823","time":"2024-01-22T08:00:57.69244951+01:00","message":"Rewound blockchain to past state"}

Environment (please complete the following information):

• OS: [Linux, MacOS]
• Go environment [ go env ]

Additional context
Add any other context about the problem here.