Node should not corrupt its state on exit

[frol]

Describe the bug

We received 3 reports today that our partner Indexer nodes got corrupted after node was aborted with

nearcore/chain/client/src/client.rs

Lines 1032 to 1034 in 0c9ad79

	if next_epoch_protocol_version > PROTOCOL_VERSION {
	panic!("The client protocol version is older than the protocol version of the network. Please update nearcore");
	}

After they recompiled the node with the newer nearcore, they could not boot the node back again:

thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', /home/ubuntu/.cargo/git/checkouts/nearcore-e3f14b9758bedfa0/fb1e621/chain/epoch_manager/src/lib.rs:1193:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'The lock was poisoned.: PoisonError { .. }', /home/ubuntu/.cargo/git/checkouts/nearcore-e3f14b9758bedfa0/fb1e621/nearcore/src/runtime/mod.rs:993:69
thread '<unnamed>' panicked at 'The lock was poisoned.: PoisonError { .. }', /home/ubuntu/.cargo/git/checkouts/nearcore-e3f14b9758bedfa0/fb1e621/nearcore/src/runtime/mod.rs:1042:69
thread '<unnamed>' panicked at 'The lock was poisoned.: PoisonError { .. }', /home/ubuntu/.cargo/git/checkouts/nearcore-e3f14b9758bedfa0/fb1e621/nearcore/src/runtime/mod.rs:1042:69
thread '<unnamed>' panicked at 'The lock was poisoned.: PoisonError { .. }', /home/ubuntu/.cargo/git/checkouts/nearcore-e3f14b9758bedfa0/fb1e621/nearcore/src/runtime/mod.rs:1042:69
thread 'main' panicked at 'assertion failed: queue.send(msg).is_ok()', /home/ubuntu/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/actix-0.11.0-beta.2/src/sync.rs:161:25

To Reproduce

Probably, you need to let the node that does not support newer protocol version to run over the protocol upgrade, so it hits the condition.

Expected behavior

nearcore should not corrupt its database and should be able to boot fine after this abort.

Version (please complete the following information):

• nearcore commit/branch: 4dbc9f8
• testnet

[bowenwang1996]

I believe this is the same issue as #3266. @mina86 please take a look

[mina86]

I believe this is the same issue as #3266.

This is something different. #3266 is about shutting down after getting a signal. Here we have a panic which aborts the process.

[mina86]

Closing in favour of #5340 as it’s the same underlying cause.