feat(chain): Upgradability functionality

[ilblackdragon]

Implemented next functionality:

• Upgradability spec: Initial spec for upgradability NEPs#64
• Adding support for upgradable BlockHeader Fixes Upgradable BlockHeader #2698
• Adding support for upgrading database Fixes Migrate state and configs by default #2498
• Nightly test that checks that network runs with 3 old and 1 new version validators and later transitions to new version as old binaries are restarted. Fixes Nightly test: upgradability #2697

To simplify reviewing, specific places:

• EpochManager computing next version: https://github.com/nearprotocol/nearcore/pull/2701/files#diff-9897fb54fafd41577969ff99367f443fR205
• Using protocol version to define which version of block structures to produce https://github.com/nearprotocol/nearcore/pull/2701/files#diff-55d2aa3a5feb6934292d5ef48562522eR399
• BlockHeader as an example of upgradable structure: https://github.com/nearprotocol/nearcore/pull/2701/files#diff-55eb814e6d41d24a229e31c397c5b96f

Test plan

1. All CI tests pass

Future plan

1. Make backward_compatible.py test as mandatory starting now and they pass.
2. New upgradable.py nightly test passes, after first release of this to beta.
3. Remove state_migration.py test, as it's superseded by upgradable.py

[gitpod-io]

[gitpod-io]

[ilblackdragon]

I've started with trying to make this backward compatible, but mid way realized that because previous version doesn't have upgradability voting on consensus level - even if it's backward compatible there is no consensus way to understand when supermajority should move to next version.

Hence, this update will still require a hard fork.

[gitpod-io]

[bowenwang1996]

Is the Cargo.lock change intended?

[ilblackdragon]

@bowenwang1996 reset Cargo.lock back to master.

[ilblackdragon]

Is this ok for nightly? @bowenwang1996 @SkidanovAlex
https://nightly.neartest.com/s3/upgradability_728a0043e8e3209f06229aadc8cf5f0b848614ea_20200605_162859

Also why are they not sorted the same?

[bowenwang1996]

@ilblackdragon I cannot open that link.

[ilblackdragon]

@bowenwang1996 Opens fine for me - just wait 30-40 seconds.

[SkidanovAlex]

It is unlikely that we will change the Block or BlockHeader, changes will likely go via ShardChunkHeader and BlockHeaderInner* (and couple other structs).

What is the plan once they change? If we change ShardChunkHeader at the moment when the current Block was BlockV5, is the idea to create a new struct ShardChunkHeaderV5 that will be included in BlockV5, and have ShardChunkHeader have the new structure? It will be pretty ugly in the code, given that ShardChunkHeader on itself is not versioned, so it's not an enum we can match in the places that receive ShardChunkHeader.

[ilblackdragon]

The versioning is done on the level of protocol messages, e.g. what is sent around in the network in PeerMessage (https://github.com/nearprotocol/nearcore/blob/4a6fbd718c856b5d37b848df1b124eff9703d83f/chain/network/src/types.rs#L401). ShardChunkHeader is not part of it.

[MaksymZavershynskyi]

LGTM for genesis tools.