fix(state_sync_massive): Epoch length was too short

[mfornet]

Epoch length was too short (20) and observer when tried
to sync, used old epoch, but right now, trying to sync from
any epoch other than current epoch or previous epoch is considered
malicious behavior.

Fixes #3130

Test plan

state_sync_massive.py passes in nightly

[gitpod-io]

[mfornet]

Waiting for nightly test to pass:
http://nayduck.eastus.cloudapp.azure.com:3000/#/test/33015

[bowenwang1996]

@mfornet looks like it failed

[SkidanovAlex]

@mfornet you also do not trigger state sync right now I think, I believe it needs to be at least two epochs in today for the observer to actually trigger the state sync (so you probably need to change 101 and 201 to 201 and 301 correspondingly).
I would assert using the log tracker that the state sync was indeed triggered, the same way that we do in state_sync.py

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@f85e142). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #3208   +/-   ##
=========================================
  Coverage          ?   87.71%           
=========================================
  Files             ?      217           
  Lines             ?    44113           
  Branches          ?        0           
=========================================
  Hits              ?    38695           
  Misses            ?     5418           
  Partials          ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f85e142...367d38e. Read the comment docs.

[SkidanovAlex]

Given it's all green to be pushed, requesting changes until it actually passes in nayduck to prevent accidental merge-in.

[mfornet]

This test is performing really poorly on nayduck while passing locally. I've need to adjust few times the size of the epoch, since the protocol relies in the assumption that the node will be able to sync in less than one epoch.

node0 logs
node1 logs

Most notably on nayduck, nodes produced 115 blocks in 20 minutes which is a sign that something is going really wrong. I see in logs from node0 that there are a lot of messages being dropped (which is very weird in this setup with only 2 nodes). However this should not be an issue anymore after #3088 lands.

What puzzles me is that it works every single time locally while failing on nayduck. Probably the reason is a combination of (low performant machine + small epoch length + large state). Notice that in the run above, the node2 didn't even started, so the issue is not with the new node syncing.

@SkidanovAlex @bowenwang1996 are you aware of any protocol mechanism that heavily depends on the epoch length being large enough, otherwise defaulting to significant slowdown?

[bowenwang1996]

@mfornet can you disable storage check in the test? Looks like it is causing quite a bit of slowdown while the node is running

[mfornet]

@mfornet can you disable storage check in the test? Looks like it is causing quite a bit of slowdown while the node is running

It passed nayduck after removing storage_check 💯
http://nayduck.eastus.cloudapp.azure.com:3000/#/test/33017

@SkidanovAlex we can unblock and merge this PR already

[bowenwang1996]

It passed nayduck after removing storage_check 💯

Epic 🚀

[SkidanovAlex]

Can we add the assert that the node actually state_synced (not block synced)? Like at the end of state_sync.py:

assert tracker.check("transition to State Sync")

[mfornet]

Can we add the assert that the node actually state_synced (not block synced)? Like at the end of state_sync.py:

assert tracker.check("transition to State Sync")

http://nayduck.eastus.cloudapp.azure.com:3000/#/run/401