Implement finalization based fork choice rule

[frolosofsky]

This PR introduces commits based initial full sync (UIP-21) and the new longest justified chain fork choice rule (UIP-12). These changes are interconnected and cannot be separated onto two PRs.

The good entry point to this PR is finalization/cache.h as it describes the updated workflow of finalization states.

Full sync. The entry point of the changes is net_processing.cpp/SendMessages, where initial getheaders is replaced by getcommits. Now, when a node gets connected with a peer, its initiates commits-based message exchange where we fetch commits dynasty by dynasty and restore finalization state of a remote chain.
More details are in UIP-21.

Fork choice. Added one level higher check when node decides which chain to choose. So, the first cadidate to switch is a longest justified chain, if node doesn't know such candidate, fallback to bitcoin rules. The entry point of changes is validation.cpp/FindMostWorkChain.
More details are in UIP-12

Collected various TODOs from this PR we have to address: #555.

Update

This PR has been reworked a lot, and in fact, most of the current changes are by @kostyantyn who added a bunch of functional tests and reworked the way we apply the rule.

Finalization repository and processor have been extracted in a separate PR #634 and merged.

Also, I completely removed commits full sync from this PR. One the one hand, it reduces LOC in this PR by ~1k. On the other, we don't switch to the longest justified chain which has less work than the current active chain because of bitcoin simply rejects (and don't request) such chains. It has been fixed by the commits-full-sync, so I will return such functionality as well as functional test in the upcoming PR.

One of the worst scenarios we've fixed in this PR was an undoing of the finalization during reorgs. We implemented two guards to defend against this scenario. We launched all the tests for every guard independently, and they seem to work. Finally, we enabled both of them.

The first guard is on fork-choice level,

• https://github.com/dtr-org/unit-e/pull/525/files#diff-499d7ee7998a27095063ed7b4dd7c119R242
• https://github.com/dtr-org/unit-e/pull/525/files#diff-24efdb00bfbe56b140fb006b562cc70bR73

The second one is on the finalization state processing level:

• https://github.com/dtr-org/unit-e/pull/525/files#diff-d123cf04bac46e7f3d40761315ea2ebdR121
• https://github.com/dtr-org/unit-e/pull/525/files#diff-6c43b7bcd15121e2ff6720e71ce43a6aR129

Fixes #267.

Signed-off-by: Stanislav Frolov [email protected]

[Gnappuraz]

Just to be sure cause I want to review it well before it gets merged, sorry about that.

[Gnappuraz]

Besides the comments below I have an other observation that is the fact that finalizationstate is becoming too big and handling too many things which makes it extremely hard to read (it took me more then 30mins to understand the whole processing and I'm quite accustomed to the class). We need a way to split it out, and keep only the casper stuff inside, everything else has to go, extracted somewhere else.

[frolosofsky]

Besides the comments below I have an other observation that is the fact that finalizationstate is becoming too big and handling too many things which makes it extremely hard to read (it took me more then 30mins to understand the whole processing and I'm quite accustomed to the class). We need a way to split it out, and keep only the casper stuff inside, everything else has to go, extracted somewhere else.

Fair enough. Will move cache implementation including corresponding processing of a new tip to separate file.

[Gnappuraz]

Fair enough. Will move cache implementation including corresponding processing of a new tip to separate file.

Thanks, this will help greatly, I really had hard time reviewing this. It has become too messy. I find largely easier to review 1.2k loc that are well organized then 900 messy ones :)

[scravy]

For me, not being so deeply engaged in the finalization implementation, it is very hard to review this. I would appreciate the use of less auto to more easily follow what is what and I would appreciate more comments on some vital things (what does FRESH or FRESH_BLOCK mean? These things are not obvious from just looking at it).

[scravy]

Stick to code style as agreed and accepted in developer notes.

what I complained about has been fixed

Releasing that I'd like to review it before merging

[Gnappuraz]

utACK b99f3ec

[AM5800]

ACK 300f334
I compiled, tested and even debugged some parts of this PR on my machine.

Overall looking at how this PR progressed I can say - great work guys!!