[NEO3] Adding a Fallback Consensus

[toghrulmaharram]

Problem

In the current implementation, a block is published as soon as the Speaker p's proposal receives at least n - f [(n - 1)/3] signatures from the Consensus Nodes in a certain timeframe. If not enough signatures are received during the aforementioned timeframe, a View Change v is required to elect a new Speaker via p = h [block index] - v mod n [number of Consensus Nodes] to propose a new block. However, if the number of malicious nodes (both intentionally or unintentionally) exceeds f, but is below 2f + 1, the consensus stalls and has to be manually reset.

Solution

The proposed solution requires a few changes to the current implementation to become functional. Firstly, the NextConsensus attribute of the block header has to be abandoned in favour of a ViewChange attribute which is going to be equal to a size of 1 byte. Secondly, a list of Standby Nodes will have to be elected alongside the Consensus Nodes during the voting procedure (would include the candidates which did not receive enough votes to become a Consensus Node). To avoid the consensus stalls, a fallback consensus will kick in if the v > n, requiring n Standby Nodes to confirm the last available block proposal. In case the proposal receives at least n - f signatures from the Standby Nodes, the Standby Nodes that received the most votes are swapped in for the Consensus Nodes which failed to sign the latest block proposal. The swapped out Consensus Nodes will be demoted to the end of the Standby Node list.

[anthdm]

@toghrulmaharramov I like the idea of a fallback mechanism. However, changing the field to ViewChange is kinda dBFT specific. It would be nice to create a mechanism that could work with all consensus algorithms. Unless its very clear that NEO will always have a BFT like consensus algorithm in place.

[toghrulmaharram]

@anthdm I was writing this with the current implementation in mind. Thanks to the block header versioning, we could easily modify the ViewChange field to another field in case we decide to change the consensus protocol later down the line (for example, your HoneyBadger BFT implementation).

[lerider]

This is in my opinion one of the most important improvements that are being suggested at the moment.

[toghrulmaharram]

@lerider I had discovered an issue with the protocol when writing the formal proof for the fallback that will prevent the fallback consensus from working with the current version of dBFT. I will PM you the details on Discord later today.

[vncoelho]

Hi @toghrulmaharramov, we miss you, my friend!
I hope everything is going well in your new studies.

You should see the new dBFT 2.0. Solid as a rock! :D You are also part of it.

I will leave this closed for now, Tog. I believe that the discussion now might be in another direction.

[igormcoelho]

I think the idea is still valid Vitor, lets keep open.

[vncoelho]

This is not possible nowadays, @igormcoelho:
"However, if the number of malicious nodes (both intentionally or unintentionally) exceeds f, but is below 2f + 1, the consensus stalls and has to be manually reset."
This is a byzantine case. There is no reason to reset. Reset would cause other problems if honest nodes lose their information.

But the solution has other insights, I believe it is better to open a new issue with the specific modification idea, but not related to the problem.

[igormcoelho]

I guess there's a way to implement togs idea as soon as we have the interop for GetConsensusScript I proposed

[vncoelho]

This is a quite complex situation to deal with StandyBy Validators.

I would prefer this proposal here: neo-project/proposals#82

Lightning vote is like moving the voting even to the mempool level if certain conditions happen.
In this sense, NEO holders would be able to modify validators during block creation.

[vncoelho]

@igormcoelho and @toghrulmaharramov, I think that this proposal can be closed.

Minor adjust:

• In the description of the problem " exceeds f, but is below 2f + 1, the consensus stalls and has to be manually reset.".
  If it exceed f there is no solution, reset will not solve the problem, since the f nodes would stay byzantine. There is no reason to reset.

About the fallback consensus:

• the fallback consensus looks like a workaround to me, I insist that we focus on the power of the voting that we currently have and just move it to the mempool layer. @erikzhang @jsolman