Hypothesis: A slow follower doesn't cause performance issues

[ChrisKujawa]

Hypothesis

We expect that running a three node cluster with a slow follower should not fail nor perform worse then a normal running cluster.

Context:

An interesting paper on fail-slow fault tolerance. https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-yoo.pdf Their measurements show that popular raft based systems (mongodb, TiDB, RethinkDB) cannot tolerate a slow follower. A slow follower in 3-replica setup can result upto 17-41% decrease in throughput.
Has anybody evaluated how zeebe reacts to a slow follower? I was under the assumption that raft can tolerate a slow follower as the quorum is still fast.

See related slack thread

https://camunda.slack.com/archives/CSQ2E3BT4/p1622013774075600

@deepthidevaki wants to do an experiment which covers this as part of a chaos day, scheduled for 15.06.2021.

[ChrisKujawa]

Was part of a chaos day https://zeebe-io.github.io/zeebe-chaos/chaos_experiment/broker/performance/follower/2021/06/18/Slow-Followers.html

Would be interesting that we run it after building state on followers.