Block rpc handling until state store is caught up #5911
Conversation
Here, we ensure that when leader only responds to RPC calls when state store is up to date. At leadership transition or launch with restored state, the server local store might not be caught up with latest raft logs and may return a stale read. The solution here is to have an RPC consistency read gate, enabled when `establishLeadership` completes before we respond to RPC calls. `establishLeadership` is gated by a `raft.Barrier` which ensures that all prior raft logs have been applied. Conversely, the gate is disabled when leadership is lost. This is very much inspired by https://github.com/hashicorp/consul/pull/3154/files
| @@ -289,6 +289,8 @@ func (s *Server) establishLeadership(stopCh chan struct{}) error { | |||
| return err | |||
| } | |||
|
|
|||
| s.setConsistentReadReady() | |||
There was a problem hiding this comment.
Is there a reason we need to wait till the end of this method? This method is called after the barrier write, so any reason not to toggle this at the beginning of this method rather than wait for all the other housekeeping that happens in this method?
There was a problem hiding this comment.
I was being defensive for fear of an RPC handling accessing/mutating brokers before they are enabled. I'd hope that establishing leadership is a quick call and happens rarely such that optimizing to find exact ideal place to set consistent readiness isn't worth it. What do you think?
There was a problem hiding this comment.
Yeah agreed that its a minor optimization, lets leave it as is
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Here, we ensure that when leader only responds to RPC calls when state
store is up to date. At leadership transition or launch with restored
state, the server local store might not be caught up with latest raft
logs and may return a stale read.
The solution here is to have an RPC consistency read gate, enabled when
establishLeadershipcompletes before we respond to RPC calls.establishLeadershipis gated by araft.Barrierwhich ensures thatall prior raft logs have been applied.
Conversely, the gate is disabled when leadership is lost.
This fixes the underlying issue in #5906 .
This is very much inspired by https://github.com/hashicorp/consul/pull/3154/files