Skip to content

Commit

Permalink
kvserver: apply kv.raft.command.max_size to outgoing raft msg batching
Browse files Browse the repository at this point in the history
In #71050, we saw evidence of very large (2.3+GiB) Raft messages being
sent on the stream, which overwhelmed both the sender and the receiver.
Raft messages are batched up before sending and so what must have
happened here is that a large number of reasonably-sized messages (up to
64MiB in this case due to the max_size setting) were merged into a giant
blob. As of this commit, we apply the max_size chunking on the batching
step before sending messages as well.

Closes #71050.

Release note: None
  • Loading branch information
tbg committed Oct 5, 2021
1 parent 782bc96 commit 9ccf33e
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions pkg/kv/kvserver/raft_transport.go
Original file line number Diff line number Diff line change
Expand Up @@ -487,15 +487,18 @@ func (t *RaftTransport) processQueue(
case err := <-errCh:
return err
case req := <-ch:
maxBytes := MaxCommandSize.Get(&t.st.SV)
batch.Requests = append(batch.Requests, *req)
req.release()
// Pull off as many queued requests as possible.
//
// TODO(peter): Think about limiting the size of the batch we send.
// Pull off as many queued requests as possible, within reason.
for done := false; !done; {
select {
case req = <-ch:
batch.Requests = append(batch.Requests, *req)
maxBytes -= int64(req.Size())
if maxBytes <= 0 {
done = true
}
req.release()
default:
done = true
Expand Down

0 comments on commit 9ccf33e

Please sign in to comment.