You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the realtime replication queue is in memory, and of a limited fix size. Change this to use the overflow queue, to allow for:
larger backlogs;
backlogs to persist across reboots.
The reason why originally this was simply an in-memory queue, was that full-sync will always fill the gaps. Although full-sync is now cheaper than key-listing, especially when gaps are small, it is still a slow operation.
If there are o(100K) or greater gaps, full-sync is going to be more time consuming and expensive than draining from a queue, if such a queue exists.
In implementing this, the queue should persist across backlogs but start empty. There should be an admin command to empty a previous real-time queue on the node, so that replaying such a persisted backlog is an operator choice.
The text was updated successfully, but these errors were encountered:
The
riak_kv_overflow_queue
has been added as part of this enhancement to the reaper/eraser queues.Currently the realtime replication queue is in memory, and of a limited fix size. Change this to use the overflow queue, to allow for:
The reason why originally this was simply an in-memory queue, was that full-sync will always fill the gaps. Although full-sync is now cheaper than key-listing, especially when gaps are small, it is still a slow operation.
If there are o(100K) or greater gaps, full-sync is going to be more time consuming and expensive than draining from a queue, if such a queue exists.
In implementing this, the queue should persist across backlogs but start empty. There should be an admin command to empty a previous real-time queue on the node, so that replaying such a persisted backlog is an operator choice.
The text was updated successfully, but these errors were encountered: