Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: reproposals generated after range creation #33466

Closed
andreimatei opened this issue Jan 3, 2019 · 2 comments
Closed

storage: reproposals generated after range creation #33466

andreimatei opened this issue Jan 3, 2019 · 2 comments
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. no-issue-activity X-stale

Comments

@andreimatei
Copy link
Contributor

andreimatei commented Jan 3, 2019

We've discovered in #33453 that reproposals are quite likely after a range is created because the first message that pops out of Raft is an empty message indicating a leadership change, and that causes every other pending proposal to be re-proposed. Some of these reproposals are unnecessary, as the proposals would have made it to the right leader just fine.
We've speculated that this might be a significant slowdown for restores, which propose large commands immediately after ranges are created.
We should find a way to improve things - perhaps not re-proposing too eagerly or not re-proposing large messages when the reproposal reason is "new leader".

cc @bdarnell in case you've got an opinion

Jira issue: CRDB-4687

@andreimatei andreimatei added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. labels Jan 3, 2019
@bdarnell
Copy link
Contributor

bdarnell commented Jan 4, 2019

We need to be careful with this. A new range goes through a couple of transitions. Initially, it has no leader (raft.lead = None). During this state it simply drops any proposals on the floor; these proposals must be reproposed once the new leader is established (that's why this reproposal trigger exists). After the leader election, raft.lead != None and the new leader proposes its empty entry. Commands submitted during this phase are the ones you're concerned about here - they will be queued up behind the empty entry and will almost certainly commit without being reproposed.

To distinguish the two cases, we need ErrProposalDropped: #21849 A proposal that returns ErrProposalDropped should be retried on the next reproposal trigger. A proposal that does not experience this error can be retried less aggressively (I'm not sure at the moment exactly what rules should apply)

@lunevalex lunevalex added the A-kv-replication Relating to Raft, consensus, and coordination. label Jul 27, 2020
@jlinder jlinder added the T-kv KV Team label Jun 16, 2021
@erikgrinaker erikgrinaker added T-kv-replication and removed T-kv KV Team labels May 31, 2022
@tbg tbg removed their assignment May 31, 2022
Copy link

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. no-issue-activity X-stale
Projects
None yet
Development

No branches or pull requests

6 participants