Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kvserver: log when raft send/recv queue fills up #86645

Merged
merged 1 commit into from
Aug 25, 2022

Conversation

tbg
Copy link
Member

@tbg tbg commented Aug 23, 2022

Inspired by https://github.com/cockroachlabs/support/issues/1770.

If either the raft send or receive queue fills up, wide-spread outages
can occur as replication progress stalls. We have metrics that can
indicate this, but straightforward logging is also appropriate to direct
attention to the fact, which this commit achieves.

Touches #79755

Release justification: important logging improvement
Release note: None

Inspired by cockroachlabs/support#1770.

If either the raft send or receive queue fills up, wide-spread outages
can occur as replication progress stalls. We have metrics that can
indicate this, but straightforward logging is also appropriate to direct
attention to the fact, which this commit achieves.

Touches cockroachdb#79755

Release justification: important logging improvement
Release note: None
@tbg tbg requested a review from a team as a code owner August 23, 2022 09:37
@tbg tbg requested a review from pav-kv August 23, 2022 09:37
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@tbg tbg requested a review from a team August 23, 2022 09:54
@tbg
Copy link
Member Author

tbg commented Aug 24, 2022

TFTRs!
bors r=erikgrinaker

@craig
Copy link
Contributor

craig bot commented Aug 24, 2022

Build failed (retrying...):

@craig
Copy link
Contributor

craig bot commented Aug 24, 2022

Build failed (retrying...):

@craig
Copy link
Contributor

craig bot commented Aug 24, 2022

Build failed:

@tbg
Copy link
Member Author

tbg commented Aug 25, 2022

bors r=pavelkalinnikov

@craig
Copy link
Contributor

craig bot commented Aug 25, 2022

Build succeeded:

@craig craig bot merged commit 54bc65f into cockroachdb:master Aug 25, 2022
@tbg
Copy link
Member Author

tbg commented Sep 20, 2022

blathers backport 21.2

@tbg tbg deleted the log-raft-queue-full branch September 20, 2022 15:20
@blathers-crl
Copy link

blathers-crl bot commented Sep 20, 2022

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from c1cd955 to blathers/backport-release-21.2-86645: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 21.2 failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@tbg
Copy link
Member Author

tbg commented Sep 20, 2022

blathers backport 22.1

@blathers-crl
Copy link

blathers-crl bot commented Sep 20, 2022

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from c1cd955 to blathers/backport-release-22.1-86645: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.1 failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants