Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv: log errors loudly when stuck in NotLeaseHolderErrors backoff #83830

Closed
nvanbenschoten opened this issue Jul 5, 2022 · 1 comment
Closed

kv: log errors loudly when stuck in NotLeaseHolderErrors backoff #83830

nvanbenschoten opened this issue Jul 5, 2022 · 1 comment
Labels
A-kv-client Relating to the KV client and the KV interface. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. no-issue-activity T-kv KV Team X-stale

Comments

@nvanbenschoten
Copy link
Member

nvanbenschoten commented Jul 5, 2022 •
edited by cockroach-jira-scripts
Loading

Related to #83687.

When a request appears to be stuck in a NotLeaseHolderErrors loop (i.e. two replicas redirecting to each other), we should start loudly logging (log.Warning) the errors that the DistSender is seeing. This avoids the need to drop into logspy like we had to in order to understand #83687. The full NotLeaseHolderErrors are important, as they point at the problematic range and at the replicas in that range that are incoherent. This allows an operator to dig further.

Jira issue: CRDB-17288
@nvanbenschoten nvanbenschoten added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-client Relating to the KV client and the KV interface. E-quick-win Likely to be a quick win for someone experienced. labels Jul 5, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Jul 5, 2022
@github-actions GitHub Actions
Copy link

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 8, 2024
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-client Relating to the KV client and the KV interface. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. no-issue-activity T-kv KV Team X-stale
Projects
No open projects
KV
Status: Closed
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nvanbenschoten