Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

checker: fix the issue that unhealthy region cause panic #5494

Merged
merged 3 commits into from
Sep 13, 2022
Merged

checker: fix the issue that unhealthy region cause panic #5494

merged 3 commits into from
Sep 13, 2022

Conversation

nolouch
Copy link
Contributor

@nolouch nolouch commented Sep 13, 2022 •
edited
Loading

Signed-off-by: nolouch [email protected]

What problem does this PR solve?

Issue Number: Close #5491

What is changed and how does it work?

checker: fix the issue that unhealthy region cause panic

Check List

Tests

  • Unit test

Release note

Fix the issue that unhealthy region cause panic

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 13, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • CabinfeverB
  • disksing

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 13, 2022
@nolouch nolouch requested review from CabinfeverB and removed request for disksing September 13, 2022 08:33
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. and removed do-not-merge/needs-triage-completed labels Sep 13, 2022
@codecov
Copy link

codecov bot commented Sep 13, 2022 •
edited
Loading

Codecov Report

Base: 75.55% // Head: 75.50% // Decreases project coverage by -0.05% ⚠️

Coverage data is based on head (b7297ef) compared to base (772c480).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5494      +/-   ##
==========================================
- Coverage   75.55%   75.50%   -0.06%     
==========================================
  Files         320      320              
  Lines       31642    31650       +8     
==========================================
- Hits        23907    23897      -10     
- Misses       5670     5690      +20     
+ Partials     2065     2063       -2
Flag Coverage Δ
unittests 75.50% <100.00%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/mock/mockcluster/mockcluster.go 94.17% <100.00%> (+0.05%) ⬆️
server/schedule/checker/rule_checker.go 81.25% <100.00%> (+1.34%) ⬆️
pkg/dashboard/keyvisual/input/core.go 0.00% <0.00%> (-33.34%) ⬇️
pkg/dashboard/adapter/manager.go 79.31% <0.00%> (-9.20%) ⬇️
server/storage/kv/etcd_kv.go 79.71% <0.00%> (-5.80%) ⬇️
server/storage/storage.go 67.30% <0.00%> (-3.85%) ⬇️
server/tso/tso.go 68.53% <0.00%> (-3.38%) ⬇️
server/tso/allocator_manager.go 62.66% <0.00%> (-2.60%) ⬇️
server/storage/endpoint/meta.go 61.36% <0.00%> (-2.28%) ⬇️
server/region_syncer/client.go 84.32% <0.00%> (-2.24%) ⬇️
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

CabinfeverB
CabinfeverB reviewed Sep 13, 2022
View reviewed changes
server/schedule/checker/rule_checker.go Outdated
@@ -236,10 +236,14 @@ func (c *RuleChecker) fixLooseMatchPeer(region *core.RegionInfo, fit *placement.
checkerCounter.WithLabelValues("rule_checker", "fix-peer-role").Inc()
return operator.CreatePromoteLearnerOperator("fix-peer-role", c.cluster, region, peer)
}
if region.GetLeader() == nil {
checkerCounter.WithLabelValues("rule_checker", "region-no-leader").Inc()
return nil, nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can return an error, so L158 can return

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree

Copy link
Member

@rleungx rleungx Sep 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the region doesn't have a leader, is it necessary to check it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rleungx I think no need, address it.

@nolouch
address comment
90bd77d 
Signed-off-by: nolouch <[email protected]>
@nolouch
Copy link
Contributor Author

nolouch commented Sep 13, 2022

ptal @HunDunDM @CabinfeverB

@nolouch
address comment
b7297ef 
Signed-off-by: nolouch <[email protected]>
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 13, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 13, 2022
@rleungx
Copy link
Member

rleungx commented Sep 13, 2022

/merge

@ti-chi-bot
Copy link
Member

@rleungx: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b7297ef

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 13, 2022
@ti-chi-bot ti-chi-bot merged commit 16b374a into tikv:master Sep 13, 2022
@ti-chi-bot ti-chi-bot mentioned this pull request Sep 13, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5495.

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Sep 13, 2022
@nolouch @ti-chi-bot
This is an automated cherry-pick of tikv#5494
6647e88 
close tikv#5491

Signed-off-by: ti-chi-bot <[email protected]>
@nolouch nolouch deleted the fix-rule-panic branch September 13, 2022 11:40
ti-chi-bot added a commit that referenced this pull request Sep 20, 2022
@ti-chi-bot @nolouch
checker: fix the issue that unhealthy region cause panic (#5494) (#5495)
f11e367 
close #5491, ref #5494

checker: fix the issue that unhealthy region cause panic

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HunDunDM HunDunDM HunDunDM left review comments

@CabinfeverB CabinfeverB CabinfeverB approved these changes

@disksing disksing disksing approved these changes

@rleungx rleungx Awaiting requested review from rleungx

Assignees
No one assigned
Labels
needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

multi-region-dr: pd leader panic repeatly
6 participants
@nolouch @ti-chi-bot @rleungx @disksing @HunDunDM @CabinfeverB