Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi-region-dr: pd leader panic repeatly #5491

Closed
mayjiang0203 opened this issue Sep 13, 2022 · 2 comments · Fixed by #5494
Closed

multi-region-dr: pd leader panic repeatly #5491

mayjiang0203 opened this issue Sep 13, 2022 · 2 comments · Fixed by #5494
Assignees
@nolouch
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@mayjiang0203
Copy link

mayjiang0203 commented Sep 13, 2022 •
edited
Loading

Bug Report

What did you do?

What did you expect to see?

What did you see instead?

image

[2022/09/11 13:18:38.927 +00:00] [INFO] [operator_controller.go:450] ["add operator"] [region-id=86007] [operator="\"rule-split-region {split: region 86007 use policy USEKEY and keys [7480000000000000FF5800000000000000F8]} (kind:split, region:86007(61, 39), createAt:2022-09-11 13:18:38.92786507 +0000 UTC m=+116.187335222, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[split region with policy USEKEY])\""] [additional-info="{\"region-end-key\":\"7480000000000000FF5900000000000000F8\",\"region-start-key\":\"7480000000000000FF5700000000000000F8\"}"]
[2022/09/11 13:18:38.928 +00:00] [INFO] [operator_controller.go:652] ["send schedule command"] [region-id=86007] [step="split region with policy USEKEY"] [source=create]
[2022/09/11 13:18:38.928 +00:00] [FATAL] [log.go:72] [panic] [recover="\"invalid memory address or nil pointer dereference\""] [stack="github.com/tikv/pd/pkg/logutil.LogPanic\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/logutil/log.go:72\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:220\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:818\ngithub.com/tikv/pd/server/schedule/checker.(*RuleChecker).fixLooseMatchPeer\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/checker/rule_checker.go:242\ngithub.com/tikv/pd/server/schedule/checker.(*RuleChecker).fixRulePeer\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/checker/rule_checker.go:156\ngithub.com/tikv/pd/server/schedule/checker.(*RuleChecker).CheckWithFit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/checker/rule_checker.go:118\ngithub.com/tikv/pd/server/schedule/checker.(*Controller).CheckRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/checker/checker_controller.go:96\ngithub.com/tikv/pd/server/cluster.(*coordinator).patrolRegions\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:149"]

What version of PD are you using (pd-server -V)?

/ # /pd-server -V
Release Version: v6.1.1
Edition: Community
Git Commit Hash: 4ab9c0e
Git Branch: heads/refs/tags/v6.1.1
UTC Build Time: 2022-08-23 08:25:18
@mayjiang0203 mayjiang0203 added the type/bug The issue is confirmed as a bug. label Sep 13, 2022
@nolouch
Copy link
Contributor

nolouch commented Sep 13, 2022

some region without leader will cause this issue.

@nolouch
Copy link
Contributor

nolouch commented Sep 13, 2022

/assign @nolouch

@nolouch nolouch mentioned this issue Sep 13, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Sep 13, 2022
@nolouch
checker: fix the issue that unhealthy region cause panic (#5494)
16b374a 
close #5491

checker: fix the issue that unhealthy region cause panic

Signed-off-by: nolouch <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Sep 13, 2022
Merged
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Sep 13, 2022
@nolouch @ti-chi-bot
This is an automated cherry-pick of tikv#5494
6647e88 
close tikv#5491

Signed-off-by: ti-chi-bot <[email protected]>
ti-chi-bot added a commit that referenced this issue Sep 20, 2022
@ti-chi-bot @nolouch
checker: fix the issue that unhealthy region cause panic (#5494) (#5495)
f11e367 
close #5491, ref #5494

checker: fix the issue that unhealthy region cause panic

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nolouch nolouch

Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

checker: fix the issue that unhealthy region cause panic nolouch/pd
4 participants
@nolouch @VelocityLight @mayjiang0203 @ti-chi-bot