br panic when inject pd io delay during br restore

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、br restore
2、inject pd io delay 1s

2. What did you expect to see? (Required)

br restore success

3. What did you see instead (Required)

br failed with panic
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x280a3e5] goroutine 9160 [running]: github.com/tikv/pd/client.(*client).ScanRegions(0xc00082a000, {0x68a7ea8, 0xc00feaed20}, {0xc00fcb1f60, 0x1b, 0x1b}, {0xc00fcb1f80, 0x1b, 0x1b}, 0x80) /root/go/pkg/mod/github.com/tikv/pd/[email protected]/client.go:1086 +0x625 github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).ScanRegions(0x100000001?, {0x68a7ea8?, 0xc00feaed20?}, {0xc00fcb1f60?, 0x0?, 0x1a3185c5000?}, {0xc00fcb1f80?, 0x0?, 0x0?}, 0x80) /workspace/source/tidb/br/pkg/restore/split/client.go:523 +0x57 github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1() /workspace/source/tidb/br/pkg/restore/split/split.go:122 +0x186 github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1({0x0?, 0x0?}) /workspace/source/tidb/br/pkg/utils/retry.go:217 +0x13 github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]({0x68a7ea8?, 0xc00feaed20}, {0x6884188, 0xc00fcb1fa0}, 0xc00213aba0) /workspace/source/tidb/br/pkg/utils/retry.go:235 +0x98 github.com/pingcap/tidb/br/pkg/utils.WithRetry({0x68a7ea8?, 0xc00feaed20?}, 0x0?, {0x6884188?, 0xc00fcb1fa0?}) /workspace/source/tidb/br/pkg/utils/retry.go:216 +0x50 github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion({0x68a7ea8?, 0xc00feaed20?}, {0x68cfe00?, 0xc002131100}, {0xc00fcb1f60, 0x1b, 0x1b}, {0xc00fcb1f80, 0x1b, 0x1b}, ...) /workspace/source/tidb/br/pkg/restore/split/split.go:117 +0x2ab github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1() /workspace/source/tidb/br/pkg/restore/import.go:548 +0x179 github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1({0x57034a0?, 0xc00fead1d0?}) /workspace/source/tidb/br/pkg/utils/retry.go:217 +0x13 github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]({0x68a7e38?, 0xc00a08e0a0}, {0x687da68, 0xc00fce5b40}, 0xc00213ba38) /workspace/source/tidb/br/pkg/utils/retry.go:235 +0x98 github.com/pingcap/tidb/br/pkg/utils.WithRetry({0x68a7e38?, 0xc00a08e0a0?}, 0x1?, {0x687da68?, 0xc00fce5b40?}) /workspace/source/tidb/br/pkg/utils/retry.go:216 +0x50 github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles(0xc001f44020, {0x68a7e38?, 0xc00a08e0a0}, {0xc008fcf9e0?, 0x1, 0xc4}, 0xc004b56910, 0xc000ea4bb0, 0x0) /workspace/source/tidb/br/pkg/restore/import.go:544 +0x445 github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1(0xc009aecef0, {0x68a7cb0?, 0xc004e44fc0}, 0xc001f44000, {0x68a7e38, 0xc00a08e0a0}, 0x0?, {0xc008fcf9e0, 0x1, 0xc4}) /workspace/source/tidb/br/pkg/restore/client.go:1458 +0x17c github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2() /workspace/source/tidb/br/pkg/restore/client.go:1459 +0x1bd github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1() /workspace/source/tidb/br/pkg/utils/worker.go:76 +0x6c golang.org/x/sync/errgroup.(*Group).Go.func1() /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2632 /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x96 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x280a3e5] goroutine 9323 [running]: github.com/tikv/pd/client.(*client).ScanRegions(0xc00082a000, {0x68a7ea8, 0xc00f38aa10}, {0xc00f3f42a0, 0x1b, 0x1b}, {0xc00f3f42c0, 0x1b, 0x1b}, 0x80) /root/go/pkg/mod/github.com/tikv/pd/[email protected]/client.go:1086 +0x625 github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).ScanRegions(0x0?, {0x68a7ea8?, 0xc00f38aa10?}, {0xc00f3f42a0?, 0x0?, 0x0?}, {0xc00f3f42c0?, 0x0?, 0x0?}, 0x80) /workspace/source/tidb/br/pkg/restore/split/client.go:523 +0x57 github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1() /workspace/source/tidb/br/pkg/restore/split/split.go:122 +0x186 github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1({0x0?, 0x0?}) /workspace/source/tidb/br/pkg/utils/retry.go:217 +0x13 github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]({0x68a7ea8?, 0xc00f38aa10}, {0x6884188, 0xc00f3f42e0}, 0xc006524ba0) /workspace/source/tidb/br/pkg/utils/retry.go:235 +0x98 github.com/pingcap/tidb/br/pkg/utils.WithRetry({0x68a7ea8?, 0xc00f38aa10?}, 0x0?, {0x6884188?, 0xc00f3f42e0?}) /workspace/source/tidb/br/pkg/utils/retry.go:216 +0x50 github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion({0x68a7ea8?, 0xc00f38aa10?}, {0x68cfe00?, 0xc002131100}, {0xc00f3f42a0, 0x1b, 0x1b}, {0xc00f3f42c0, 0x1b, 0x1b}, ...) /workspace/source/tidb/br/pkg/restore/split/split.go:117 +0x2ab github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1() /workspace/source/tidb/br/pkg/restore/import.go:548 +0x179 github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1({0x1e2554f?, 0x7f629c662e60?}) /workspace/source/tidb/br/pkg/utils/retry.go:217 +0x13 github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]({0x68a7e38?, 0xc009f7a0f0}, {0x687da68, 0xc00f3ecfe0}, 0xc006525a38) /workspace/source/tidb/br/pkg/utils/retry.go:235 +0x98 github.com/pingcap/tidb/br/pkg/utils.WithRetry({0x68a7e38?, 0xc009f7a0f0?}, 0x1?, {0x687da68?, 0xc00f3ecfe0?}) /workspace/source/tidb/br/pkg/utils/retry.go:216 +0x50 github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles(0xc001f44020, {0x68a7e38?, 0xc009f7a0f0}, {0xc006f970d0?, 0x1, 0xe6}, 0xc004b56c30, 0xc000ea4bb0, 0x0) /workspace/source/tidb/br/pkg/restore/import.go:544 +0x445 github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1(0xc00f8beef0, {0x68a7cb0?, 0xc004e44fc0}, 0xc001f44000, {0x68a7e38, 0xc009f7a0f0}, 0x0?, {0xc006f970d0, 0x1, 0xe6}) /workspace/source/tidb/br/pkg/restore/client.go:1458 +0x17c github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2() /workspace/source/tidb/br/pkg/restore/client.go:1459 +0x1bd github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1() /workspace/source/tidb/br/pkg/utils/worker.go:76 +0x6c golang.org/x/sync/errgroup.(*Group).Go.func1() /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5700 /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x96

4. What is your TiDB version? (Required)

tidb version:
./tidb-server -V
Release Version: v7.5.2
Edition: Community
Git Commit Hash: 39ea2b3
Git Branch: HEAD
UTC Build Time: 2024-05-29 15:07:12
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-30T14:14:32.686+0800

[Lily2025]

/type bug
/severity major
/assign 3pointer

[BornChanger]

@Lily2025 please close it since the problem does not affect 8.x, and all cherry-picks are done for down-level lts

[Leavrth]

fix by #53739