Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PiTR restore failed due to restored_ts > log backup range #36246

Closed
fubinzh opened this issue Jul 15, 2022 · 0 comments · Fixed by #36255 or #36305
Closed

PiTR restore failed due to restored_ts > log backup range #36246

fubinzh opened this issue Jul 15, 2022 · 0 comments · Fixed by #36255 or #36305
Labels
component/br This issue is related to BR of TiDB. feature/developing the related feature is in development found/automation Found by automation tests severity/critical type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Jul 15, 2022

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Start K8S cluster PD/TiDB: 4U 8G, TiKV: 8U 16G
  2. Start log backup
  3. Run full backup
  4. Run workload (QPS: insert&update 1K)
  5. Wait checkpoint move forward TS1 and do PiTR restore to TS1

2. What did you expect to see? (Required)

PiTR restore should succeed

3. What did you see instead (Required)

PiTR restore failed

[2022/07/15 15:13:51.431 +08:00] [INFO] [br.go:563] ["waitLogBackupCheckpoint success"] ["current checkpoint"=2022/07/15 07:12:24.000 +00:00]
...
2022-07-15T15:13:51.490+0800    INFO    host/host.go:30 Execute command {"command": " /br  restore  point \"--storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Log?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\" \"--pd\" \"http://dst-tidb-pd.fb-pitr-4u-t6djr:2379/\" \"--restored-ts\" \"434600400949870596\" \"--full-backup-storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Full?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\"", "timeout": "60m", "resource name": "br"}
2022-07-15T15:13:51.493+0800    INFO    k8s/client.go:267       it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
Detail BR log in /tmp/br.log.2022-07-15T07.13.51Z 
Error: restore log from 434600322529230849(2022-07-15 07:04:21.992 +0000 UTC) to 434600400949870596(2022-07-15 07:09:21.143 +0000 UTC),  but the current existed log from 434600322031419393(2022-07-15 07:04:20.093 +0000 UTC) to 434600397227425834(2022-07-15 07:09:06.943 +0000 UTC): [BR:Common:ErrInvalidArgument]invalid argument
2022-07-15T15:13:51.962+0800    INFO    host/host.go:37 Execute command error   {"command": " /br  restore  point \"--storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Log?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\" \"--pd\" \"http://dst-tidb-pd.fb-pitr-4u-t6djr:2379/\" \"--restored-ts\" \"434600400949870596\" \"--full-backup-storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Full?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\"", "exit code": 1, "stdout": "", "stderr": "Detail BR log in /tmp/br.log.2022-07-15T07.13.51Z \nError: restore log from 434600322529230849(2022-07-15 07:04:21.992 +0000 UTC) to 434600400949870596(2022-07-15 07:09:21.143 +0000 UTC),  but the current existed log from 434600322031419393(2022-07-15 07:04:20.093 +0000 UTC) to 434600397227425834(2022-07-15 07:09:06.943 +0000 UTC): [BR:Common:ErrInvalidArgument]invalid argument\n", "error": "command terminated with exit code 1"}
2022-07-15T15:13:51.962+0800    INFO    host/host.go:41 Execute command finished        {"command": " /br  restore  point \"--storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Log?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\" \"--pd\" \"http://dst-tidb-pd.fb-pitr-4u-t6djr:2379/\" \"--restored-ts\" \"434600400949870596\" \"--full-backup-storage\" \"s3://tmp/br-log-backup847ce747-ae23-4c21-a6c5-e81ca2cfa0a5/Full?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true\"", "execute duration": "472.100135ms"}

4. What is your TiDB version? (Required)

/ # /br -V
Release Version: v6.2.0-alpha
Git Commit Hash: be50ebc
Git Branch: heads/refs/tags/v6.2.0-alpha
Go Version: go1.18.2
UTC Build Time: 2022-07-14 11:02:31
Race Enabled: false

@fubinzh fubinzh added type/bug The issue is confirmed as a bug. component/br This issue is related to BR of TiDB. severity/critical labels Jul 15, 2022
@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-6.1 labels Jul 15, 2022
@fubinzh fubinzh added feature/developing the related feature is in development found/automation Found by automation tests and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.0 may-affects-6.1 labels Jul 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/br This issue is related to BR of TiDB. feature/developing the related feature is in development found/automation Found by automation tests severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
2 participants