Add Support for Restoring Specific Backups #8824

mattlord · 2021-09-15T20:44:42Z

Description

By default Vitess will only make practical use of the latest backup of a given shard. While this makes perfect sense for the common use cases there are times where you may want to restore a specific backup. For example:

In order to extract a portion of the data that can then be merged with the current state. For example if you later realize that you accidentally deleted some records in a table that you shouldn't have last week, and you need to perform a restore so that you can copy those specific records back to the live data set.
To perform validation, forensics, analysis on the system state at that time.
A specific PITR for whatever reason ...

This work supports that in two different ways:

You can start a tablet with: -restore_from_backup -restore_from_backup_ts 2021-04-29.133050
You can use vtctl with: vtctlclient -server=<vtctld-server>:<vtctld-port> RestoreFromBackup -backup_timestamp=2021-04-29.133050 <tablet-alias>

This compared with PITR

While Vitess supports a method to accomplish PITR, it's a fairly involved process with the details left up to the user (e.g. binlog servers). This work offers another method that can potentially be used in various ways/circumstances as an alternative. For example, let's say we realize that 1-2 days ago we made a mistake and want to look at the previous state of the data on one tablet in the shard... we could do that with:

$ vtctlclient -server=<vtctld-server>:<vtctld-port> RestoreFromBackup \
    -backup_timestamp=$(date -d"2 days ago" +"%Y-%m-%d").000000 <tablet-alias> && \
      vtctlclient -server=<vtctld-server>:<vtctld-port> StopReplication <tablet-alias>

This will stop MySQL replication on the tablet ASAP after the restore completes and prevent it from being automatically re-started (via tablet repair). This allows you to do whatever you like on that tablet before deciding to later:

Throw it away (perhaps we added this new tablet just for this purpose)
Let it catch up with replication normally (just starting replication again via vtctlclient StartReplication)
Perform a Restore on it using the latest shard backup
...

Co-authored-by: Guido Iaquinti [email protected]
Signed-off-by: Matt Lord [email protected]

Related Issue(s)

This is a continuation of: #7998
This solves: #4905

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation is needed: Document Restores from backup timestamp website#834

By default Vitess will only make practical use of the latest backup of a given shard. While this makes perfect sense for the common use cases there are times where you need to restore a specific backup. For example: 1. In order to extract a portion of the data that can then be merged with the current state. For example if you later realize that you accidentally deleted some records in a table that you shouldn't have last week, and you need to perform a restore so that you can copy those specific records back to the live data set. 2. To perform validation, forensics, analysis on the system state at that time. 3. A specific PITR for whatever reason ... This is a continuation of: vitessio#7998 This solves: vitessio#4905 Co-authored-by: Guido Iaquinti <[email protected]> Signed-off-by: Matt Lord <[email protected]>

guidoiaquinti · 2021-09-16T06:20:03Z

Once tests are added LGTM. Thank you for moving this forward!

mattlord · 2021-09-17T02:41:02Z

I've tested and verified the tablet flag as well as the vtctlclient flag:

Backups

$ vtctlclient ListBackups commerce/0
2021-09-17.034338.zone1-0000000101
2021-09-17.034524.zone1-0000000101

Updated help output

$ vtctlclient RestoreFromBackup --help
Usage: RestoreFromBackup [-backup_timestamp=yyyy-MM-dd.HHmmss] <tablet alias>

Stops mysqld and restores the data from the latest backup or if a timestamp is specified then the most recent backup at or before that time.

  -backup_timestamp string
    	Use the backup taken at or before this timestamp rather than using the latest backup.

Invalid timestamp

$ vtctlclient RestoreFromBackup -backup_timestamp=222 zone1-101
RestoreFromBackup Error: rpc error: code = Unknown desc = TabletManager.RestoreFromBackup on zone1-0000000101 error: unable to parse the backup timestamp value provided of '222': unable to parse the backup timestamp value provided of '222'
E0917 03:46:24.807775    3246 main.go:76] remote error: rpc error: code = Unknown desc = TabletManager.RestoreFromBackup on zone1-0000000101 error: unable to parse the backup timestamp value provided of '222': unable to parse the backup timestamp value provided of '222'

Uses latest / 2021-09-17.034524

$ vtctlclient RestoreFromBackup zone1-101
I0917 03:46:49.446409    3252 main.go:67] I0917 03:46:49.445436 backup.go:264] I0917 03:46:49.444950 backup.go:270] Restore: looking for a suitable backup to restore
I0917 03:46:49.446658    3252 main.go:67] I0917 03:46:49.445792 backup.go:264] I0917 03:46:49.445654 backupengine.go:222] Restore: found latest backup commerce/0 2021-09-17.034524.zone1-0000000101 to restore
...
I0917 03:46:57.013383    3252 main.go:67] I0917 03:46:57.013171 backup.go:264] I0917 03:46:57.012888 backup.go:356] Restore: restarting mysqld after mysql_upgrade

Uses the older one based on the timestamp

$ vtctlclient RestoreFromBackup -backup_timestamp=2021-09-17.034338 zone1-101
I0917 03:48:57.366954    5613 main.go:67] I0917 03:48:57.366055 backup.go:264] I0917 03:48:57.363727 backup.go:270] Restore: looking for a suitable backup to restore
I0917 03:48:57.367262    5613 main.go:67] I0917 03:48:57.366087 backup.go:264] I0917 03:48:57.364443 backupengine.go:224] Restore: found backup commerce/0 2021-09-17.034338.zone1-0000000101 to restore using the specified timestamp of '2021-09-17.034338'
...
I0917 03:49:04.905961    5613 main.go:67] I0917 03:49:04.905797 backup.go:264] I0917 03:49:04.905601 backup.go:356] Restore: restarting mysqld after mysql_upgrade

Now I have a good idea of what to test and how in the test suite. 🙂

Signed-off-by: Matt Lord <[email protected]>

deepthi

Looking good, except protobuf definition is missing.

go/vt/proto/tabletmanagerdata/tabletmanagerdata.pb.go

And add missing protobuf change Signed-off-by: Matt Lord <[email protected]>

mm == minutes (two digits) MM == month (two digits) Signed-off-by: Matt Lord <[email protected]>

deepthi

LGTM.
Let's get another set of eyes before merging.
Can you please create a website PR to document the new usage (vttablet flag & vtctl command)?

deepthi · 2021-09-23T23:57:38Z

go/test/endtoend/backup/vtctlbackup/backup_utils.go

+	err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", "-allow_primary=true", replica2.Alias)
+	require.Nil(t, err)


This is a clever way to simulate the test scenario 👍

This documents the work done in: vitessio/vitess#8824 Signed-off-by: Matt Lord <[email protected]>

mattlord · 2021-09-24T02:45:32Z

Can you please create a website PR to document the new usage (vttablet flag & vtctl command)?

vitessio/website#834 (noted in the description now and linked in comments)

ajm188

I wanted to bring up one comment about the types we're using, but if you don't feel it's worth it to change then I'm happy to merge as-is.

ajm188 · 2021-09-24T11:00:55Z

go/vt/vttablet/tabletmanager/restore.go

+	// Check if we should use the latest (default) or a specified backup timestamp for the restore
+	if restoreFromBackupTs != "" {
+		startTime, err = time.Parse(mysqlctl.BackupTimestampFormat, restoreFromBackupTs)
+		if err != nil {
+			return vterrors.New(vtrpcpb.Code_INVALID_ARGUMENT, fmt.Sprintf("unable to parse the backup timestamp value provided of '%s'", restoreFromBackupTs))
+		}
+	}


I skimmed the rest of the PR and it doesn't seem like anyone brought this up. Another way to do this would be to push the parsing further up to the cli, and enforce stricter types closer to this function. In this way we could:

commandRestoreBackup in go/vt/vtctl/vtctl.go parses its backupTimestampStr into a time.Time, using mysqlctl.BackupTimestampFormat

we pass time.Time to all the tmc functions

when making the gRPC request, we change that protobuf definition to be a vttime.Time, and call protoutil.TimeToProto to convert the time.Time to a vttime.Time, and convert back on the tabletmanager side.

What do you think?

I briefly considered this but wasn't sure of the benefits (avoiding the RPC call with an invalid vtctl CLI parameter is one). I don't mind making this change if you prefer it? Sounds like you do or you wouldn't have mentioned it 😄 ... so I'll work on that now. Thanks!

OK, after trying it again I'm remembering WHY I didn't go this route last time. 🙂

What makes it feel awkward/clunky to use time.Time coming from the vtctl side is that there's also a corresponding vttablet flag that's used directly in the tmc:

vitess/go/vt/vttablet/tabletmanager/restore.go

Line 49 in a7a951b

restoreFromBackupTs = flag.String("restore_from_backup_ts", "", "(init restore parameter) if set, restore the latest backup taken at or before this timestamp. Example: '2021-04-29.133050'")

As things are now, it's uniform handling (always a string) on the tmc side -- whether the value came from a vtctl call and RPC or from the vttablet flag. Maybe I'm missing something though?

I understand the pull to use a more precise type... but it also feels like it makes the code more complex with little practical benefit. If you feel strongly about it though I can continue.

OK, I just switched to this method. Thank you for the nudge! I feel better about it too and It wasn't really that awkward in the end. :-)

Signed-off-by: Matt Lord <[email protected]>

ajm188

lgtm!

ajm188 · 2021-09-26T10:59:13Z

go/vt/vttablet/tabletmanager/tm_init.go

@@ -636,9 +636,21 @@ func (tm *TabletManager) handleRestore(ctx context.Context) (bool, error) {
 			// Open the state manager after restore is done.
 			defer tm.tmState.Open()

+			// Zero date will cause us to use the latest, which is the default


Follow-up to: vitessio#8824 I noticed this oversight after it was merged. :-) Signed-off-by: Matt Lord <[email protected]>

mattlord requested review from deepthi and guidoiaquinti September 15, 2021 20:44

mattlord mentioned this pull request Sep 15, 2021

vttablet - add '-restore_from_backup_ts' flag #7998

Closed

2 tasks

mattlord self-assigned this Sep 15, 2021

mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: TabletManager Component: Backup and Restore Component: vtctl and removed Component: TabletManager labels Sep 15, 2021

mattlord force-pushed the RestoreFromTimestamp branch from 597aa29 to 50938fd Compare September 15, 2021 22:01

mattlord force-pushed the RestoreFromTimestamp branch 5 times, most recently from 1caad45 to 863df25 Compare September 17, 2021 02:17

mattlord force-pushed the RestoreFromTimestamp branch 2 times, most recently from 40a66cf to 0d9f2c8 Compare September 17, 2021 03:21

Add support for vtctl restores from a timestamp

bf78488

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the RestoreFromTimestamp branch from 0d9f2c8 to bf78488 Compare September 17, 2021 03:35

deepthi reviewed Sep 17, 2021

View reviewed changes

go/vt/proto/tabletmanagerdata/tabletmanagerdata.pb.go Outdated Show resolved Hide resolved

mattlord force-pushed the RestoreFromTimestamp branch from e61794f to 4b80776 Compare September 17, 2021 23:42

Add unit tests for tablet restore with timestamp

4bfc2c1

And add missing protobuf change Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the RestoreFromTimestamp branch 5 times, most recently from 9db70ac to 25fa195 Compare September 21, 2021 01:11

mattlord force-pushed the RestoreFromTimestamp branch from dff915a to f8ec301 Compare September 21, 2021 16:08

Correct time format in help output

a7a951b

mm == minutes (two digits) MM == month (two digits) Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the RestoreFromTimestamp branch from ee31291 to a7a951b Compare September 23, 2021 18:41

deepthi removed request for shlomi-noach and rohit-nayak-ps September 23, 2021 23:54

deepthi approved these changes Sep 24, 2021

View reviewed changes

deepthi requested review from sougou and ajm188 September 24, 2021 00:01

mattlord added a commit to vitessio/website that referenced this pull request Sep 24, 2021

Document Restores from backup timestamp

ddfe567

This documents the work done in: vitessio/vitess#8824 Signed-off-by: Matt Lord <[email protected]>

mattlord mentioned this pull request Sep 24, 2021

Document Restores from backup timestamp vitessio/website#834

Merged

ajm188 approved these changes Sep 24, 2021

View reviewed changes

mattlord force-pushed the RestoreFromTimestamp branch 5 times, most recently from 0920d09 to 67db03f Compare September 25, 2021 04:25

Be more precise and use time.Time instead of string

f9c6938

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the RestoreFromTimestamp branch from 67db03f to f9c6938 Compare September 25, 2021 05:47

ajm188 approved these changes Sep 26, 2021

View reviewed changes

ajm188 merged commit 84aad0d into vitessio:main Sep 26, 2021

mattlord deleted the RestoreFromTimestamp branch September 27, 2021 14:28

mattlord mentioned this pull request Sep 27, 2021

Correctly identify backup timestamp var as a string #8891

Merged

3 tasks

mattlord added a commit to planetscale/vitess that referenced this pull request Sep 27, 2021

Correctly identify backup timestamp var as a string

4ff4a32

Follow-up to: vitessio#8824 I noticed this oversight after it was merged. :-) Signed-off-by: Matt Lord <[email protected]>

frouioui added the release notes label Sep 28, 2021

mattlord mentioned this pull request Oct 25, 2021

Add a way to run a restore from a specific backup #4905

Closed

deepthi mentioned this pull request Jul 19, 2023

Bug Report: Snapshot keyspace restores always restore from latest backup #13556

Closed

pbibra mentioned this pull request Jul 20, 2023

check keyspace snapshot time for restore slackhq/vitess#105

Closed

frouioui mentioned this pull request Oct 17, 2024

Feature Request: If a restore fails, try an earlier backup. #13320

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support for Restoring Specific Backups #8824

Add Support for Restoring Specific Backups #8824

mattlord commented Sep 15, 2021 •

edited

Loading

guidoiaquinti commented Sep 16, 2021

mattlord commented Sep 17, 2021 •

edited

Loading

deepthi left a comment

deepthi left a comment

deepthi Sep 23, 2021

mattlord commented Sep 24, 2021

ajm188 left a comment

ajm188 Sep 24, 2021

mattlord Sep 24, 2021

mattlord Sep 24, 2021 •

edited

Loading

mattlord Sep 25, 2021

ajm188 left a comment

ajm188 Sep 26, 2021

		err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", "-allow_primary=true", replica2.Alias)
		require.Nil(t, err)

Add Support for Restoring Specific Backups #8824

Add Support for Restoring Specific Backups #8824

Conversation

mattlord commented Sep 15, 2021 • edited Loading

Description

This compared with PITR

Related Issue(s)

Checklist

guidoiaquinti commented Sep 16, 2021

mattlord commented Sep 17, 2021 • edited Loading

Backups

Updated help output

Invalid timestamp

Uses latest / 2021-09-17.034524

Uses the older one based on the timestamp

deepthi left a comment

Choose a reason for hiding this comment

deepthi left a comment

Choose a reason for hiding this comment

deepthi Sep 23, 2021

Choose a reason for hiding this comment

mattlord commented Sep 24, 2021

ajm188 left a comment

Choose a reason for hiding this comment

ajm188 Sep 24, 2021

Choose a reason for hiding this comment

mattlord Sep 24, 2021

Choose a reason for hiding this comment

mattlord Sep 24, 2021 • edited Loading

Choose a reason for hiding this comment

mattlord Sep 25, 2021

Choose a reason for hiding this comment

ajm188 left a comment

Choose a reason for hiding this comment

ajm188 Sep 26, 2021

Choose a reason for hiding this comment

mattlord commented Sep 15, 2021 •

edited

Loading

mattlord commented Sep 17, 2021 •

edited

Loading

mattlord Sep 24, 2021 •

edited

Loading