OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171

shlomi-noach · 2021-11-09T12:35:41Z

This PR implements:

-postpone-completion

-postpone-completion flag in ddl_strategy. This flag indicates that the migration should not auto-complete. This applies for:

any CREATE TABLE
any DROP TABLE
ALTER table in online strategy
ALTER table in gh-ost strategy

Note that this flag is not supported for pt-osc strategy.

Behavior of migrations with this flag:

an ALTER table begins, runs, but does not cut-over.
CREATE or DROP migrations are silently not even scheduled

Complimenting this flag is:

ALTER VITESS_MIGRATION '-uuid-' COMPLETE

This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately - cut-over depends on polling interval, replication lag, etc)
For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over automatically when they complete their work, as if -postpone-completion wasn't indicated
For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the schedulers discretion. there is no guarantee that they will be scheduled to run immediately.

tests added:

vrepl endtoend
gh-ost endtoend

This PR extends #9160 (not yet merged at this time)
Tracking issue: #6926

Signed-off-by: Shlomi Noach <[email protected]>

…f cut-over for 'online' migrations until value is cleared. ALTER VITESS_MIGRATION ... COMPLETE now implemented and clears the flag. tests added. Signed-off-by: Shlomi Noach <[email protected]>

…etion>0 Signed-off-by: Shlomi Noach <[email protected]>

… has postpone_completion Signed-off-by: Shlomi Noach <[email protected]>

Signed-off-by: Shlomi Noach <[email protected]>

…eue posponed migrations. New test fails in th emeantime Signed-off-by: Shlomi Noach <[email protected]>

…ction fields while queued Signed-off-by: Shlomi Noach <[email protected]>

…anity check at end of switch Signed-off-by: Shlomi Noach <[email protected]>

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2021-11-17T08:47:36Z

As #9160 is merged, this PR is now ready to review. /cc @deepthi @sougou

Signed-off-by: Shlomi Noach <[email protected]>

frouioui

I do not have a lot of context on OnlineDDL, but the changes look good to me. We should wait for someone with more context to review this too :)

go/vt/vttablet/onlineddl/executor.go

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2021-11-21T07:53:11Z

Noting that -postpone-completion implementation is incomplete, as it will withhold -declarative migrations; this will be addressed in a separate PR.

deepthi

A couple of documentation nits, otherwise LGTM.
It's a pleasure to review a PR that is more tests than code 💯

deepthi · 2021-11-21T13:44:00Z

doc/releasenotes/13_0_0_summary.md

 ```sql
 alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup
 ```
-This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no
+
+This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As reminder, once a migration's artifacts are cleaned up, the migration is no


Suggested change

This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As reminder, once a migration's artifacts are cleaned up, the migration is no

This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no

deepthi · 2021-11-21T13:44:45Z

doc/releasenotes/13_0_0_summary.md

+
+- For running `ALTER`s (`online` and `gh-ost`) which are ready to cut-over: cut-over imminently (though not immediately - cut-over depends on polling interval, replication lag, etc)
+- For running `ALTER`s (`online` and `gh-ost`) which are only partly through the migration: they will cut-over automatically when they complete their work, as if `-postpone-completion` wasn't indicated
+- For queued `CREATE` and `DROP` migrations: "unblock" them from being scheduled. They'll be scheduled at the schedulers discretion. there is no guarantee that they will be scheduled to run immediately.


Suggested change

- For queued `CREATE` and `DROP` migrations: "unblock" them from being scheduled. They'll be scheduled at the schedulers discretion. there is no guarantee that they will be scheduled to run immediately.

- For queued `CREATE` and `DROP` migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler's discretion. there is no guarantee that they will be scheduled to run immediately.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2021-11-22T11:56:51Z

Dcoumentation PR: vitessio/website#887

shlomi-noach added 6 commits November 9, 2021 12:12

Adding -postpone-completion flag to ddl_strategy

ba81b88

Signed-off-by: Shlomi Noach <[email protected]>

persisting postpone_completion in schema_migrations table. Holding of…

6c7f9a4

…f cut-over for 'online' migrations until value is cleared. ALTER VITESS_MIGRATION ... COMPLETE now implemented and clears the flag. tests added. Signed-off-by: Shlomi Noach <[email protected]>

do not schedule 'CREATE', 'DROP' migrations which have postpone_compl…

a562ad5

…etion>0 Signed-off-by: Shlomi Noach <[email protected]>

change: don't even mark a 'CREATE' or 'DROP' migration as ready if it…

2798c84

… has postpone_completion Signed-off-by: Shlomi Noach <[email protected]>

support -postpone-completion for gh-ost migrations

a8a1f0a

Signed-off-by: Shlomi Noach <[email protected]>

updated release notes

3f57689

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving release notes (needs details) This PR needs to be listed in the release notes in a dedicated section (deprecation notice, etc...) labels Nov 9, 2021

shlomi-noach requested review from dbussink and deepthi November 9, 2021 12:35

more cleanup

e832585

Signed-off-by: Shlomi Noach <[email protected]>

This was referenced Nov 10, 2021

Document ALTER VITESS_MIGRATION ... COMPLETE vitessio/website#869

Closed

Document -postpone-completion ddl_strategy flag vitessio/website#870

Closed

shlomi-noach added 4 commits November 11, 2021 09:19

test posponed revert: already notice there's a problem with how we qu…

8a104cb

…eue posponed migrations. New test fails in th emeantime Signed-off-by: Shlomi Noach <[email protected]>

pre-evaluate queued REVERT migrations, populate mysql_table and ddl_a…

ee8ea81

…ction fields while queued Signed-off-by: Shlomi Noach <[email protected]>

refactor duplicate code in if block, to be removed in v14. Restored s…

323d774

…anity check at end of switch Signed-off-by: Shlomi Noach <[email protected]>

endtoend test for postponed DROP ddl

547be37

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach mentioned this pull request Nov 11, 2021

Concurrent migrations #9192

Merged

3 tasks

shlomi-noach requested a review from sougou November 15, 2021 13:11

shlomi-noach marked this pull request as ready for review November 17, 2021 08:47

shlomi-noach requested review from frouioui, GuptaManan100, harshit-gangal and systay as code owners November 17, 2021 08:47

merged main; resolved conflict

b5bbf87

Signed-off-by: Shlomi Noach <[email protected]>

frouioui approved these changes Nov 17, 2021

View reviewed changes

go/vt/vttablet/onlineddl/executor.go Outdated Show resolved Hide resolved

unindent block

a0db3c5

Signed-off-by: Shlomi Noach <[email protected]>

deepthi approved these changes Nov 21, 2021

View reviewed changes

typos, fixes per review

d26db89

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach merged commit 15c49ba into vitessio:main Nov 21, 2021

shlomi-noach deleted the online-ddl-postpone-completion branch November 21, 2021 15:08

shlomi-noach mentioned this pull request Nov 22, 2021

Documenting postponed migrations and 'ALTER VITESS_MIGRATION ... COMPLETE' command vitessio/website#887

Merged

shlomi-noach mentioned this pull request Nov 30, 2021

Online DDL: vtctl OnlineDDL supports 'complete' command #9298

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171

OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171

shlomi-noach commented Nov 9, 2021 •

edited

Loading

shlomi-noach commented Nov 17, 2021

frouioui left a comment

shlomi-noach commented Nov 21, 2021

deepthi left a comment

deepthi Nov 21, 2021

shlomi-noach Nov 21, 2021

deepthi Nov 21, 2021

shlomi-noach Nov 21, 2021

shlomi-noach commented Nov 22, 2021

	This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As reminder, once a migration's artifacts are cleaned up, the migration is no
	This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no

	- For queued `CREATE` and `DROP` migrations: "unblock" them from being scheduled. They'll be scheduled at the schedulers discretion. there is no guarantee that they will be scheduled to run immediately.
	- For queued `CREATE` and `DROP` migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler's discretion. there is no guarantee that they will be scheduled to run immediately.

OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171

OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171

Conversation

shlomi-noach commented Nov 9, 2021 • edited Loading

-postpone-completion

ALTER VITESS_MIGRATION '-uuid-' COMPLETE

shlomi-noach commented Nov 17, 2021

frouioui left a comment

Choose a reason for hiding this comment

shlomi-noach commented Nov 21, 2021

deepthi left a comment

Choose a reason for hiding this comment

deepthi Nov 21, 2021

Choose a reason for hiding this comment

shlomi-noach Nov 21, 2021

Choose a reason for hiding this comment

deepthi Nov 21, 2021

Choose a reason for hiding this comment

shlomi-noach Nov 21, 2021

Choose a reason for hiding this comment

shlomi-noach commented Nov 22, 2021

shlomi-noach commented Nov 9, 2021 •

edited

Loading