From c849e22d47152e9b7913c7547e3a0ba10690c38e Mon Sep 17 00:00:00 2001 From: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Date: Tue, 8 Aug 2023 09:34:50 +0300 Subject: [PATCH] Docs: Online DDL, @@migration_context, --skip-preflight Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> --- .../en/docs/18.0/concepts/query-rewriting.md | 1 + .../vtctl/schema-version-permissions.md | 5 ++- .../schema-changes/advanced-usage.md | 36 ++++++++++++------- .../schema-changes/audit-and-control.md | 3 +- .../schema-changes/concurrent-migrations.md | 2 +- .../managed-online-schema-changes.md | 5 ++- 6 files changed, 30 insertions(+), 22 deletions(-) diff --git a/content/en/docs/18.0/concepts/query-rewriting.md b/content/en/docs/18.0/concepts/query-rewriting.md index f167eab3d..8a62ac0f7 100644 --- a/content/en/docs/18.0/concepts/query-rewriting.md +++ b/content/en/docs/18.0/concepts/query-rewriting.md @@ -95,6 +95,7 @@ Here is a list of all the system variables that are handled by Vitess and how th | charset | VitessAware | | names | VitessAware | | session_uuid | VitessAware | +| migration_context | VitessAware | | enable_system_settings | VitessAware | | read_after_write_gtid | VitessAware | | read_after_write_timeout | VitessAware | diff --git a/content/en/docs/18.0/reference/programs/vtctl/schema-version-permissions.md b/content/en/docs/18.0/reference/programs/vtctl/schema-version-permissions.md index c0ae4b558..f2a228199 100644 --- a/content/en/docs/18.0/reference/programs/vtctl/schema-version-permissions.md +++ b/content/en/docs/18.0/reference/programs/vtctl/schema-version-permissions.md @@ -149,11 +149,11 @@ Validates that the schema on the primary tablet for shard 0 matches the schema o ### ApplySchema -Applies the schema change to the specified keyspace on all shards. The recommended method of applying schema changes is via [Online DDL](../../../../user-guides/schema-changes/managed-online-schema-changes/), which ensures migrations are non-blocking and keeps replication in sync throughout the operation. --ddl_strategy is used to instruct migrations via vreplication, `gh-ost` or `pt-osc` with optional parameters. -request_context allows the user to specify a custom request context for online DDL migrations. If --skip_preflight, SQL goes directly to shards without going through sanity checks. +Applies the schema change to the specified keyspace on all shards. The recommended method of applying schema changes is via [Online DDL](../../../../user-guides/schema-changes/managed-online-schema-changes/), which ensures migrations are non-blocking and keeps replication in sync throughout the operation. --ddl_strategy is used to instruct migrations via vreplication, `gh-ost` or `pt-osc` with optional parameters. -request_context allows the user to specify a custom request context for online DDL migrations. #### Example -
ApplySchema -- [--wait_replicas_timeout=10s] [--ddl_strategy=] [--request_context=] [--skip_preflight] {--sql=<sql> || --sql-file=<filename>} <keyspace>
+
ApplySchema -- [--wait_replicas_timeout=10s] [--ddl_strategy=] [--request_context=] {--sql=<sql> || --sql-file=<filename>} <keyspace>
#### Flags @@ -162,7 +162,6 @@ Applies the schema change to the specified keyspace on all shards. The recommend | caller_id | string | Effective caller ID used for the operation and should map to an ACL name which grants this identity the necessary permissions to perform the operation (this is only necessary when strict table ACLs are used). | | ddl_strategy | string | Online DDL strategy, compatible with @@ddl_strategy session variable (examples: 'gh-ost', 'pt-osc', 'gh-ost --max-load=Threads_running=100' (default "direct"). | | request_context | string | For Only DDL, optionally supply a custom unique string used as context for the migration(s) in this command. By default a unique context is auto-generated by Vitess. | -| skip_preflight | Boolean | Skip pre-apply schema checks, and directly forward schema change query to shards. Ignored when using Online DDL | | sql | string | A list of semicolon-delimited SQL commands. | | sql-file | string | Identifies the file that contains the SQL commands. This file needs to exist on the server, rather than on the client. | | wait_replicas_timeout | Duration | The amount of time to wait for replicas to receive the schema change via replication (default 10s). | diff --git a/content/en/docs/18.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/18.0/user-guides/schema-changes/advanced-usage.md index 434193192..a38be59b9 100644 --- a/content/en/docs/18.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/18.0/user-guides/schema-changes/advanced-usage.md @@ -33,9 +33,9 @@ Thus, the new migration does not get to execute if an identical previous migrati Usage: ```sh -$ vtctldclient ApplySchema -- --migration_context="1111-2222" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce +$ vtctldclient ApplySchema -- --migration_context="1111-2222" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce -$ vtctldclient ApplySchema -- --migration_context="1111-2222" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce +$ vtctldclient ApplySchema -- --migration_context="1111-2222" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce ``` In the above, the two calls are identical. Specifically, they share the exact same `--migration_context` value of `1111-2222`, and the exact same `--sql`. @@ -49,6 +49,16 @@ Notes: - Continuing the above example: if the new migration is successful, the previous migration remains in `failed` state. - The decision whether to run the migration or to mark it as implicitly `complete` only takes place when the migration is scheduled to run. +The context may also be set via VTGate: + +```sql +mysql> set @@ddl_strategy='vitess --'; +mysql> set @@migration_context='1111-2222'; +mysql> alter table customer add column status int unsigned not null; +``` + +By default, the migration context for an Online DDL issued via VTGate is the value of `@@session_uuid`. If `@@migration_context` is non-empty, then its value is used. + ## Duplicate migration indication You may go one step beyond [duplicate migration detection](#duplicate-migration-detection) by explicitly supplying migration UUIDs such that duplicate migrations are never submitted in the first place. @@ -56,9 +66,9 @@ You may go one step beyond [duplicate migration detection](#duplicate-migration- Consider the following example, note `--uuid_list` flag: ```sh -$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce +$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce -$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce +$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce ``` Normally Vitess generates a `UUID` for each migration, thus having a new, unique ID per migration. With `-uuid_list`, you can force Vitess into using your own supplied UUID. There cannot be two migrations with the same `UUID`. Therefore, any subsequent submission of a migration with an already existing `UUID` is implicitly discarded. The 2nd call does return the migration `UUID`, but is internally discarded. @@ -86,7 +96,7 @@ The user may submit multiple migrations such that non auto-completes. The user c Consider the following: ```sh -$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "alter table customer add column country int not null default 0; alter table order add column shipping_country int not null default 0" commerce +$ vtctldclient ApplySchema --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "alter table customer add column country int not null default 0; alter table order add column shipping_country int not null default 0" commerce 29231906_776f_11ec_a656_0a43f95f28a3 3cc4ae0e_776f_11ec_a656_0a43f95f28a3 ``` @@ -98,14 +108,14 @@ A `show vitess_migrations like '29231906_776f_11ec_a656_0a43f95f28a3'` presents When all migrations for the relevant UUIDs show `1` for `ready_to_complete`, the user can then either: ```sh -$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration complete all" commerce +$ vtctldclient ApplySchema --sql "alter vitess_migration complete all" commerce ``` Assuming these are the only migrations awaiting completion, or, explicitly issue a complete for each of the migrations: ```sh -$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3' complete all" commerce -$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete all" commerce +$ vtctldclient ApplySchema --sql "alter vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3' complete all" commerce +$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete all" commerce ``` ## Near instant REVERTs @@ -125,14 +135,14 @@ The use case and workflow is as follows: Consider the following example. We run a 5 hour long migration to drop an index: ```sh -$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess' --sql "alter table customer drop index joined_timestamp_idx" commerce +$ vtctldclient ApplySchema --ddl_strategy='vitess' --sql "alter table customer drop index joined_timestamp_idx" commerce 29231906_776f_11ec_a656_0a43f95f28a3 ``` As soon as the migration completes, we run: ```sh -$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "revert vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3'" commerce +$ vtctldclient ApplySchema --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "revert vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3'" commerce 3cc4ae0e_776f_11ec_a656_0a43f95f28a3 ``` @@ -141,7 +151,7 @@ The above begins a `REVERT` migration that is open-ended (does not complete), vi Finally, if we are satisfied that the `drop index` migration went well, we issue: ```sh -$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' cancel" commerce +$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' cancel" commerce ``` That is, we cancel the `REVERT` operation. @@ -149,7 +159,7 @@ That is, we cancel the `REVERT` operation. Or, should we have not dropped the index? If our migration seems to have been wrong, we run: ```sh -$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete" commerce +$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete" commerce ``` Which means we want to apply the revert. Since the revert is already running in the background, it is likely that binary log processing is up to date, and cut-over is near instantaneous. @@ -171,7 +181,7 @@ Note that there can be scenarios with impossible ordering. Those hardly make sen An example for in-order submission: ```sh -$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --allow-concurrent --in-order-completion' --sql "create table t1 (id int primary key); create view v1 as select id from t1;" commerce +$ vtctldclient ApplySchema --ddl_strategy='vitess --allow-concurrent --in-order-completion' --sql "create table t1 (id int primary key); create view v1 as select id from t1;" commerce ``` {{< info >}} diff --git a/content/en/docs/18.0/user-guides/schema-changes/audit-and-control.md b/content/en/docs/18.0/user-guides/schema-changes/audit-and-control.md index 118fcbee9..82a309e32 100644 --- a/content/en/docs/18.0/user-guides/schema-changes/audit-and-control.md +++ b/content/en/docs/18.0/user-guides/schema-changes/audit-and-control.md @@ -66,7 +66,7 @@ a2994c92_f1d4_11ea_afa3_f875a4d24e90 You my run multiple migrations withing the same `ApplySchema` command: ```shell -$ vtctlclient -- ApplySchema --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce +$ vtctlclient -- ApplySchema --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce 3091ef2a_4b87_11ec_a827_0a43f95f28a3 ``` @@ -75,7 +75,6 @@ $ vtctlclient -- ApplySchema --skip_preflight --ddl_strategy "vitess" --sql "ALT - `--ddl_strategy`: by default migrations run directly via MySQL standard DDL. This flag must be aupplied to indicate an online strategy. See also [DDL strategies](../ddl-strategies) and [ddl_strategy flags](../ddl-strategy-flags). - `--migration_context `: all migrations in a `ApplySchema` command are logically grouped via a unique _context_. A unique value will be supplied automatically. The user may choose to supply their own value, and it's their responsibility to provide with a unique value. Any string format is accepted. The context can then be used to search for migrations, via `SHOW VITESS_MIGRATIONS LIKE 'the-context'`. It is visible in `SHOW VITESS_MIGRATIONS ...` output as the `migration_context` column. -- `--skip_preflight`: skip an internal Vitess schema validation. When running an online DDL it's recommended to add `--skip_preflight`. In future Vitess versions this flag may be removed or default to `true`. ## Tracking migrations diff --git a/content/en/docs/18.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/18.0/user-guides/schema-changes/concurrent-migrations.md index 493cb61f4..aabf08cf1 100644 --- a/content/en/docs/18.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/18.0/user-guides/schema-changes/concurrent-migrations.md @@ -46,7 +46,7 @@ mysql> create table sample_table(id int primary key); or, via `vtctl`: ```shell -vtctl ApplySchema -- --skip_preflight --ddl_strategy "vitess --allow-concurrent" -sql "REVERT VITESS_MIGRATION '3091ef2a_4b87_11ec_a827_0a43f95f28a3'" +vtctldclient ApplySchema --ddl-strategy "vitess --allow-concurrent" -sql "REVERT VITESS_MIGRATION '3091ef2a_4b87_11ec_a827_0a43f95f28a3'" ``` ## Restrictions and eligibility diff --git a/content/en/docs/18.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/18.0/user-guides/schema-changes/managed-online-schema-changes.md index 0e91f7979..e8015916a 100644 --- a/content/en/docs/18.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/18.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -136,12 +136,12 @@ mysql> drop table customer; #### Executing an Online DDL via vtctlclient/ApplySchema ```shell -$ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED" commerce +$ vtctldclient ApplySchema --ddl-strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED" commerce a2994c92_f1d4_11ea_afa3_f875a4d24e90 ``` You my run multiple migrations withing the same `ApplySchema` command: ```shell -$ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce +$ vtctldclient ApplySchema --ddl-strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce 3091ef2a_4b87_11ec_a827_0a43f95f28a3 ``` @@ -150,7 +150,6 @@ $ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALT - `--ddl_strategy`: by default migrations run directly via MySQL standard DDL. This flag must be applied to indicate an online strategy. See also [DDL strategies](../ddl-strategies) and [ddl_strategy flags](../ddl-strategy-flags). - `-migration_context `: all migrations in a `ApplySchema` command are logically grouped via a unique _context_. A unique value will be supplied automatically. The user may choose to supply their own value, and it's their responsibility to provide with a unique value. Any string format is accepted. The context can then be used to search for migrations, via `SHOW VITESS_MIGRATIONS LIKE 'the-context'`. It is visible in `SHOW VITESS_MIGRATIONS ...` output as the `migration_context` column. -- `--skip_preflight`: skip an internal Vitess schema validation. When running an online DDL it's recommended to add `--skip_preflight`. In future Vitess versions this flag may be removed or default to `true`. ## Migration flow and states