Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Online DDL, @@migration_context, --skip-preflight #1558

Merged
merged 2 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions content/en/docs/18.0/concepts/query-rewriting.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Here is a list of all the system variables that are handled by Vitess and how th
| charset | VitessAware |
| names | VitessAware |
| session_uuid | VitessAware |
| migration_context | VitessAware |
| enable_system_settings | VitessAware |
| read_after_write_gtid | VitessAware |
| read_after_write_timeout | VitessAware |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,11 +149,11 @@ Validates that the schema on the primary tablet for shard 0 matches the schema o

### ApplySchema

Applies the schema change to the specified keyspace on all shards. The recommended method of applying schema changes is via [Online DDL](../../../../user-guides/schema-changes/managed-online-schema-changes/), which ensures migrations are non-blocking and keeps replication in sync throughout the operation. --ddl_strategy is used to instruct migrations via vreplication, `gh-ost` or `pt-osc` with optional parameters. -request_context allows the user to specify a custom request context for online DDL migrations. If --skip_preflight, SQL goes directly to shards without going through sanity checks.
Applies the schema change to the specified keyspace on all shards. The recommended method of applying schema changes is via [Online DDL](../../../../user-guides/schema-changes/managed-online-schema-changes/), which ensures migrations are non-blocking and keeps replication in sync throughout the operation. --ddl_strategy is used to instruct migrations via vreplication, `gh-ost` or `pt-osc` with optional parameters. -request_context allows the user to specify a custom request context for online DDL migrations.

#### Example

<pre class="command-example">ApplySchema -- [--wait_replicas_timeout=10s] [--ddl_strategy=<ddl_strategy>] [--request_context=<unique-request-context>] [--skip_preflight] {--sql=&lt;sql&gt; || --sql-file=&lt;filename&gt;} &lt;keyspace&gt;</pre>
<pre class="command-example">ApplySchema -- [--wait_replicas_timeout=10s] [--ddl_strategy=<ddl_strategy>] [--request_context=<unique-request-context>] {--sql=&lt;sql&gt; || --sql-file=&lt;filename&gt;} &lt;keyspace&gt;</pre>

#### Flags

Expand All @@ -162,7 +162,6 @@ Applies the schema change to the specified keyspace on all shards. The recommend
| caller_id | string | Effective caller ID used for the operation and should map to an ACL name which grants this identity the necessary permissions to perform the operation (this is only necessary when strict table ACLs are used). |
| ddl_strategy | string | Online DDL strategy, compatible with @@ddl_strategy session variable (examples: 'gh-ost', 'pt-osc', 'gh-ost --max-load=Threads_running=100' (default "direct"). |
| request_context | string | For Only DDL, optionally supply a custom unique string used as context for the migration(s) in this command. By default a unique context is auto-generated by Vitess. |
| skip_preflight | Boolean | Skip pre-apply schema checks, and directly forward schema change query to shards. Ignored when using Online DDL |
| sql | string | A list of semicolon-delimited SQL commands. |
| sql-file | string | Identifies the file that contains the SQL commands. This file needs to exist on the server, rather than on the client. |
| wait_replicas_timeout | Duration | The amount of time to wait for replicas to receive the schema change via replication (default 10s). |
Expand Down
36 changes: 23 additions & 13 deletions content/en/docs/18.0/user-guides/schema-changes/advanced-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ Thus, the new migration does not get to execute if an identical previous migrati
Usage:

```sh
$ vtctldclient ApplySchema -- --migration_context="1111-2222" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
$ vtctldclient ApplySchema -- --migration_context="1111-2222" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce

$ vtctldclient ApplySchema -- --migration_context="1111-2222" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
$ vtctldclient ApplySchema -- --migration_context="1111-2222" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
```

In the above, the two calls are identical. Specifically, they share the exact same `--migration_context` value of `1111-2222`, and the exact same `--sql`.
Expand All @@ -49,16 +49,26 @@ Notes:
- Continuing the above example: if the new migration is successful, the previous migration remains in `failed` state.
- The decision whether to run the migration or to mark it as implicitly `complete` only takes place when the migration is scheduled to run.

The context may also be set via VTGate:

```sql
mysql> set @@ddl_strategy='vitess --';
mysql> set @@migration_context='1111-2222';
mysql> alter table customer add column status int unsigned not null;
```

By default, the migration context for an Online DDL issued via VTGate is the value of `@@session_uuid`. If `@@migration_context` is non-empty, then its value is used.

## Duplicate migration indication

You may go one step beyond [duplicate migration detection](#duplicate-migration-detection) by explicitly supplying migration UUIDs such that duplicate migrations are never submitted in the first place.

Consider the following example, note `--uuid_list` flag:

```sh
$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce

$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --skip_preflight --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
$ vtctldclient ApplySchema -- --uuid_list "73380089_7764_11ec_a656_0a43f95f28a3" --ddl_strategy='vitess' --sql "alter table customer add column status int unsigned not null" commerce
```

Normally Vitess generates a `UUID` for each migration, thus having a new, unique ID per migration. With `-uuid_list`, you can force Vitess into using your own supplied UUID. There cannot be two migrations with the same `UUID`. Therefore, any subsequent submission of a migration with an already existing `UUID` is implicitly discarded. The 2nd call does return the migration `UUID`, but is internally discarded.
Expand Down Expand Up @@ -86,7 +96,7 @@ The user may submit multiple migrations such that non auto-completes. The user c
Consider the following:

```sh
$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "alter table customer add column country int not null default 0; alter table order add column shipping_country int not null default 0" commerce
$ vtctldclient ApplySchema --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "alter table customer add column country int not null default 0; alter table order add column shipping_country int not null default 0" commerce
29231906_776f_11ec_a656_0a43f95f28a3
3cc4ae0e_776f_11ec_a656_0a43f95f28a3
```
Expand All @@ -98,14 +108,14 @@ A `show vitess_migrations like '29231906_776f_11ec_a656_0a43f95f28a3'` presents
When all migrations for the relevant UUIDs show `1` for `ready_to_complete`, the user can then either:

```sh
$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration complete all" commerce
$ vtctldclient ApplySchema --sql "alter vitess_migration complete all" commerce
```

Assuming these are the only migrations awaiting completion, or, explicitly issue a complete for each of the migrations:

```sh
$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3' complete all" commerce
$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete all" commerce
$ vtctldclient ApplySchema --sql "alter vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3' complete all" commerce
$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete all" commerce
```

## Near instant REVERTs
Expand All @@ -125,14 +135,14 @@ The use case and workflow is as follows:
Consider the following example. We run a 5 hour long migration to drop an index:

```sh
$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess' --sql "alter table customer drop index joined_timestamp_idx" commerce
$ vtctldclient ApplySchema --ddl_strategy='vitess' --sql "alter table customer drop index joined_timestamp_idx" commerce
29231906_776f_11ec_a656_0a43f95f28a3
```

As soon as the migration completes, we run:

```sh
$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "revert vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3'" commerce
$ vtctldclient ApplySchema --ddl_strategy='vitess --postpone-completion --allow-concurrent' --sql "revert vitess_migration '29231906_776f_11ec_a656_0a43f95f28a3'" commerce
3cc4ae0e_776f_11ec_a656_0a43f95f28a3
```

Expand All @@ -141,15 +151,15 @@ The above begins a `REVERT` migration that is open-ended (does not complete), vi
Finally, if we are satisfied that the `drop index` migration went well, we issue:

```sh
$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' cancel" commerce
$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' cancel" commerce
```

That is, we cancel the `REVERT` operation.

Or, should we have not dropped the index? If our migration seems to have been wrong, we run:

```sh
$ vtctldclient ApplySchema --skip_preflight --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete" commerce
$ vtctldclient ApplySchema --sql "alter vitess_migration '3cc4ae0e_776f_11ec_a656_0a43f95f28a3' complete" commerce
```

Which means we want to apply the revert. Since the revert is already running in the background, it is likely that binary log processing is up to date, and cut-over is near instantaneous.
Expand All @@ -171,7 +181,7 @@ Note that there can be scenarios with impossible ordering. Those hardly make sen
An example for in-order submission:

```sh
$ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --allow-concurrent --in-order-completion' --sql "create table t1 (id int primary key); create view v1 as select id from t1;" commerce
$ vtctldclient ApplySchema --ddl_strategy='vitess --allow-concurrent --in-order-completion' --sql "create table t1 (id int primary key); create view v1 as select id from t1;" commerce
```

{{< info >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ a2994c92_f1d4_11ea_afa3_f875a4d24e90

You my run multiple migrations withing the same `ApplySchema` command:
```shell
$ vtctlclient -- ApplySchema --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce
$ vtctlclient -- ApplySchema --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to also change this to vtctldclient?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, and at a couple dozen other places as well.

3091ef2a_4b87_11ec_a827_0a43f95f28a3
```

Expand All @@ -75,7 +75,6 @@ $ vtctlclient -- ApplySchema --skip_preflight --ddl_strategy "vitess" --sql "ALT
- `--ddl_strategy`: by default migrations run directly via MySQL standard DDL. This flag must be aupplied to indicate an online strategy. See also [DDL strategies](../ddl-strategies) and [ddl_strategy flags](../ddl-strategy-flags).
- `--migration_context <unique-value>`: all migrations in a `ApplySchema` command are logically grouped via a unique _context_. A unique value will be supplied automatically. The user may choose to supply their own value, and it's their responsibility to provide with a unique value. Any string format is accepted.
The context can then be used to search for migrations, via `SHOW VITESS_MIGRATIONS LIKE 'the-context'`. It is visible in `SHOW VITESS_MIGRATIONS ...` output as the `migration_context` column.
- `--skip_preflight`: skip an internal Vitess schema validation. When running an online DDL it's recommended to add `--skip_preflight`. In future Vitess versions this flag may be removed or default to `true`.

## Tracking migrations

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ mysql> create table sample_table(id int primary key);
or, via `vtctl`:

```shell
vtctl ApplySchema -- --skip_preflight --ddl_strategy "vitess --allow-concurrent" -sql "REVERT VITESS_MIGRATION '3091ef2a_4b87_11ec_a827_0a43f95f28a3'"
vtctldclient ApplySchema --ddl-strategy "vitess --allow-concurrent" -sql "REVERT VITESS_MIGRATION '3091ef2a_4b87_11ec_a827_0a43f95f28a3'"
```

## Restrictions and eligibility
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,12 @@ mysql> drop table customer;
#### Executing an Online DDL via vtctlclient/ApplySchema

```shell
$ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED" commerce
$ vtctldclient ApplySchema --ddl-strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED" commerce
a2994c92_f1d4_11ea_afa3_f875a4d24e90
```
You my run multiple migrations withing the same `ApplySchema` command:
```shell
$ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce
$ vtctldclient ApplySchema --ddl-strategy "vitess" --sql "ALTER TABLE demo MODIFY id bigint UNSIGNED; CREATE TABLE sample (id int PRIMARY KEY); DROP TABLE another;" commerce
3091ef2a_4b87_11ec_a827_0a43f95f28a3
```

Expand All @@ -150,7 +150,6 @@ $ vtctlclient ApplySchema -- --skip_preflight --ddl_strategy "vitess" --sql "ALT
- `--ddl_strategy`: by default migrations run directly via MySQL standard DDL. This flag must be applied to indicate an online strategy. See also [DDL strategies](../ddl-strategies) and [ddl_strategy flags](../ddl-strategy-flags).
- `-migration_context <unique-value>`: all migrations in a `ApplySchema` command are logically grouped via a unique _context_. A unique value will be supplied automatically. The user may choose to supply their own value, and it's their responsibility to provide with a unique value. Any string format is accepted.
The context can then be used to search for migrations, via `SHOW VITESS_MIGRATIONS LIKE 'the-context'`. It is visible in `SHOW VITESS_MIGRATIONS ...` output as the `migration_context` column.
- `--skip_preflight`: skip an internal Vitess schema validation. When running an online DDL it's recommended to add `--skip_preflight`. In future Vitess versions this flag may be removed or default to `true`.

## Migration flow and states

Expand Down