New VReplication workflows cli UX. Allow reads/writes to be switched independently #7071
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rohit-nayak-ps
force-pushed
the
rn-switch-order
branch
from
4cb151e to
c1c7a24
Compare
rohit-nayak-ps
changed the title
rohit-nayak-ps
changed the title
sougou
reviewed
Dec 24, 2020
Signed-off-by: Rohit Nayak <[email protected]>
…ble. Setup routing rules for new flow Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
…w Progress, ReverseReads/Writes Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
…raffic/ReverseTraffic instead of SwitchWrites/SwitchReads, added Complete (similar to DropSources) and Abort (drop targets) commands Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
…st actions Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
rohit-nayak-ps
force-pushed
the
rn-switch-order
branch
from
9296809 to
2a7a492
Compare
…: vtgate seems to be quoting some characters now Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
rohit-nayak-ps
changed the title
rohit-nayak-ps
requested review from
ajm188,
deepthi,
doeg,
harshit-gangal,
shlomi-noach and
systay
as code owners
Signed-off-by: Rohit Nayak <[email protected]>
sougou
approved these changes
Jan 3, 2021
|
|
||
| _ = subFlags.Bool("v2", true, "") | ||
|
|
||
| _ = dryRun //TODO: add dry run functionality |
There was a problem hiding this comment.
I saw some code for dry run. Do you just have to pass the flag in?
There was a problem hiding this comment.
Most of the original dry run code will still apply here, but wanted to review and update it if necessary for the new wrappers as a separate PR that should be ready this week.
Signed-off-by: Rohit Nayak <[email protected]>
This was referenced Jan 4, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The motivation and design for this issue are outlined in the RFC at #7225. Some implementation details are provided below: more detailed documentation will be added to the vitess.io website soon.
The main change is that all subcommands for a MoveTables or Reshard workflow is routed through the new VReplicationWorkflow object. VReplicationWorkflow first deduces the state of a workflow from the parameters in the _vt sidecar and topo and invokes the associated wrangler methods. The only changes made in wrangler related to this PR is to allow switching tablet types in any order.
vtctld mods
MoveTables and Reshard commands now take a -v2 parameter which causes the new functionality to be invoked. These will continue to run side-by-side with the current functionality. We will first deprecate v1 and remove v1-only code in a subsequent release making v2 functionality as default.
Routing rule changes
MoveTables v1 uses routing rules to tell vtgate, which keyspace to route queries to. In v1 we always need to run SwitchReads before SwitchWrites. MoveTables sets up routing rules pointing primary/replica/rdonly tables to the source keyspace. SwitchReads would change the routing rules for related tablet types to point to the target and reversal point it back to the source. SwitchWrites would delete all rules except the one for the table.
Since tablet types can now be switched (and reversed) in any order we keep the routing rules through the lifetime of the workflow. We keep the routing rules in place and currently DropSources does not delete them. This PR continues to keep the routing rules. A follow-up PR will add the removal of routing rules to the Complete/Abort subcommands.
Abort
It is possible to abort and cleanup a created workflow as long as traffic has not been switched. This deletes the vreplication artifacts and optionally, delete data: target tables/shards.
Progress
We report the progress of a workflow by showing the percentage of data copied across targets, if workflow is in copy state, and the lag between the last source event after the copy phase is completed. It is too expensive to get actual row counts of tables, so we use the statistics available in information_schema to approximate copy progress. This data can be significantly off (upto 50-60%) depending on the utilization of the underlying mysql server resources.
Notes:
Todos: upcoming PRs will achieve these