VSCopy: Resume the copy phase consistently from given GTID and lastpk

[yoheimuta]

Description

The current vstream API doesn't resume a copy phase given the below VGTID.
Instead, it starts replicating the table from the given GTID while ignoring the lastpk information.

vgtid {
  shard_gtids {
    keyspace: "test_sharded_keyspace"
    shard: "-80"
    gtid: "MySQL56/ff540f7a-1d4d-11ed-93b5-0242ac110002:1-28"
    table_p_ks {
      table_name: "t1"
      lastpk {
        rows {
          lengths: 4
          values: "1000"
 ...

This PR allows vstream consumers to deal with the VGTID event as an opaque offset.
Given the non-empty GTID and the lastpk, the API is supposed to do the following:

1. Catch up the t1 table from "MySQL56/ff540f7a-1d4d-11ed-93b5-0242ac110002:1-28" to the nearly latest
2. Copy the t1 table with the condition "pk > 1000"

Related Issue(s)

Discussion: https://vitess.slack.com/archives/C0PQY0PTK/p1661155731380559

Implementation

Modify uvstreamer Init

Allow a non-empty position when tablePKs are specified. In this case an initlal catchup+fastforward phase will run before restarting copy phase on incompletely copied tables. (reference: https://vitess.io/docs/design-docs/vreplication/life-of-a-stream/)

Change how we filter events

Since we now run a catchup phase when we restart table copies, it is possible that duplicate events can be sent for rows that are not already copied, since we don't yet have logic to compare primary keys. To minimise this we filter out events for tables whose copy phase has not yet started.

Test changes

1. New test TestVStreamCopyResume: passes both tablePKs and a specific position to validate new logic
   https://github.com/vitessio/vitess/pull/11103/files?diff=unified&w=1#diff-c98b8f318aac471272bf650c7824713c9664b46895380acf322ec3498a2fe578R236

2. Updated test TestVStreamCopyCompleteFlow to account for change in logic for selecting events to send

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation PR: Document VStream website#1216

Pending issues

1. Filter out events for rows not copied, for tables whose primary keys can be compared in code (essentially those that our eval engine can compare). This is not critical, since clients have to handle duplicate events in any case.
2. Batch the copy phase (like we do for vreplication) so that we don't end up with a large vreplication lag (and possibly rotated binlogs). We didn't expect this to be critical since we are just streaming events, rather than inserting rows on the target, like we do for vreplication. But it is possible that some clients do more cpu/IO-intensive operations than just sending them to event sinks.

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

• Is it really necessary to add this flag?
• Flag names should be clear and intuitive (as far as possible)
• Help text should be descriptive.
• Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow should be required, the maintainer team should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from VTop, if used there.

[mattlord]

I think that we need to find a way to implement this high level feature w/o changing the basic copy phase event filtering for all VStreams in uvstreamer (the unified vstreamer abstraction/primitive that everything else uses) as this changes key behavior for VReplication workflows and other things.

I also don't think that change should be necessary as we should be able to mirror the VReplication MoveTables/Reshard behavior here. @rohit-nayak-ps and I can discuss further.

[yoheimuta]

@rohit-nayak-ps Thank you for your updates!
I reviewed it, and looks good to me 👍

[mattlord]

Looks good and makes sense to me! I only had some very minor question/comments/suggestions.

I'll push a commit now to mention this new feature in the 16.0 release notes.

We should also add a new VStream page in the 16.0 VReplication docs that can reference the existing copy table design doc. There we can touch on what it is, what use cases it serves, how it works and how to use it. We don't have any docs on the VTGate VStream API feature today (only the somewhat dated design doc) — so many people don't even know that it exists. I can get a PR started for that and add a reference in the description.

[mattlord]

My review comments were extremely minor -- really only comment related -- so I'm going to approve now that I pushed an update to the release notes and started on the docs work (which is now linked): vitessio/website#1216

Thank you so much for the time you spent on this @yoheimuta !

[rohit-nayak-ps]

lgtm

[systay]

some nit-picks, but they are not stopping this PR