Slack vitess 9 2021.03.16r0

[setassociative]

Upstream Release Notes

These are the notes for the official release. Later on, the PR's that we explicitly call out are the ones that are not in the notes that we consider interesting / risky for our deployment.

https://github.com/vitessio/vitess/releases/tag/v9.0.0

Slack changes

• Backport: [vtctld] Fix accidentally-broken legacy vtctl output format vitessio/vitess#7292 (on the release): Backport: [vtctld] Fix accidentally-broken legacy vtctl output format
• Fix all occurrences of fmt.Sprint(x) where x is int vitessio/vitess#7244: Formatting fix by amason.
• [vtadmin-web] Query vtadmin-api from vtadmin-web with fetch + react-query vitessio/vitess#7239: VTAdmin beginnings.
• [vtctldclient] vtctldclient generator vitessio/vitess#7238: Boilerplate generator for vtctld client.
• [vtadmin-api] Add static file service discovery implementation vitessio/vitess#7229: VTadmin static pages
• [vtadmin] Add cluster protos to discovery and vtsql package constructors vitessio/vitess#7224: VTAdmin Add cluster protos to discovery and vtsql package constructors
• Add FindAllShardsInKeyspace to vtctldserver vitessio/vitess#7201: Add FindAllShardsInKeyspace to vtctldserver
• [vtadmin-web] The tiniest possible first implementation of vtadmin-web vitessio/vitess#7218: [vtadmin-web] The tiniest possible first implementation of vtadmin-web
• Initial vtadmin-api, clusters, and service discovery vitessio/vitess#7187: Initial vtadmin-api, clusters, and service discovery
• Initial implementation of vtctld service vitessio/vitess#7128: Initial implementation of vtctld service
• https://github.com/vitessio/vitess/pull/7002/files: Better error for query pool timeouts.
• use ephemeral buffer when reading rows vitessio/vitess#6990: use ephemeral buffer when reading rows
• Use Dba user when Vexec is runAsAdmin vitessio/vitess#7731: Use Dba user when Vexec is runAsAdmin

Non Slack changes (interesting)

• [9.0] don't try to compare varchars in vtgate vitessio/vitess#7271: Backport of: While trying to upgrade to v8.0.0 we have noticed that case-insensitive vindexes seem to not be working anymore.
  • [rb] functionally I believe this won't be a net-new change, it's included in the 8.0 build [8.0] don't try to compare varchars in vtgate vitessio/vitess#7270
• Vault Support vitessio/vitess#7233: This seems to add support for Vault secrets on Vitess.
• healthcheck: healthy list should be recomputed when a tablet is removed vitessio/vitess#7176: healthcheck: healthy list should be recomputed when a tablet is removed
• https://github.com/vitessio/vitess/pull/6977/files: Bug which caused the connection to not close in case of error writing an error packet (MySQL protocol)
• backup: Use pargzip instead of pgzip for compression. vitessio/vitess#7037: backup: Use pargzip instead of pgzip for compression (check with Renan)
• Manage read your own writes system settings vitessio/vitess#6871: Manage read your own writes system settings
• healthcheck: use isIncluded correctly to fix replica/rdonly routing bug vitessio/vitess#6904: Fixes to new health check.
• healthcheck: healthy list should be recomputed when a tablet is removed vitessio/vitess#7176: More fixes to new health check.

PR's we should be really really careful:

• https://github.com/vitessio/vitess/pull/7055/files:
  • Cherry picked make sure to not log sensitive information vitessio/vitess#7774 to avoid this issue.
• Refactoring of plan building vitessio/vitess#7054: Big refactor of plan building
• https://github.com/vitessio/vitess/pull/7115/files: Check if we rely on any of these methods. Some vtctld deprecations.
• https://github.com/vitessio/vitess/pull/6783/files: enable client session tracking in mysql protocol
• vttablet: fast and reliable state transitions vitessio/vitess#7011: A big refactor into tablet transitions and how query killer works.
  • [rb] seems fine, minor curiosity about oltp/olap QL division

Features that we haven been wanting

• vttablet: debug/env page to change variables in real-time vitessio/vitess#7189: Dynamic variable changes!

VReplication changes

• Backport 7361: vtctl: Add missing err checks for VReplication v2 vitessio/vitess#7363: vtctl: Add missing err checks for VReplication v2
• Backport 7342: Workflow Show: use timeUpdated to calculate vreplication lag vitessio/vitess#7354: Backport Workflow Show: use timeUpdated to calculate vreplication lag vitessio/vitess#7342: Workflow Show: use timeUpdated to calculate vreplication lag.
• Backport 7297: VStreamer Events: remove preceding zeroes from decimals in Row Events vitessio/vitess#7340: Backport 7297: VStreamer Events: remove preceding zeroes from decimals in Row Events
• Backport 7275: VReplication: Miscellaneous improvements vitessio/vitess#7349: Backport 7275: VReplication: Miscellaneous improvements
• Backport #7255: VReplication DryRun: Report current dry run results for v2 commands vitessio/vitess#7345: Backport VReplication DryRun: Report current dry run results for v2 commands vitessio/vitess#7255: VReplication DryRun: Report current dry run results for v2 commands
• Backport #7276: Vreplication V2 Workflows: rename Abort to Cancel vitessio/vitess#7339: Backport Vreplication V2 Workflows: rename Abort to Cancel vitessio/vitess#7276: Vreplication V2 Workflows: rename Abort to Cancel
• DropSources: change table rename logic vitessio/vitess#7230: DropSources: change table rename logic
• V2 Workflow Start: wait for streams to start and report errors if any while starting a workflow vitessio/vitess#7248: V2 Workflow Start: wait for streams to start and report errors if any while starting a workflow
• New VReplication workflows cli UX. Allow reads/writes to be switched independently vitessio/vitess#7071: These PR seems to change the ergonomics of some of the VReplication commands. We should check with Rohit how this could impact us.
• Make VReplication relay log size & rows configurable. vitessio/vitess#6992: Make VReplication relay log size & rows configurable.
• SwitchWrites bug: reverse replication workflows can have wrong start positions vitessio/vitess#7169: SwitchWrites bug: reverse replication workflows can have wrong start positions (check with Rohit)
• Vtctld Workflow command: minor fixes vitessio/vitess#7008: Vtctld Workflow command: minor fixes
• VRepl/Tablet Picker: improve observability of selected tablet vitessio/vitess#6999: VRepl/Tablet Picker: improve observability of selected tablet
• VDiff: lock keyspace while snapshoting, restart target in case of errors vitessio/vitess#7012: VDiff improvements
• VReplication: refactored and enhanced support for JSON columns vitessio/vitess#6829: Support for JSON columns. Yay!
• Vtctld Workflow Command: fix error state in Workflow Show vitessio/vitess#6970: Vtctld Workflow Command: fix error state in Workflow Show
• https://github.com/vitessio/vitess/pull/6963/files: Binary PK: fix bug where padding of binary columns was being done incorrectly (check with Rohit)
• https://github.com/vitessio/vitess/pull/6892/files: Tool to check vreplication issues in prod.
• VReplication: Set time zone to UTC while streaming rows vitessio/vitess#6845: Issue that created problems with timezones in vreplication

Online schema DDL

• These are explicitly on the release, but still worth mentioning. We can start playing with all the online schema changes functionalities that Shlomi has been working on.

Query parser / routing changes

• Move auto_increment from reserved to non reserved keyword vitessio/vitess#7162: Move auto_increment from reserved to non reserved keyword
• Refactor Show plans vitessio/vitess#7185: Refactor show plans
• [9.0] Reset Session for Reserved Connection when the connection id is not found vitessio/vitess#7544: Fixes a bug in how sessions are reseted when they are marked as reserved. I don't think we rely on this, but is on the query serving path.
• Support for lock and unlock tables vitessio/vitess#7139: Support for lock and unlock tables
• Merge SelectDBA routes when possible vitessio/vitess#7140: Merge SelectDBA routes when possible
• Rewrite joins written with the USING construct vitessio/vitess#6660: Rewrite joins written with the USING construct
• Make sure to check all GROUP BY columns vitessio/vitess#7080: Make sure to check all GROUP BY columns
• separate sub-query and derived table into different structs vitessio/vitess#7081: separate sub-query and derived table into different structs
• UNION DISTINCT support on vtgate vitessio/vitess#7029: UNION DISTINCT support on vtgate
• Improved support for UNION vitessio/vitess#7007: Improved support for UNION
• Route INFORMATION_SCHEMA queries vitessio/vitess#6932: Route INFORMATION_SCHEMA queries (this is very cool!) now we can get access to this tables from vtgate.

Constructing the branch

# find point of divergence between 8 & 9
git merge-base upstream/release-8.0 upstream/release-9.0

# reset to common point to generate a good diff and set a stake in that sha
git checkout 098845592f16d235cb440bdafe070b4dcc0eb07b
git checkout -b slack-vitess-9-2021.03.16r0

# start the branch to generate this PR / the diff
git checkout -b slack-vitess-9-2021.03.16r0.wip

# set that branch to be what we want to deploy
git reset --hard upstream/release-9.0

# cherry-pick some changes

Cherry Picked Changes

$ git llog upstream/release-9.0...slack-vitess-9-2021.03.16r0.wip
2021-03-15 - 7f8a0bbb5 - setassociative - Fix bug in vtgate when running with keyspace filtering (#199)
2021-03-09 - 8da0a142f - setassociative - Backport vitessio#7523 into 8.0 (#198)
2021-03-08 - 60efc9139 - setassociative - Backport some panic protection during ERS (#196)
2020-03-05 - 00381f49f - mdemmer - add a hack to enable routing to RDONLY tablets for REPLICA queries
2019-02-25 - 3d3f66091 - setassociative - Patch sqltypes.MakeRowTrusted to lean on field count (#126)

[rafael]

LGTM