6.0 Slack vitess release merge (w/consul 1.5.1, no k3s)

[ajm188]

Git Shenanigans

To avoid reverting back to consul 1.4.5, but also without needing to go all the way to 1.8.0, we needed to pull out the k8s topo bits, which were done in the following commits:

• fd3d73f
• 257355a (the actual revert)
• f6f2cfa (follow up)
• b2d8d6d (follow up)

Slack Changes

• 🎉 commit vitessio@924824b, zmagg added PR Pass in cell to EnsureVSchema and GetOrCreateShard vitessio/vitess#5930 fixes EnsureVSchema and GetOrCreateShard to only rebuild SrvVSchema in the single cell 🎉
• 🎉 commit vitessio@5dc35b0, amason added PR Retry "use of closed network connection" in s3 backup handle vitessio/vitess#6182 adding custom retryer for closed network connections in s3 backups (this was then backported in Backport s3 retry from #6182 to release branch vitessio/vitess#6199) 🎉
• 🎉 Backport upgrade pgzip #6189 to release branch vitessio/vitess#6198 backport of pgzip upgrade 🎉
• commit ffb2b89, arka added PR Update query counts for stream queries from vtgate vitessio/vitess#6057 adding query counts for vtgate streaming
• commit a40c5e7, arka added PR Stream execute query logging vitessio/vitess#6056 adding query logging for vtgate streaming
• commit 04f66d3, rafael added PR Adds supports to target DestinationKeyspaceID to handle other vitessio/vitess#6040 adds support for DestinationKeyspaceID to handleOther
• commit 3519b08, rafael added PR Adds Autocommit to V2 API vitessio/vitess#5909 adds autocommit to V2 API
• commit vitessio@4134a82, rafael added PR Improve error codes returned by tabletserver vitessio/vitess#6012 to return clearer error codes from tabletserver
• Log when we fail to accept a conn vitessio/vitess#5881, serry added connection failure logging
• Healthcheck - limit healthchecks by keyspace vitessio/vitess#5815 serry added healthchecking by keyspace

Upstream Changes

PLANNED REPARENT FIXES

When updating from a version before this PR to a version after it, it is critical that you follow the recommended upgrade order. In particular, you must upgrade all the vttablets in the cluster before upgrading any of the vtctlds.

• ⚠️ 🎉 ⚠️ PlannedReparent fixes vitessio/vitess#6050 ⚠️ 🎉 ⚠️
• PlannedReparent: create a new context for calling UndoDemoteMaster vitessio/vitess#5841 separate contexts for UndoDemoteMaster, thanks Deepthi!

query

• commit vitessio@90de2db, sougou added PR pools: add timeouts for all conn pools vitessio/vitess#6063 imposing timeouts on all connection pools
• commit vitessio@3b68965, saifalharthi added PR Fix SHOW FULL COLUMNS for unqualified tables vitessio/vitess#6078 fixing SHOW FULL COLUMNS on unqualified tables
• ⚠️ commit vitessio@cc4f257, saifalharthi added PR VTGate Expression Evaluation vitessio/vitess#6042 refactoring vtgate expression evaluation ⚠️
• commit vitessio@a2abb6c, systay added PR Minor refactoring of vtgate plans vitessio/vitess#6090 minor refactor vtgate query plans
• commit vitessio@5c7a0ca, bumbing added PR Check zero for avgTimePerQuery vitessio/vitess#5904 fixing divby0 in avg query time stat
• 🚀 commit vitessio@752c017, harshit-gangal added PR Improve Update and Delete Query Routing for IN Clause vitessio/vitess#6116 improving update/delete routing for IN clauses 🚀
• Plan Generation for explain analyze describe optimize repair statements  vitessio/vitess#6055 harshit-gangal added plan generation for explain/analyze/describe/optimize/repair statements
• backport: Get cached vschema from executor vitessio/vitess#6220 backport getting cached vschema from executor (performance regression)
• backport #6224 to 6.0 release branch vitessio/vitess#6319 backport escaping column names in DEFAULT
• backport #6299 to 6.0 release branch vitessio/vitess#6321 backport rolling back auto-committable transactions on query errors
• Backport #6270 to 6.0 release branch vitessio/vitess#6324 backport making multi-line inserts more robust against bindvar collisions
• Make sure to protect insert lookup queries from bind var name clashes (6.0) vitessio/vitess#6327 backport protecting insert lookup queries from bindvar name clashes
• https://github.com/vitessio/vitess/pull/6047/files harshit-gangal removed handleUse from execute
• Do not cache plan with nil instructions vitessio/vitess#6045 harshit-gangal changed vtgates to not cache query plans with nil instructions
• systay added PR Make sure to only accept DDL that can be parsed, at least partially vitessio/vitess#6023 to only accept DDL that can be at least partially parsed
• Add SHOW INDEXES and EXTENDED vitessio/vitess#6006 saifalharthi added support for SHOW INDEXES and EXTENDED
• Support handling signed integers in Prepared Statements vitessio/vitess#5974 saifalharthi adds support for signed integers in prepared statements
• vtgate: list only the problem values when insert fails due to vindex.Verify vitessio/vitess#5896 bumbing changed vtgate to list only problem values when insert fails due to vindex

topo

• commit vitessio@1397484, harshit-gangal added PR use topo method for parsing target string on set target method call vitessio/vitess#6096 uses topo methods for parsing target strings
• add recover in WatchSrvVSchema and GetSrvKeyspaceNames vitessio/vitess#6051 xhh1989 added recovery in WatchSrvVSchema and GetSrvKeyspaceNames

vttablet

• 🎉 commit vitessio@ac211ce, sougou added PR vttablet: fix panic in NewDBConnNoPool vitessio/vitess#6086 fixing panic in NewDBConnNoPool 🎉
• commit vitessio@0a913c7, sougou added PR messager: fix bug in build row vitessio/vitess#6091 fixing bug in tabletmanager build row
• commit vitessio@19d215e, sougou added PR yaml: phase 1 vitessio/vitess#6070 implementing yaml params for tablet servers
• commit vitessio@ab13a0f, derekperkins added PR messager: add 33% jitter to postpone backoff vitessio/vitess#6092 adding jitter to postpone backoffs
• commit vitessio@e9f07a4, sougou added PR yaml: phase 2 DBConfigs vitessio/vitess#6094 covering yaml dbconfigs part 2
• commit vitessio@66386cc, derekperkins added PR messenger: use ackWaitTimeout in postpone query vitessio/vitess#6114 using ackWaitTimeout in postpone queries
• Backport of USE and DATABASE() fix vitessio/vitess#6158 backport of USE and DATABASE() fixes
• Backport tablet timeout issue vitessio/vitess#6197 backport tablet timeout issue
• backport #6250 to 6.0 release branch vitessio/vitess#6320 backport yaml type change (float64 for timeouts)
• tabletserver: refactor parameters vitessio/vitess#6020 sougou refactors tabletserver params
• log during init tablet vitessio/vitess#6004 harshit-gangal adds logging during init tablet
• tabletserver: SBR deprecation vitessio/vitess#5940 sougou deprecated SBR in tabletserver
• tablet query engine: introduce -enable-consolidator-replicas vitessio/vitess#5862 dweitzman adds flags to prevent OLAP queries from being run on masters

vtgate (some of the things under query are arguably vtgate 🤷‍♂️ )

• resurrect gateway interface and command line option vitessio/vitess#5992 deepthi brings back the gateway interface, defaulting to DiscoveryGateway
• Store String as Varbinary in BindVars vitessio/vitess#6054 harshit-gangal changes vtgates to store strings as varbinary in BindVars internally

vtctld

• commit 0e74446, sougou added PR vtctld: proxy must update target URLs vitessio/vitess#6062 ensuring vtctld reverse proxy URLs update when tablet URLs do
• commit vitessio@0fd8aac, sougou added PR topo: exit if topo info is bad vitessio/vitess#6119 to exit immediately if topo info is bad
• commit vitessio@0f4e80e, sougou added PR vtctld: add /debug/status to tablet URLs vitessio/vitess#6113 adding /debug/status to tablet URLs
• commit vitessio@bec4139, sougou added PR vtctld: make cell optional vitessio/vitess#6111 allowing "cell" optional when initializing a vtctld api
• Issue #5265 ; add a --skip-verify option to CopySchemaShard. vitessio/vitess#5964 aquarapid added --skip-verify to CopySchemaShard
• vtctld: flag to proxy vttablet URLs vitessio/vitess#6058 sougou added -proxy_tablets to vtctld
• vtctl: Add -force flag to DeleteCellInfo. vitessio/vitess#6013 enisoc adds -force flag to DeleteCellInfo
• Vreplication workflow renaming vitessio/vitess#5968 rohit-nayak-ps renames vrep workflow commands
• add 'keyspace exist' check when ApplyVschema vitessio/vitess#5931 Johnny-Three adds keyspace existence check when doing ApplyVSchema

vrep

• commit vitessio@90a5557, sougou added PR VReplication: Switch Reads/Writes dry-run + DropSources vitessio/vitess#6018 adding switch read/writes dry-run and dropsources
• commit vitessio@0f1551a, rohit-nayak-ps added PR DropSources: delete reverse streams on source vitessio/vitess#6121 DropSources now deletes reverse streams on source
• Vcursor refactor vitessio/vitess#5982 harshit-gangal refactors vcursor
• Filtered Vreplication: support int and varbinary filters for columns vitessio/vitess#5949 rohit-nayak-ps adds support for int/varbinary filters in vrep
• vstreamer: send immediate GTID on "current" vitessio/vitess#5891 more vstreamer changes

backups

• add option to pass in flags to xtrabackup prepare phase vitessio/vitess#5954 deepthi adds option to pass flags to xtrabackup prepare phase

metrics

• Added resource pool wait time histogram metrics vitessio/vitess#5727 Icabancia adds resource pool wait time to histogram metrics

session variables

• commit vitessio@446419e, harshit-gangal added PR Plan for Set Statement with User Defined Variables vitessio/vitess#6035 preparing for set statements with user-defined variables
• commit vitessio@48020c8, harshit-gangal added PR Planning Set Statement for System Variable with Ignore and CheckAndIgnore vitessio/vitess#6069 preparing for set statements with system variables
• commit vitessio@7589230, systay added PR Set parser update vitessio/vitess#6080 updating the set statement parser
• commit vitessio@f7c6d8a, systay added PR Allow SELECT options in any order vitessio/vitess#6082 allowing sql_modes to be set in any order (like mysql)
• commit vitessio@b3d981e, harshit-gangal added PR Set CheckAndIgnore to Log Warn than Error on failing the SysVar settings check vitessio/vitess#6099 error handling on sysvar settings check
• Implement plan description for vindex_func vitessio/vitess#6108 vindex_func implementation
• Allow many more expressions for SET user defined variables vitessio/vitess#6093 more set UDV implementations
• fix udv: return null type for non-existing udv vitessio/vitess#6308 return null type for non-existent udvs
• backport #6308 to 6.0 release branch vitessio/vitess#6323 backport

olap

• commit vitessio@5a34157, dweitzman added PR vtgate send primitive: implement StreamExecute vitessio/vitess#6077 fixing olap for queries run after a use keyspace[shard]
• commit vitessio@3de43f6, adsr added PR Fix panic with select database() in OLAP mode (#6200) vitessio/vitess#6205 fixes panic with select database() in olap mode (backported in backport #6205 to 6.0 release branch vitessio/vitess#6318)

heartbeat

• heartbeat: create table only on master with binlog ON, use app_user not dba_user vitessio/vitess#6053 deepthi added creating heartbeat tables on master only

vtexplain

• Fix Vtexplains StripComments vitessio/vitess#5984 saifalharthi fixes strip comments

java

• commit vitessio@228e6fe, rmarsch added PR Remove synchronizing VTGateConnection on itself vitessio/vitess#6088 removing code that synchronized VTGateConnection on itself

Misc

• Remove unused code vitessio/vitess#6064 removes some MarshalJSON implementations
• helm: use new commands vitessio/vitess#6052 helm stuff
• Harmonize local examples with helm improvements vitessio/vitess#6072 update local examples to match helm
• [K8s Topo] Move CRD to v1beta1 vitessio/vitess#6074 k8s helm
• Refactor CRD to support K8s 1.15 specific requirements vitessio/vitess#6075 more k8s
• Reduce race in legacy_local_example vitessio/vitess#6076 fix race in local example
• Test wordpress install vitessio/vitess#5898 wordpress
• Use apps/v1 for all statefulsets vitessio/vitess#6083 more k8s
• Add sql_mode to cnf so wordpress can create tables vitessio/vitess#6081 wordpress
• Remove unused code vitessio/vitess#6079 dead code removal
• Test cleanup vitessio/vitess#6095 test cleanup
• test: deflake acl test vitessio/vitess#6101 test flake fix
• Add unit test for go/jsonutil/json.go vitessio/vitess#6073 unit tests for jsonutil
• Misc cleanups vitessio/vitess#6105 misc cleanups
• Fixed error message on database ddl query execution vitessio/vitess#6109 fix error message in database ddl
• Local example: Drop commerce tables after they are moved to the customer keyspace vitessio/vitess#6106 local example changes
• helm: use proxy_tablets vitessio/vitess#6112 helm proxy_tablets
• vitess/lite images based on Red Hat UBI 7 vitessio/vitess#6100 vitess images based on redhat
• Fix automated release script vitessio/vitess#6122 fix release script
• Backport fix for make-release-package vitessio/vitess#6160 backport packaging script fixes
• backport #6317 to 6.0 release branch vitessio/vitess#6322 backport version bumps
• Messaging testcase flakyness remove vitessio/vitess#6038 test flakes
• Fix the initial sharding flaky test vitessio/vitess#6043 test flakes
• Fixes for flaky tests: races in TestPlayerDDL and TestTabletVStreamerrClientVStreamRows vitessio/vitess#6061 test flakes
• VCopier Flaky Test: modify how we validate expected queries vitessio/vitess#6033 test flakes
• region_json bug fix vitessio/vitess#6017 bugfix
• Vreplication flaky test: make vcopier test non-flaky vitessio/vitess#6001 test flakes
• tabletserver: fix broken test vitessio/vitess#5944 fix test

[setassociative]

This all seems fine. Some things to call out though:

• heartbeat: create table only on master with binlog ON, use app_user not dba_user vitessio/vitess#6053 I think in the cases where we see a failover that leaves one tablet thinking it's still the primary this could generate ErrantTxns on the failed primary. Also, to be clear this should be safe; it's only the alerts I've got in mind as a concern.

• Improve Update and Delete Query Routing for IN Clause vitessio/vitess#6116 This is great, we should watch for decreased scatters before/after deploy; probably wouldn't hurt to smoke test this either

• use topo method for parsing target string on set target method call vitessio/vitess#6096 I would not be super surprised if we see some changes to reported keyspace in agenda

And the following sorta worry me only in that the are large / complex with minimal discussion (and I don't know the codebase well enough to grok them).. but presumably it should be somewhat obvious if they fail 😓

• Set parser update vitessio/vitess#6080
• Vcursor refactor vitessio/vitess#5982