Merge pull request #123 from tinyspeck/slack-sync-upstream-2018-12-21.r1

Slack sync upstream 2018 12 21.r1

bootstrap.sh

@@ -307,7 +307,7 @@ fi
 case "$MYSQL_FLAVOR" in
   "MySQL56")
     myversion="$("$VT_MYSQL_ROOT/bin/mysql" --version)"
-    [[ "$myversion" =~ Distrib\ 5\.[67] ]] || fail "Couldn't find MySQL 5.6+ in $VT_MYSQL_ROOT. Set VT_MYSQL_ROOT to override search location."
+    [[ "$myversion" =~ Distrib\ 5\.[67] || "$myversion" =~ Ver\ 8\. ]] || fail "Couldn't find MySQL 5.6+ in $VT_MYSQL_ROOT. Set VT_MYSQL_ROOT to override search location."
     echo "Found MySQL 5.6+ installation in $VT_MYSQL_ROOT."
     ;;
 

doc/GettingStarted.md

@@ -271,7 +271,7 @@ In addition, Vitess requires the software and libraries listed below.
 
     ``` sh
     # Remaining commands to build Vitess
-    . ./dev.env
+    source ./dev.env
     make build
     ```
 

doc/ServerConfiguration.md

@@ -552,6 +552,7 @@ Load-balancer in front of vtgate to scale up (not covered by Vitess). Stateless,
 ### Parameters
 
 * **cells_to_watch**: which cell vtgate is in and will monitor tablets from. Cross-cell master access needs multiple cells here.
+* **keyspaces_to_watch**: Specifies that a vtgate will only be able to perform queries against or view the topology of these keyspaces
 * **tablet_types_to_wait**: VTGate waits for at least one serving tablet per tablet type specified here during startup, before listening to the serving port. So VTGate does not serve error. It should match the available tablet types VTGate connects to (master, replica, rdonly).
 * **discovery_low_replication_lag**: when replication lags of all VTTablet in a particular shard and tablet type are less than or equal the flag (in seconds), VTGate does not filter them by replication lag and uses all to balance traffic.
 * **degraded_threshold (30s)**: a tablet will publish itself as degraded if replication lag exceeds this threshold. This will cause VTGates to choose more up-to-date servers over this one. If all servers are degraded, VTGate resorts to serving from all of them.

doc/TabletRouting.md

@@ -93,36 +93,7 @@ There are two implementations of the Gateway interface:
   discovery section, one per cell) as a source of tablets, a HealthCheck module
   to watch their health, and a TabletStatsCache to collect all the health
   information. Based on this data, it can find the best tablet to use.
-* l2VTGateGateway: It keeps a map of l2vtgate processes to send queries to. See
-  next section for more details.
 
-## l2vtgate
-
-As we started increasing the number of tablets in a cell, it became clear that a
-bottleneck of the system was going to be how many tablets a single vtgate is
-connecting to. Since vtgate maintains a streaming health check connection per
-tablet, the number of these connections can grow to large numbers. It is common
-for vtgate to watch tablets in other cells, to be able to find the master
-tablet.
-
-So l2vtgate came to exist, based on very similar concepts and interfaces:
-
-* l2vtgate is an extra hop between a vtgate pool and tablets.
-* A l2vtgate pool connects to a subset of tablets, therefore it can have a
-  reasonable number of streaming health connections. Externally, it exposes the
-  QueryService RPC interface (that has the Target for the query, keyspace /
-  shard / tablet type). Internally, it uses a discoveryGateway, as usual.
-* vtgate connects to l2vtgate pools (using the l2VTGateGateway instead of the
-  discoveryGateway). It has a map of which keyspace / shard / tablet type needs
-  to go to wich l2vtgate pool. At this point, vtgate doesn't maintain any health
-  information about the tablets, it lets l2vtgate handle it.
-
-Note l2vtgate is not an ideal solution as it is now. For instance, if there are
-two cells, and the master for a shard can be in either, l2vtgate still has to
-watch the tablets in both cells, to know where the master is. Ideally, we'd want
-l2vtgate to be collocated with the tablets in a given cell, and not go
-cross-cell.
-
 # Extensions, work in progress
 
 ## Regions, cross-cell targeting
@@ -169,31 +140,6 @@ between vtgate and l2vtgate:
 This would also be a good time to merge the vtgate code that uses the VSchema
 with the code that doesn't for SrvKeyspace access.
 
-## Hybrid Gateway
-
-It would be nice to re-organize the code a bit inside vtgate to allow for an
-hybrid gateway, and get rid of l2vtgate alltogether:
-
-* vtgate would use the discoveryGateway to watch the tablets in the current cell
-  (and optionally to any other cell we still want to consider local).
-* vtgate would use l2vtgateGateway to watch the tablets in a different cell.
-* vtgate would expose the RPC APIs currently exposed by the l2vtgate process.
-
-So vtgate would watch the tablets in the local cell only, but also know what
-healthy tablets are in the other cells, and be able to send query to them
-through their vtgate. The extra hop to the other cell vtgate should be a small
-latency price to pay, compared to going cross-cell already.
-
-So queries would go one of two routes:
-
-* client(cell1) -> vtgate(cell1) -> tablet(cell1)
-* client(cell1) -> vtgate(cell1) -> vtgate(cell2) -> tablet(cell2)
-
-If the number of tablets in a given cell is still too high for the local vtgate
-pool, two or more pools can still be created, each of them knowing about a
-subset of the tablets. And they would just forward queries to each others when
-addressing the other tablet set.
-
 ## Config-based routing
 
 Another possible extension would be to group all routing options for vtgate in a

docker/lite/Dockerfile.alpine

@@ -4,8 +4,8 @@ FROM alpine:3.8 AS staging
 
 RUN mkdir -p /vt/vtdataroot/ && mkdir -p /vt/bin && mkdir -p /vt/src/vitess.io/vitess/web/vtctld2
 
-COPY --from=builder /vt/src/vitess.io/vitess/web/vtctld /vt/src/vitess.io/web/vtctld
-COPY --from=builder /vt/src/vitess.io/vitess/web/vtctld2/app /vt/src/vitess.io/web/vtctld2/app
+COPY --from=builder /vt/src/vitess.io/vitess/web/vtctld /vt/src/vitess.io/vitess/web/vtctld
+COPY --from=builder /vt/src/vitess.io/vitess/web/vtctld2/app /vt/src/vitess.io/vitess/web/vtctld2/app
 COPY --from=builder /vt/src/vitess.io/vitess/config /vt/config
 COPY --from=builder /vt/bin/mysqlctld /vt/bin/
 COPY --from=builder /vt/bin/vtctld /vt/bin/

examples/local/vtgate-up.sh

@@ -55,10 +55,11 @@ then
 fi
 
 optional_auth_args='-mysql_auth_server_impl none'
+optional_grpc_auth_args=''
 if [ "$1" = "--enable-grpc-static-auth" ];
 then
 	  echo "Enabling Auth with static authentication in grpc"
-    optional_auth_args='-grpc_auth_static_client_creds ./grpc_static_client_auth.json'
+    optional_grpc_auth_args='-grpc_auth_static_client_creds ./grpc_static_client_auth.json'
 fi
 
 if [ "$1" = "--enable-mysql-static-auth" ];
@@ -84,6 +85,7 @@ $VTROOT/bin/vtgate \
   -service_map 'grpc-vtgateservice' \
   -pid_file $VTDATAROOT/tmp/vtgate.pid \
   $optional_auth_args \
+  $optional_grpc_auth_args \
   $optional_tls_args \
   > $VTDATAROOT/tmp/vtgate.out 2>&1 &
 

go/cmd/topo2topo/topo2topo.go

@@ -18,6 +18,8 @@ package main
 
 import (
 	"flag"
+	"fmt"
+	"os"
 
 	"golang.org/x/net/context"
 	"vitess.io/vitess/go/exit"
@@ -36,6 +38,7 @@ var (
 	toServerAddress  = flag.String("to_server", "", "topology server address to copy data to")
 	toRoot           = flag.String("to_root", "", "topology server root to copy data to")
 
+	compare             = flag.Bool("compare", false, "compares data between topologies")
 	doKeyspaces         = flag.Bool("do-keyspaces", false, "copies the keyspace information")
 	doShards            = flag.Bool("do-shards", false, "copies the shard information")
 	doShardReplications = flag.Bool("do-shard-replications", false, "copies the shard replication information")
@@ -64,6 +67,14 @@ func main() {
 
 	ctx := context.Background()
 
+	if *compare {
+		compareTopos(ctx, fromTS, toTS)
+		return
+	}
+	copyTopos(ctx, fromTS, toTS)
+}
+
+func copyTopos(ctx context.Context, fromTS, toTS *topo.Server) {
 	if *doKeyspaces {
 		helpers.CopyKeyspaces(ctx, fromTS, toTS)
 	}
@@ -76,4 +87,37 @@ func main() {
 	if *doTablets {
 		helpers.CopyTablets(ctx, fromTS, toTS)
 	}
+
+}
+
+func compareTopos(ctx context.Context, fromTS, toTS *topo.Server) {
+	var err error
+	if *doKeyspaces {
+		err = helpers.CompareKeyspaces(ctx, fromTS, toTS)
+		if err != nil {
+			log.Exitf("Compare keyspaces failed: %v", err)
+		}
+	}
+	if *doShards {
+		err = helpers.CompareShards(ctx, fromTS, toTS)
+		if err != nil {
+			log.Exitf("Compare shards failed: %v", err)
+		}
+	}
+	if *doShardReplications {
+		err = helpers.CompareShardReplications(ctx, fromTS, toTS)
+		if err != nil {
+			log.Exitf("Compare shard replications failed: %v", err)
+		}
+	}
+	if *doTablets {
+		err = helpers.CompareTablets(ctx, fromTS, toTS)
+		if err != nil {
+			log.Exitf("Compare tablets failed: %v", err)
+		}
+	}
+	if err == nil {
+		fmt.Println("Topologies are in sync")
+		os.Exit(0)
+	}
 }

go/mysql/schema.go

@@ -308,8 +308,8 @@ func ShowIndexFromTableRow(table string, unique bool, keyName string, seqInIndex
 		sqltypes.MakeTrusted(sqltypes.VarChar, []byte(columnName)),
 		sqltypes.MakeTrusted(sqltypes.VarChar, []byte("A")), // Collation
 		sqltypes.MakeTrusted(sqltypes.Int64, []byte("0")),   // Cardinality
-		sqltypes.NULL,                                       // Sub_part
-		sqltypes.NULL,                                       // Packed
+		sqltypes.NULL, // Sub_part
+		sqltypes.NULL, // Packed
 		sqltypes.MakeTrusted(sqltypes.VarChar, []byte(nullableStr)),
 		sqltypes.MakeTrusted(sqltypes.VarChar, []byte("BTREE")), // Index_type
 		sqltypes.MakeTrusted(sqltypes.VarChar, []byte("")),      // Comment

go/mysql/server.go

@@ -21,6 +21,7 @@ import (
 	"fmt"
 	"io"
 	"net"
+	"strings"
 	"time"
 
 	"vitess.io/vitess/go/netutil"
@@ -273,7 +274,9 @@ func (l *Listener) handle(conn net.Conn, connectionID uint32, acceptTime time.Ti
 	// First build and send the server handshake packet.
 	salt, err := c.writeHandshakeV10(l.ServerVersion, l.authServer, l.TLSConfig != nil)
 	if err != nil {
-		log.Errorf("Cannot send HandshakeV10 packet to %s: %v", c, err)
+		if err != io.EOF {
+			log.Errorf("Cannot send HandshakeV10 packet to %s: %v", c, err)
+		}
 		return
 	}
 
@@ -547,6 +550,12 @@ func (c *Conn) writeHandshakeV10(serverVersion string, authServer AuthServer, en
 	}
 
 	if err := c.writeEphemeralPacket(); err != nil {
+		if strings.HasSuffix(err.Error(), "write: connection reset by peer") {
+			return nil, io.EOF
+		}
+		if strings.HasSuffix(err.Error(), "write: broken pipe") {
+			return nil, io.EOF
+		}
 		return nil, err
 	}
 

go/sqltypes/query_response.go

@@ -16,7 +16,9 @@ limitations under the License.
 
 package sqltypes
 
-import "reflect"
+import (
+	"vitess.io/vitess/go/vt/vterrors"
+)
 
 // QueryResponse represents a query response for ExecuteBatch.
 type QueryResponse struct {
@@ -34,7 +36,7 @@ func QueryResponsesEqual(r1, r2 []QueryResponse) bool {
 		if !r.QueryResult.Equal(r2[i].QueryResult) {
 			return false
 		}
-		if !reflect.DeepEqual(r.QueryError, r2[i].QueryError) {
+		if !vterrors.Equals(r.QueryError, r2[i].QueryError) {
 			return false
 		}
 	}

go/vt/automation/scheduler.go

@@ -82,11 +82,11 @@ func NewScheduler() (*Scheduler, error) {
 		registeredClusterOperations:    defaultClusterOperations,
 		idGenerator:                    IDGenerator{},
 		toBeScheduledClusterOperations: make(chan ClusterOperationInstance, 10),
-		state:                     stateNotRunning,
-		taskCreator:               defaultTaskCreator,
-		pendingOpsWg:              &sync.WaitGroup{},
-		activeClusterOperations:   make(map[string]ClusterOperationInstance),
-		finishedClusterOperations: make(map[string]ClusterOperationInstance),
+		state:                          stateNotRunning,
+		taskCreator:                    defaultTaskCreator,
+		pendingOpsWg:                   &sync.WaitGroup{},
+		activeClusterOperations:        make(map[string]ClusterOperationInstance),
+		finishedClusterOperations:      make(map[string]ClusterOperationInstance),
 	}
 
 	return s, nil

go/vt/binlog/binlogplayer/binlog_player_test.go

@@ -37,8 +37,8 @@ var (
 		InsertID:     0,
 		Rows: [][]sqltypes.Value{
 			{
-				sqltypes.NewVarBinary("MariaDB/0-1-1083"),    // pos
-				sqltypes.NULL,                                // stop_pos
+				sqltypes.NewVarBinary("MariaDB/0-1-1083"), // pos
+				sqltypes.NULL, // stop_pos
 				sqltypes.NewVarBinary("9223372036854775807"), // max_tps
 				sqltypes.NewVarBinary("9223372036854775807"), // max_replication_lag
 			},

go/vt/binlog/keyspace_id_resolver.go

@@ -134,18 +134,10 @@ func newKeyspaceIDResolverFactoryV3(ctx context.Context, ts *topo.Server, keyspa
 			return -1, nil, fmt.Errorf("no vschema definition for table %v", table.Name)
 		}
 
-		// The primary vindex is most likely the sharding key,
-		// and has to be unique.
-		if len(tableSchema.ColumnVindexes) == 0 {
-			return -1, nil, fmt.Errorf("no vindex definition for table %v", table.Name)
-		}
-		colVindex := tableSchema.ColumnVindexes[0]
-		if colVindex.Vindex.Cost() > 1 {
-			return -1, nil, fmt.Errorf("primary vindex cost is too high for table %v", table.Name)
-		}
-		if !colVindex.Vindex.IsUnique() {
-			// This is impossible, but just checking anyway.
-			return -1, nil, fmt.Errorf("primary vindex is not unique for table %v", table.Name)
+		// use the lowest cost unique vindex as the sharding key
+		colVindex, err := vindexes.FindVindexForSharding(table.Name.String(), tableSchema.ColumnVindexes)
+		if err != nil {
+			return -1, nil, err
 		}
 
 		// TODO @rafael - when rewriting the mapping function, this will need to change.

go/vt/callinfo/fakecallinfo/fakecallinfo.go

@@ -16,13 +16,16 @@ limitations under the License.
 
 package fakecallinfo
 
-import "html/template"
+import (
+	"fmt"
+	"html/template"
+)
 
 // FakeCallInfo gives a fake Callinfo usable in callinfo
 type FakeCallInfo struct {
 	Remote string
+	Method string
 	User   string
-	Txt    string
 	Html   string
 }
 
@@ -38,7 +41,7 @@ func (fci *FakeCallInfo) Username() string {
 
 // Text returns the text.
 func (fci *FakeCallInfo) Text() string {
-	return fci.Txt
+	return fmt.Sprintf("%s:%s(fakeRPC)", fci.Remote, fci.Method)
 }
 
 // HTML returns the html.

go/vt/dbconfigs/dbconfigs.go

@@ -100,6 +100,8 @@ func registerBaseFlags() {
 	flag.StringVar(&baseConfig.SslCaPath, "db_ssl_ca_path", "", "connection ssl ca path")
 	flag.StringVar(&baseConfig.SslCert, "db_ssl_cert", "", "connection ssl certificate")
 	flag.StringVar(&baseConfig.SslKey, "db_ssl_key", "", "connection ssl key")
+	flag.StringVar(&baseConfig.ServerName, "db_server_name", "", "server name of the DB we are connecting to.")
+
 }
 
 // The flags will change the global singleton
@@ -124,6 +126,7 @@ func registerPerUserFlags(dbc *userConfig, userKey string) {
 	flag.StringVar(&dbc.param.SslCaPath, "db-config-"+userKey+"-ssl-ca-path", "", "deprecated: use db_ssl_ca_path")
 	flag.StringVar(&dbc.param.SslCert, "db-config-"+userKey+"-ssl-cert", "", "deprecated: use db_ssl_cert")
 	flag.StringVar(&dbc.param.SslKey, "db-config-"+userKey+"-ssl-key", "", "deprecated: use db_ssl_key")
+	flag.StringVar(&dbc.param.ServerName, "db-config-"+userKey+"-server_name", "", "deprecated: use db_server_name")
 
 	flag.StringVar(&dbc.param.DeprecatedDBName, "db-config-"+userKey+"-dbname", "", "deprecated: dbname does not need to be explicitly configured")
 
@@ -246,6 +249,7 @@ func Init(defaultSocketFile string) (*DBConfigs, error) {
 				uc.param.SslCaPath = baseConfig.SslCaPath
 				uc.param.SslCert = baseConfig.SslCert
 				uc.param.SslKey = baseConfig.SslKey
+				uc.param.ServerName = baseConfig.ServerName
 			}
 		}
 	} else {