Slack sync upstream 2018 02 05

CODE_OF_CONDUCT.md

@@ -0,0 +1,3 @@
+## Community Code of Conduct
+
+Vitess adopts the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

GOVERNANCE.md

@@ -0,0 +1,115 @@
+# Vitess Governance
+
+# Overview
+
+Vitess is a meritocratic, consensus-based community project. Anyone with an interest in the project can join the community, contribute to the project design and participate in the decision making process. This document describes how that participation takes place and how to set about earning merit within the project community.
+
+# Roles And Responsibilities
+
+## Users
+
+Users are community members who have a need for the project. They are the most important members of the community and without them the project would have no purpose. Anyone can be a user; there are no special requirements.
+
+The project asks its users to participate in the project and community as much as possible. User contributions enable the project team to ensure that they are satisfying the needs of those users. Common user contributions include (but are not limited to):
+
+* evangelising about the project (e.g. a link on a website and word-of-mouth awareness raising)
+* informing developers of strengths and weaknesses from a new user perspective
+* providing moral support (a ‘thank you’ goes a long way)
+* providing financial support (the software is open source, but its developers need to eat)
+
+Users who continue to engage with the project and its community will often become more and more involved. Such users may find themselves becoming contributors, as described in the next section.
+
+## Contributors
+
+Contributors will be added to the [Collaborators list](https://github.com/youtube/vitess/settings/collaboration).
+
+Contributors are community members who contribute in concrete ways to the project. Anyone can become a contributor. There is no expectation of commitment to the project, no specific skill requirements and no selection process.
+
+In addition to their actions as users, contributors may also find themselves doing one or more of the following:
+
+* supporting new users (existing users are often the best people to support new users)
+* reporting bugs
+* identifying requirements
+* providing graphics and web design
+* programming
+* assisting with project infrastructure
+* writing documentation
+* fixing bugs
+* adding features
+
+Contributors engage with the project through the issue tracker, mailing list, the Slack channel, or by writing or editing documentation. They submit changes to the project itself via patches, which will be considered for inclusion in the project by existing committers (see next section). The Slack channel is the most appropriate place to ask for help when making that first contribution.
+
+As contributors gain experience and familiarity with the project, their profile within, and commitment to, the community will increase. At some stage, they may find themselves being nominated for committership.
+
+## Committers
+Committers will be added to the [committers list](https://github.com/orgs/youtube/teams/vitess-committers) and the [pullapprove list](https://github.com/youtube/vitess/blob/master/.pullapprove.yml). 
+
+Committers are community members who have shown that they are committed to the continued development of the project through ongoing engagement with the community. Committership allows contributors to more easily carry on with their project related activities by giving them direct access to the project’s resources. That is, they can make changes directly to project outputs, without having to submit changes via patches.
+
+This does not mean that a committer is free to do what they want. In fact, committers have no more authority over the project than contributors. While committership indicates a valued member of the community who has demonstrated a healthy respect for the project’s aims and objectives, their work continues to be reviewed by the community before acceptance in an official release.
+
+A committer is not allowed to merge their change without approval from another person. However, they are allowed to sidestep this rule under justifiable circumstances. For example:
+
+* If a CI tool is broken, they may override the tool to still submit the change.
+* Minor typos or fixes for broken tests.
+* The change was approved through other means than the standard process.
+
+Anyone can become a committer; there are no special requirements, other than to have shown a willingness and ability to participate in the project as a team player. Typically, a potential committer will need to show that they have an understanding of the project, its objectives and its strategy. They will also have provided valuable contributions to the project over a period of time.
+
+New committers can be nominated by any existing committer. Once they have been nominated, there will be a vote by the project management committee (PMC; see below). Committer voting is one of the few activities that takes place on the project’s private management list. This is to allow PMC members to freely express their opinions about a nominee without causing embarrassment. Once the vote has been held, the aggregated voting results are published on the public mailing list. The nominee is entitled to request an explanation of any ‘no’ votes against them, regardless of the outcome of the vote. This explanation will be provided by the PMC Chair (see below) and will be anonymous and constructive in nature.
+
+Nominees may decline their appointment as a committer. However, this is unusual, as the project does not expect any specific time or resource commitment from its community members. The intention behind the role of committer is to allow people to contribute to the project more easily, not to tie them in to the project in any formal way.
+
+It is important to recognise that commitership is a privilege, not a right. That privilege must be earned and once earned it can be removed by the PMC for conduct inconsistent with the [Guiding Principles](https://github.com/youtube/vitess/blob/master/GUIDING_PRINCIPLES.md) or if they drop below a level of commitment and engagement required to be a Committer, as determined by the PMC. The PMC also reserves the right to remove a person for any other reason inconsistent with the goals of Vitess.
+
+A committer who shows an above-average level of contribution to the project, particularly with respect to its strategic direction and long-term health, may be nominated to become a member of the PMC. This role is described below.
+
+## Project management committee
+
+PMC members will be added to the [list of administrators](https://github.com/orgs/youtube/teams/vitess-admin).
+
+The current list of PMC members and their github handles are:
+
+* Alain Jobart (alainjobart)
+* Anthony Yeh (enisoc)
+* Michael Berlin (michael-berlin)
+* Michael Demmer (demmer)
+* Sugu Sougoumarane (sougou)
+
+Given that YouTube has built and supported this project, they reserve the right to appoint two seats on the PMC for a period of one year from the time this document becomes effective. After one year has passed, the normal rules of engagement apply as described below.
+
+The project management committee consists of those individuals identified as ‘project owners’ on the development site. The PMC has additional responsibilities over and above those of a committer. These responsibilities ensure the smooth running of the project. PMC members are expected to review code contributions, participate in strategic planning, approve changes to the governance model and manage the copyrights within the project outputs.
+
+Members of the PMC do not have significant authority over other members of the community, although it is the PMC that votes on new committers, and makes all major decisions for the future with respect to Vitess. It also makes decisions when community consensus cannot be reached. In addition, the PMC has access to the project’s private mailing list and its archives. This list is used for sensitive issues, such as votes for new committers and legal matters that cannot be discussed in public. It is never used for project management or planning.
+
+Membership of the PMC is by invitation from the existing PMC members. A nomination will result in discussion and then a vote by the existing PMC members. PMC membership votes are subject to consensus approval of the current PMC members.
+
+The number of PMC members should be limited to 7. This number is chosen to ensure that sufficient points of view are represented, while preserving the efficiency of the decision making process.
+
+The PMC is responsible for maintaining the [Guiding Principles](https://github.com/youtube/vitess/blob/master/GUIDING_PRINCIPLES.md) and the code of conduct. It is also responsible for ensuring that those rules and principles are followed.
+
+## PMC Chair
+
+The current PMC chair is: Sugu Sougoumarane.
+
+The PMC Chair is a single individual, voted for by the PMC members. Once someone has been appointed Chair, they remain in that role until they choose to retire, or the PMC casts a two-thirds majority vote to remove them.
+
+The PMC Chair has no additional authority over other members of the PMC: the role is one of coordinator and facilitator. The Chair is also expected to ensure that all governance processes are adhered to, and has the casting vote when the project fails to reach consensus.
+
+# Support
+
+All participants in the community are encouraged to provide support for new users within the project management infrastructure. This support is provided as a way of growing the community. Those seeking support should recognise that all support activity within the project is voluntary and is therefore provided as and when time allows. A user requiring guaranteed response times or results should therefore seek to purchase a support contract from a community member. However, for those willing to engage with the project on its own terms, and willing to help support other users, the community support channels are ideal.
+
+# Contribution Process
+Anyone can contribute to the project, regardless of their skills, as there are many ways to contribute. For instance, a contributor might be active on the Slack channel and issue tracker, or might supply patches. The various ways of contributing are described in more detail in a separate document.
+
+The Slack channel list is the most appropriate place for a contributor to ask for help when making their first contribution.
+
+# Decision Making Process
+
+Decisions about the future of the project are made by the PMC. New proposals and ideas can be brought to the PMC’s attention through the Slack channel or by filing an issue. If necessary, the PMC will seek input from others to come to the final decision.
+
+The PMC’s decision is itself governed by the project’s [Guiding Principles](https://github.com/youtube/vitess/blob/master/GUIDING_PRINCIPLES.md), which shall be used to reach consensus. If a consensus cannot be reached, a simple majority voting process will be used to reach resolution. In case of a tie, the PMC chair has the casting vote.
+
+# Credits
+The contents of this document are based on http://oss-watch.ac.uk/resources/meritocraticgovernancemodel by Ross Gardler and Gabriel Hanganu.

GUIDING_PRINCIPLES.md

@@ -0,0 +1,27 @@
+# Guiding Principles
+
+The Vitess project operates under the guiding principles described below. The PMC is responsible for upholding these principles and keeping this document updated with any changes.
+
+# Mission Statement
+
+*“To be the most performant, scalable, and available NewSQL storage system in the Cloud.”*
+
+Vitess can currently run on bare-metal. However, the trend is clear that applications are moving to the cloud. For the foreseeable future, we should continue to support the ability to run on bare-metal, because it will facilitate migration to the cloud.
+
+The Vitess architecture is generic enough to accommodate any other RDBMS in the future. However, Vitess currently runs on MySQL and has many features that provide compatibility with it. Vitess should remain focused on MySQL until all frictions are removed for those who wish to migrate from MySQL to Vitess.
+
+# Vision
+
+The vision of the Vitess project is to serve the software community by gaining popularity and adoption.
+
+# Values
+
+Vitess is driven by high technical standards, and these must be maintained. It is also important to take into account that multiple humans are participating in the project, and we must ensure that we all treat each other well. In this respect, we will honor the following values:
+
+* Technical excellence
+* Innovation and creativity
+* Fairness and equality
+* Diversity
+* Inclusiveness
+* Openness
+* Adherence to the [Code of Conduct](https://github.com/youtube/vitess/blob/master/CODE_OF_CONDUCT.md)

doc/Messaging.md

@@ -142,7 +142,7 @@ A received (or processed) message can be acknowledged using the `MessageAck`
 API call. This call accepts the following parameters:
 
 * `Name`: Name of the message table.
-* `Keyspace`: Keypsace where the message table is present. This field can be
+* `Keyspace`: Keyspace where the message table is present. This field can be
   empty if the table name is unique across all keyspaces.
 * `Ids`: The list of ids that need to be acked.
 
@@ -184,18 +184,24 @@ message and the back-off is less aggressive.
 
 You can also view messages using regular `select` queries.
 
-## Known limitations
+## Undocumented features
 
-The message feature is currently in alpha, and can be improved. Here is the
-list of possible limitations/improvements:
+These are features that were previously known limitations, but have since been supported
+and are awaiting further documentation.
 
 * Flexible columns: Allow any number of application defined columns to be in
   the message table.
 * No ACL check for receivers: To be added.
-* Proactive scheduling: Upcoming messages can be proactively scheduled for
-  timely delivery instead of waiting for the next polling cycle.
 * Monitoring support: To be added.
 * Dropped tables: The message engine does not currently detect dropped tables.
+
+## Known limitations
+
+The message feature is currently in alpha, and can be improved. Here is the
+list of possible limitations/improvements:
+
+* Proactive scheduling: Upcoming messages can be proactively scheduled for
+  timely delivery instead of waiting for the next polling cycle.
 * Changed properties: Although the engine detects new message tables, it does
   not refresh properties of an existing table.
 * A `SELECT` style syntax for subscribing to messages.

docker/k8s/Dockerfile

@@ -21,6 +21,7 @@ COPY --from=base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificat
 COPY --from=base /vt/bin/mysqlctld /vt/bin/
 COPY --from=base /vt/bin/vtctld /vt/bin/
 COPY --from=base /vt/bin/vtctl /vt/bin/
+COPY --from=base /vt/bin/vtctlclient /vt/bin/
 COPY --from=base /vt/bin/vtgate /vt/bin/
 COPY --from=base /vt/bin/vttablet /vt/bin/
 COPY --from=base /vt/bin/vtworker /vt/bin/

docker/k8s/orchestrator/Dockerfile

@@ -0,0 +1,23 @@
+FROM vitess/k8s AS k8s
+
+FROM debian:stretch-slim
+
+# Copy certs to allow https calls
+COPY --from=k8s /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
+
+# Copy vtctlclient to be used to notify 
+COPY --from=k8s /vt/bin/vtctlclient /usr/bin/
+
+RUN apt-get update && \
+    apt-get upgrade -qq && \
+    apt-get install wget -qq --no-install-recommends && \
+    wget https://github.com/github/orchestrator/releases/download/v3.0.6/orchestrator_3.0.6_amd64.deb && \
+    dpkg -i orchestrator_3.0.6_amd64.deb && \
+    rm orchestrator_3.0.6_amd64.deb && \
+    apt-get purge wget -qq && \
+    apt-get autoremove -qq && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/local/orchestrator
+CMD ["./orchestrator", "--config=/conf/orchestrator.conf.json", "http"]

docker/k8s/pmm-client/Dockerfile

@@ -0,0 +1,17 @@
+FROM vitess/base AS base
+
+FROM debian:stretch-slim
+
+# Copy CA certs for https calls
+COPY --from=base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
+
+RUN apt-get update && \
+    apt-get upgrade -qq && \
+    apt-get install wget -qq --no-install-recommends && \
+    wget https://www.percona.com/redir/downloads/pmm-client/1.6.1/binary/debian/stretch/x86_64/pmm-client_1.6.1-1.stretch_amd64.deb && \
+    dpkg -i pmm-client_1.6.1-1.stretch_amd64.deb && \
+    rm pmm-client_1.6.1-1.stretch_amd64.deb && \
+    apt-get purge wget -qq && \
+    apt-get autoremove -qq && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*

go/mysql/query.go

@@ -507,7 +507,7 @@ func (c *Conn) writeColumnDefinition(field *querypb.Field) error {
 	pos = writeByte(data, pos, byte(typ))
 	pos = writeUint16(data, pos, uint16(flags))
 	pos = writeByte(data, pos, byte(field.Decimals))
-	pos += 2
+	pos = writeUint16(data, pos, uint16(0x0000))
 
 	if pos != len(data) {
 		return fmt.Errorf("internal error: packing of column definition used %v bytes instead of %v", pos, len(data))

go/mysql/server.go

@@ -116,23 +116,28 @@ type Listener struct {
 	connectionID uint32
 }
 
-// NewListener creates a new Listener.
-func NewListener(protocol, address string, authServer AuthServer, handler Handler) (*Listener, error) {
-	listener, err := net.Listen(protocol, address)
-	if err != nil {
-		return nil, err
-	}
-
+// NewFromListener creares a new mysql listener from an existing net.Listener
+func NewFromListener(l net.Listener, authServer AuthServer, handler Handler) (*Listener, error) {
 	return &Listener{
 		authServer: authServer,
 		handler:    handler,
-		listener:   listener,
+		listener:   l,
 
 		ServerVersion: DefaultServerVersion,
 		connectionID:  1,
 	}, nil
 }
 
+// NewListener creates a new Listener.
+func NewListener(protocol, address string, authServer AuthServer, handler Handler) (*Listener, error) {
+	listener, err := net.Listen(protocol, address)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewFromListener(listener, authServer, handler)
+}
+
 // Addr returns the listener address.
 func (l *Listener) Addr() net.Addr {
 	return l.listener.Addr()

go/mysql/server_test.go

@@ -158,6 +158,45 @@ func getHostPort(t *testing.T, a net.Addr) (string, int) {
 	return host, port
 }
 
+func TestConnectionFromListener(t *testing.T) {
+	th := &testHandler{}
+
+	authServer := NewAuthServerStatic()
+	authServer.Entries["user1"] = []*AuthServerStaticEntry{{
+		Password: "password1",
+		UserData: "userData1",
+	}}
+	// Make sure we can create our own net.Listener for use with the mysql
+	// listener
+	listener, err := net.Listen("tcp", ":0")
+	if err != nil {
+		t.Fatalf("net.Listener failed: %v", err)
+	}
+
+	l, err := NewFromListener(listener, authServer, th)
+	if err != nil {
+		t.Fatalf("NewListener failed: %v", err)
+	}
+	defer l.Close()
+	go l.Accept()
+
+	host, port := getHostPort(t, l.Addr())
+
+	// Setup the right parameters.
+	params := &ConnParams{
+		Host:  host,
+		Port:  port,
+		Uname: "user1",
+		Pass:  "password1",
+	}
+
+	c, err := Connect(context.Background(), params)
+	if err != nil {
+		t.Errorf("Should be able to connect to server but found error: %v", err)
+	}
+	c.Close()
+}
+
 func TestConnectionWithoutSourceHost(t *testing.T) {
 	th := &testHandler{}
 

go/vt/discovery/tablet_stats_cache_wait_test.go

@@ -54,7 +54,8 @@ func (a TargetArray) Less(i, j int) bool {
 func TestFindAllKeyspaceShards(t *testing.T) {
 	ctx := context.Background()
 	ts := memorytopo.NewServer("cell1", "cell2")
-	flag.Set("srv_topo_cache_ttl", "0s") // No caching values
+	flag.Set("srv_topo_cache_ttl", "0s")     // No caching values
+	flag.Set("srv_topo_cache_refresh", "0s") // No caching values
 	rs := srvtopo.NewResilientServer(ts, "TestFindAllKeyspaceShards")
 
 	// No keyspace / shards.