bump etcd/raft

github.com/coreos/etcd/Documentation/dev-guide/api_reference_v3.md

@@ -68,7 +68,8 @@ This is a generated documentation. Please read the proto files for more.
 | Alarm | AlarmRequest | AlarmResponse | Alarm activates, deactivates, and queries alarms regarding cluster health. |
 | Status | StatusRequest | StatusResponse | Status gets the status of the member. |
 | Defragment | DefragmentRequest | DefragmentResponse | Defragment defragments a member's backend database to recover storage space. |
-| Hash | HashRequest | HashResponse | Hash returns the hash of the local KV state for consistency checking purpose. This is designed for testing; do not use this in production when there are ongoing transactions. |
+| Hash | HashRequest | HashResponse | Hash computes the hash of the KV's backend. This is designed for testing; do not use this in production when there are ongoing transactions. |
+| HashKV | HashKVRequest | HashKVResponse | HashKV computes the hash of all MVCC keys up to a given revision. |
 | Snapshot | SnapshotRequest | SnapshotResponse | Snapshot sends a snapshot of the entire backend from a member over a stream to a client. |
 | MoveLeader | MoveLeaderRequest | MoveLeaderResponse | MoveLeader requests current leader node to transfer its leadership to transferee. |
 
@@ -402,6 +403,7 @@ CompactionRequest compacts the key-value store up to a given revision. All super
 | create_revision | create_revision is the creation revision of the given key | int64 |
 | mod_revision | mod_revision is the last modified revision of the given key. | int64 |
 | value | value is the value of the given key, in bytes. | bytes |
+| lease | lease is the lease id of the given key. | int64 |
 | range_end | range_end compares the given target to all keys in the range [key, range_end). See RangeRequest for more details on key ranges. | bytes |
 
 
@@ -440,6 +442,24 @@ Empty field.
 
 
 
+##### message `HashKVRequest` (etcdserver/etcdserverpb/rpc.proto)
+
+| Field | Description | Type |
+| ----- | ----------- | ---- |
+| revision | revision is the key-value store revision for the hash operation. | int64 |
+
+
+
+##### message `HashKVResponse` (etcdserver/etcdserverpb/rpc.proto)
+
+| Field | Description | Type |
+| ----- | ----------- | ---- |
+| header |  | ResponseHeader |
+| hash | hash is the hash value computed from the responding member's MVCC keys up to a given revision. | uint32 |
+| compact_revision | compact_revision is the compacted revision of key-value store when hash begins. | int64 |
+
+
+
 ##### message `HashRequest` (etcdserver/etcdserverpb/rpc.proto)
 
 Empty field.
@@ -451,7 +471,7 @@ Empty field.
 | Field | Description | Type |
 | ----- | ----------- | ---- |
 | header |  | ResponseHeader |
-| hash | hash is the hash value computed from the responding member's key-value store. | uint32 |
+| hash | hash is the hash value computed from the responding member's KV's backend. | uint32 |
 
 
 

github.com/coreos/etcd/Documentation/dev-guide/apispec/swagger/rpc.swagger.json

@@ -858,23 +858,23 @@
         "tags": [
           "Maintenance"
         ],
-        "summary": "Hash returns the hash of the local KV state for consistency checking purpose.\nThis is designed for testing; do not use this in production when there\nare ongoing transactions.",
-        "operationId": "Hash",
+        "summary": "HashKV computes the hash of all MVCC keys up to a given revision.",
+        "operationId": "HashKV",
         "parameters": [
           {
             "name": "body",
             "in": "body",
             "required": true,
             "schema": {
-              "$ref": "#/definitions/etcdserverpbHashRequest"
+              "$ref": "#/definitions/etcdserverpbHashKVRequest"
             }
           }
         ],
         "responses": {
           "200": {
             "description": "(empty)",
             "schema": {
-              "$ref": "#/definitions/etcdserverpbHashResponse"
+              "$ref": "#/definitions/etcdserverpbHashKVResponse"
             }
           }
         }
@@ -1017,7 +1017,8 @@
         "VERSION",
         "CREATE",
         "MOD",
-        "VALUE"
+        "VALUE",
+        "LEASE"
       ]
     },
     "EventEventType": {
@@ -1471,6 +1472,11 @@
           "type": "string",
           "format": "byte"
         },
+        "lease": {
+          "description": "lease is the lease id of the given key.",
+          "type": "string",
+          "format": "int64"
+        },
         "mod_revision": {
           "description": "mod_revision is the last modified revision of the given key.",
           "type": "string",
@@ -1552,14 +1558,42 @@
         }
       }
     },
+    "etcdserverpbHashKVRequest": {
+      "type": "object",
+      "properties": {
+        "revision": {
+          "description": "revision is the key-value store revision for the hash operation.",
+          "type": "string",
+          "format": "int64"
+        }
+      }
+    },
+    "etcdserverpbHashKVResponse": {
+      "type": "object",
+      "properties": {
+        "compact_revision": {
+          "description": "compact_revision is the compacted revision of key-value store when hash begins.",
+          "type": "string",
+          "format": "int64"
+        },
+        "hash": {
+          "description": "hash is the hash value computed from the responding member's MVCC keys up to a given revision.",
+          "type": "integer",
+          "format": "int64"
+        },
+        "header": {
+          "$ref": "#/definitions/etcdserverpbResponseHeader"
+        }
+      }
+    },
     "etcdserverpbHashRequest": {
       "type": "object"
     },
     "etcdserverpbHashResponse": {
       "type": "object",
       "properties": {
         "hash": {
-          "description": "hash is the hash value computed from the responding member's key-value store.",
+          "description": "hash is the hash value computed from the responding member's KV's backend.",
           "type": "integer",
           "format": "int64"
         },

github.com/coreos/etcd/Documentation/dev-guide/experimental_apis.md

@@ -4,4 +4,4 @@ For the most part, the etcd project is stable, but we are still moving fast! We
 
 ## The current experimental API/features are:
 
-(none currently)
+- [KV ordering](https://godoc.org/github.com/coreos/etcd/clientv3/ordering) wrapper. When an etcd client switches endpoints, responses to serializable reads may go backward in time if the new endpoint is lagging behind the rest of the cluster. The ordering wrapper caches the current cluster revision from response headers. If a response revision is less than the cached revision, the client selects another endpoint and reissues the read. Enable in grpcproxy with `--experimental-serializable-ordering`.

github.com/coreos/etcd/Documentation/dev-internal/release.md

@@ -53,7 +53,7 @@ All releases version numbers follow the format of [semantic versioning 2.0.0](ht
 Run release script in root directory:
 
 ```
-./scripts/release.sh ${VERSION}
+TAG=gcr.io/etcd-development/etcd ./scripts/release.sh ${VERSION}
 ```
 
 It generates all release binaries and images under directory ./release.
@@ -90,13 +90,41 @@ The public key for GPG signing can be found at [CoreOS Application Signing Key](
 - Select whether it is a pre-release.
 - Publish the release!
 
+## Publish docker image in gcr.io
+
+- Push docker image:
+
+```
+gcloud docker -- login -u _json_key -p "$(cat /etc/gcp-key-etcd.json)" https://gcr.io
+
+for TARGET_ARCH in "-arm64" "-ppc64le" ""; do
+  gcloud docker -- push gcr.io/etcd-development/etcd:${VERSION}${TARGET_ARCH}
+done
+```
+
+- Add `latest` tag to the new image on [gcr.io](https://console.cloud.google.com/gcr/images/etcd-development/GLOBAL/etcd?project=etcd-development&authuser=1) if this is a stable release.
+
 ## Publish docker image in Quay.io
 
+- Build docker images with quay.io:
+
+```
+for TARGET_ARCH in "amd64" "arm64" "ppc64le"; do
+  TAG=quay.io/coreos/etcd GOARCH=${TARGET_ARCH} \
+    BINARYDIR=release/etcd-${VERSION}-linux-${TARGET_ARCH} \
+    BUILDDIR=release \
+    ./scripts/build-docker ${VERSION}
+done
+```
+
 - Push docker image:
 
 ```
 docker login quay.io
-docker push quay.io/coreos/etcd:${VERSION}
+
+for TARGET_ARCH in "-arm64" "-ppc64le" ""; do
+  docker push quay.io/coreos/etcd:${VERSION}${TARGET_ARCH}
+done
 ```
 
 - Add `latest` tag to the new image on [quay.io](https://quay.io/repository/coreos/etcd?tag=latest&tab=tags) if this is a stable release.

github.com/coreos/etcd/Documentation/docs.md

@@ -58,14 +58,7 @@ Administrators who need a fault-tolerant etcd cluster for either development or
  - [Maintenance][maintenance]
  - [Failure modes][failures]
  - [Disaster recovery][recovery]
-
-### Upgrading and compatibility
-
- - [Version numbers][versioning]
- - [Migrate applications from using API v2 to API v3][v2_migration]
- - [Upgrading a v2.3 cluster to v3.0][v3_upgrade]
- - [Upgrading a v3.0 cluster to v3.1][v31_upgrade]
- - [Upgrading a v3.1 cluster to v3.2][v32_upgrade]
+ - [Upgrading][upgrading]
 
 ## Learning
 
@@ -115,9 +108,7 @@ To learn more about the concepts and internals behind etcd, read the following p
 [freebsd_platform]: platforms/freebsd.md
 [aws_platform]: platforms/aws.md
 [experimental]: dev-guide/experimental_apis.md
-[v3_upgrade]: upgrades/upgrade_3_0.md
-[v31_upgrade]: upgrades/upgrade_3_1.md
-[v32_upgrade]: upgrades/upgrade_3_2.md
 [authentication]: op-guide/authentication.md
 [auth_design]: learning/auth_design.md
 [tuning]: tuning.md
+[upgrading]: upgrades/upgrading-etcd.md

github.com/coreos/etcd/Documentation/op-guide/container.md

@@ -79,11 +79,15 @@ export NODE1=192.168.1.21
 Run the latest version of etcd:
 
 ```
+REGISTRY=quay.io/coreos/etcd
+# available from v3.2.5
+REGISTRY=gcr.io/etcd-development/etcd
+
 docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=${DATA_DIR}:/etcd-data \
-  --name etcd quay.io/coreos/etcd:latest \
+  --name etcd ${REGISTRY}:latest \
   /usr/local/bin/etcd \
   --data-dir=/etcd-data --name node1 \
   --initial-advertise-peer-urls http://${NODE1}:2380 --listen-peer-urls http://${NODE1}:2380 \
@@ -100,6 +104,10 @@ etcdctl --endpoints=http://${NODE1}:2379 member list
 ### Running a 3 node etcd cluster
 
 ```
+REGISTRY=quay.io/coreos/etcd
+# available from v3.2.5
+REGISTRY=gcr.io/etcd-development/etcd
+
 # For each machine
 ETCD_VERSION=latest
 TOKEN=my-etcd-token
@@ -120,7 +128,7 @@ docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=${DATA_DIR}:/etcd-data \
-  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
+  --name etcd ${REGISTRY}:${ETCD_VERSION} \
   /usr/local/bin/etcd \
   --data-dir=/etcd-data --name ${THIS_NAME} \
   --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
@@ -135,7 +143,7 @@ docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=${DATA_DIR}:/etcd-data \
-  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
+  --name etcd ${REGISTRY}:${ETCD_VERSION} \
   /usr/local/bin/etcd \
   --data-dir=/etcd-data --name ${THIS_NAME} \
   --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
@@ -150,7 +158,7 @@ docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=${DATA_DIR}:/etcd-data \
-  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
+  --name etcd ${REGISTRY}:${ETCD_VERSION} \
   /usr/local/bin/etcd \
   --data-dir=/etcd-data --name ${THIS_NAME} \
   --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
@@ -173,11 +181,17 @@ To provision a 3 node etcd cluster on bare-metal, the examples in the [baremetal
 
 The etcd release container does not include default root certificates. To use HTTPS with certificates trusted by a root authority (e.g., for discovery), mount a certificate directory into the etcd container:
 
+```
+REGISTRY=quay.io/coreos/etcd
+# available from v3.2.5
+REGISTRY=gcr.io/etcd-development/etcd
+```
+
 ```
 rkt run \
   --volume etcd-ssl-certs-bundle,kind=host,source=/etc/ssl/certs/ca-certificates.crt \
   --mount volume=etcd-ssl-certs-bundle,target=/etc/ssl/certs/ca-certificates.crt \
-  quay.io/coreos/etcd:latest -- --name my-name \
+  ${REGISTRY}:latest -- --name my-name \
   --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \
   --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \
   --discovery https://discovery.etcd.io/c11fbcdc16972e45253491a24fcf45e1
@@ -188,7 +202,7 @@ docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
-  quay.io/coreos/etcd:latest \
+  ${REGISTRY}:latest \
   /usr/local/bin/etcd --name my-name \
   --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \
   --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \

github.com/coreos/etcd/Documentation/op-guide/v2-migration.md

@@ -6,7 +6,7 @@ Migrating an application from the API v2 to the API v3 involves two steps: 1) mi
 
 ## Migrate client library
 
-API v3 is different from API v2, thus application developers need to use a new client library to send requests to etcd API v3. The documentation of the client v3 is available at https://godoc.org/github.com/coreos/etcd/clientv3. 
+API v3 is different from API v2, thus application developers need to use a new client library to send requests to etcd API v3. The documentation of the client v3 is available at https://godoc.org/github.com/coreos/etcd/clientv3.
 
 There are some notable differences between API v2 and API v3:
 
@@ -38,13 +38,17 @@ Second, migrate the v2 keys into v3 with the [migrate][migrate_command] (`ETCDCT
 
 Restart the etcd members and everything should just work.
 
+For etcd v3.3+, run `ETCDCTL_API=3 etcdctl endpoint hashkv --cluster` to ensure key-value stores are consistent post migration.
+
+**Warn**: When v2 store has expiring TTL keys and migrate command intends to preserve TTLs, migration may be inconsistent with the last committed v2 state when run on any member with a raft index less than the last leader's raft index.
+
 ### Online migration
 
 If the application cannot tolerate any downtime, then it must migrate online. The implementation of online migration will vary from application to application but the overall idea is the same.
 
 First, write application code using the v3 API. The application must support two modes: a migration mode and a normal mode. The application starts in migration mode. When running in migration mode, the application reads keys using the v3 API first, and, if it cannot find the key, it retries with the API v2. In normal mode, the application only reads keys using the v3 API. The application writes keys over the API v3 in both modes. To acknowledge a switch from migration mode to normal mode, the application watches on a switch mode key. When switch key’s value turns to `true`, the application switches over from migration mode to normal mode.
 
-Second, start a background job to migrate data from the store v2 to the mvcc store by reading keys from the API v2 and writing keys to the API v3. 
+Second, start a background job to migrate data from the store v2 to the mvcc store by reading keys from the API v2 and writing keys to the API v3.
 
 After finishing data migration, the background job writes `true` into the switch mode key to notify the application that it may switch modes.
 

github.com/coreos/etcd/Documentation/upgrades/upgrading-etcd.md

@@ -0,0 +1,19 @@
+# Upgrading etcd clusters and applications
+
+This section contains documents specific to upgrading etcd clusters and applications.
+
+## Moving from etcd API v2 to API v3
+* [Migrate applications from using API v2 to API v3][migrate-apps]
+
+## Upgrading an etcd v3.x cluster
+* [Upgrade etcd from 3.0 to 3.1][upgrade-3-1]
+* [Upgrade etcd from 3.1 to 3.2][upgrade-3-2]
+
+## Upgrading from etcd v2.3
+* [Upgrade a v2.3 cluster to v3.0][upgrade-cluster]
+
+
+[migrate-apps]: ../op-guide/v2-migration.md
+[upgrade-cluster]: upgrade_3_0.md
+[upgrade-3-1]: upgrade_3_1.md
+[upgrade-3-2]: upgrade_3_2.md