distsql: avoid dead nodes #13655

vivekmenezes · 2017-02-17T20:40:28Z

I'm seeing an index backfill not terminating because it keeps hitting the same error. I've not had the time to debug this issue but it's certainly reproducible and definitely a bug introduced in the move of index backfilling to use the distsql framework. I see some grpc connection failures, which could be the reason for this.

W170217 20:34:17.833040 16 storage/store.go:3419  [n2,s2,r245/3:/Table/51/1/73{33033…-70240…},@c4203b4e00] handle raft ready: 1.3s [processed=0]
W170217 20:34:17.853786 91 storage/node_liveness.go:290  [n2,hb] slow heartbeat took 1.1s
I170217 20:34:18.428086 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:18.820403 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:19.439318 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:19.667104 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:20.318431 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:20.832591 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:21.288176 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:21.326393 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:21.431718 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:21.480199 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:21.517440 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:21.520305 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:21.532715 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:21.680353 2346509 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 603493 (id=610a38b2, encoded size=56559063, 54 rocksdb batches, 202 log entries)
I170217 20:34:21.779644 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:22.681822 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:22.706063 61 server/status/runtime.go:228  runtime stats: 3.1 GiB RSS, 134 goroutines, 105 MiB/207 MiB/415 MiB GO alloc/idle/total, 2.2 GiB/2.7 GiB CGO alloc/total, 43556.55cgo/sec, 2.61/0.40 %(u/s)time, 0.00 %gc (24x)
I170217 20:34:22.899653 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:23.513572 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:23.837014 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:23.917211 14 storage/engine/rocksdb.go:1104  batch [2/478/0] commit took 1.146814597s (>500ms):
goroutine 14 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42537c700, 0x7f8a0822db00, 0xc4207e3500, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420836700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b5bb050, 0xc43c8ed800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xd9)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.919322 13 storage/engine/rocksdb.go:1104  batch [2/476/0] commit took 1.151287522s (>500ms):
goroutine 13 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc4264ad6c0, 0x7f8a0822db00, 0xc420843cb0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc42084a380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b613c50, 0xc450d13200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x28)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.921331 66 storage/engine/rocksdb.go:1104  batch [2/479/0] commit took 1.153445109s (>500ms):
goroutine 66 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42da49dc0, 0x7f8a0822db00, 0xc4207a10e0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4207e1500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b63a840, 0xc42bb42c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x86)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.921641 16 storage/engine/rocksdb.go:1104  batch [2/525/0] commit took 1.181706318s (>500ms):
goroutine 16 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42645c8c0, 0x7f8a0822db00, 0xc42248d890, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4211f6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b60df50, 0xc42510ca80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x157)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.921922 2346509 storage/engine/rocksdb.go:1104  batch [204/87559/59] commit took 1.1906832s (>500ms):
goroutine 2346509 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422b1fa40, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc423bb9500, 0x7f8a0822dbe8, 0xc429c5c1b0, 0xe7419eb1b2380a61, 0x4a0e94d823699baf, 0xc427d7cc00, 0x36, 0x40, 0xc424d9f500, 0xca, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:670 +0xb30
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc423bb9500, 0xe7419eb1b2380a61, 0x4a0e94d823699baf, 0xc427d7cc00, 0x36, 0x40, 0xc424d9f500, 0xca, 0xe0, 0xc4240f6480, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2660 +0x12ea
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc425356d80, 0xc4240f6540, 0xe7419eb1b2380a61, 0x4a0e94d823699baf, 0xc427d7cc00, 0x36, 0x40, 0xc424d9f500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc4203b0700, 0xc4240f6480, 0x7f8a0347b218, 0xc4500e6ab0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2795 +0x5a3
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x27f1260, 0xc4500e6ab0, 0xc42042bf80, 0x7f8a0822dbe8, 0xc42b582990, 0xc423c29738, 0xc423c29740)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x7f8a0822dbe8, 0xc42b582990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203a5950, 0x1b6618f, 0x3, 0x1, 0x0, 0x0, 0xc42b582de0, 0x7f8a0822dbe8, 0xc42b582990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:283 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:284 +0x14a
I170217 20:34:23.922555 2346509 storage/replica_raftstorage.go:604  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applied Raft snapshot in 2242ms [clear=885ms batch=165ms entries=0ms commit=1191ms]
W170217 20:34:23.922970 12 storage/engine/rocksdb.go:1104  batch [2/1001/0] commit took 1.182683998s (>500ms):
goroutine 12 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc4264ac8c0, 0x7f8a0822db00, 0xc4208540f0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4209ca700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc428c11950, 0xc424f3f080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x155)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.925415 67 storage/engine/rocksdb.go:1104  batch [2/480/0] commit took 1.158261468s (>500ms):
goroutine 67 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42645d180, 0x7f8a0822db00, 0xc420800900, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420848a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady(0xc420848a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2595 +0xd3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady(0xc4203b0700, 0x13a)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3406 +0x13a
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:223 +0x28c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:23.925496 67 storage/store.go:3419  [n2,s2,r314/3:/Table/51/1/44{33633…-64456…},@c420848a80] handle raft ready: 1.2s [processed=0]
W170217 20:34:23.925617 15 storage/engine/rocksdb.go:1104  batch [2/590/0] commit took 1.186285189s (>500ms):
goroutine 15 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42da49340, 0x7f8a0822db00, 0xc420f697a0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420b47180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b666fc0, 0xc4254a1200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x159)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
I170217 20:34:24.394858 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:24.789632 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:25.458758 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:25.704925 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:26.468201 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:26.523853 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:27.522309 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:27.541870 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:27.592922 2355635 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 608425 (id=f2174b8f, encoded size=56751586, 54 rocksdb batches, 232 log entries)
I170217 20:34:28.334822 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:28.348287 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:28.535148 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:28.618487 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:29.491489 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:29.680029 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:29.992277 2355635 storage/engine/rocksdb.go:1104  batch [234/100253/59] commit took 1.324677569s (>500ms):
goroutine 2355635 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc424f501c0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc423bb9500, 0x7f8a0822dbe8, 0xc429c5c1b0, 0xa4ed28d8f4b17f2, 0x262b7f322c927981, 0xc42a0f4000, 0x36, 0x40, 0xc42043b000, 0xe8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:670 +0xb30
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc423bb9500, 0xa4ed28d8f4b17f2, 0x262b7f322c927981, 0xc42a0f4000, 0x36, 0x40, 0xc42043b000, 0xe8, 0x100, 0xc425a9d200, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2660 +0x12ea
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc428ca2780, 0xc425a9d2c0, 0xa4ed28d8f4b17f2, 0x262b7f322c927981, 0xc42a0f4000, 0x36, 0x40, 0xc42043b000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc4203b0700, 0xc425a9d200, 0x7f8a0347b218, 0xc442dabc60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2795 +0x5a3
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x27f1260, 0xc442dabc60, 0xc42042bf80, 0x7f8a0822dbe8, 0xc4271255c0, 0xc42091ff38, 0xc42091ff40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x7f8a0822dbe8, 0xc4271255c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203a5950, 0x1b6618f, 0x3, 0x1, 0x0, 0x0, 0xc427125650, 0x7f8a0822dbe8, 0xc4271255c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:283 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:284 +0x14a
I170217 20:34:29.992817 2355635 storage/replica_raftstorage.go:604  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applied Raft snapshot in 2399ms [clear=879ms batch=194ms entries=1ms commit=1325ms]
W170217 20:34:29.993470 15 storage/engine/rocksdb.go:1104  batch [3/237/3] commit took 1.324838584s (>500ms):
goroutine 15 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42130ddc0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc4209ca700, 0x7f8a0822dbe8, 0xc4208540f0, 0xc42b650000, 0x8, 0x0, 0x22d97, 0x22c78, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc4209ca700, 0x7f8a0822dbe8, 0xc4208540f0, 0xc42b650000, 0x8, 0x22d97, 0x300000003, 0x3, 0xc420aaf810, 0x22c78, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4209ca700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423c55b30, 0xc4268b6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x155)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.993574 16 storage/engine/rocksdb.go:1104  batch [2/690/0] commit took 1.321304297s (>500ms):
goroutine 16 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc421644540, 0x7f8a0822db00, 0xc420814cc0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420812e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady(0xc420812e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2595 +0xd3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady(0xc4203b0700, 0x48)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3406 +0x13a
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:223 +0x28c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.993723 14 storage/engine/rocksdb.go:1104  batch [1/50/0] commit took 1.324639873s (>500ms):
goroutine 14 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc425184540, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420326e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423f0df50, 0xc42585be00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x7)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.993770 67 storage/engine/rocksdb.go:1104  batch [1/50/0] commit took 1.323973446s (>500ms):
goroutine 67 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42116d6c0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc42086ee00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423e9ad80, 0xc4240c8600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.994364 12 storage/engine/rocksdb.go:1104  batch [1/51/0] commit took 1.325366646s (>500ms):
goroutine 12 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc4222a8000, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4207ab180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423916f90, 0xc420de8900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xa4)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.994915 66 storage/engine/rocksdb.go:1104  batch [2/422/0] commit took 1.324405228s (>500ms):
goroutine 66 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42116ce00, 0x7f8a0822db00, 0xc4207e25d0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4207cbc00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423ebd680, 0xc4268b6a10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.996974 13 storage/engine/rocksdb.go:1104  batch [3/232/3] commit took 1.325531537s (>500ms):
goroutine 13 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc421644380, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc4203b5c00, 0x7f8a0822dbe8, 0xc4203c9e90, 0xc4380ae998, 0x8, 0x0, 0x1c63, 0x1941, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc4203b5c00, 0x7f8a0822dbe8, 0xc4203c9e90, 0xc4380ae998, 0x8, 0x1c63, 0x300000003, 0x2, 0xc420ebcaa0, 0x1941, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4203b5c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423916c30, 0xc4240c8000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xc5)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:29.999818 16 storage/store.go:3419  [n2,s2,r72/2:/Table/51/1/33{19296…-61407…},@c420812e00] handle raft ready: 1.3s [processed=0]
I170217 20:34:30.655071 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:30.789974 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:30.943671 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:31.009413 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:31.039314 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:31.053054 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:31.807099 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:31.815502 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:32.290991 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:32.321578 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:32.466564 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:32.535163 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:32.573565 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:32.644347 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:32.707930 61 server/status/runtime.go:228  runtime stats: 3.1 GiB RSS, 142 goroutines, 88 MiB/221 MiB/416 MiB GO alloc/idle/total, 2.0 GiB/2.6 GiB CGO alloc/total, 38790.01cgo/sec, 2.49/0.38 %(u/s)time, 0.01 %gc (24x)
I170217 20:34:32.736566 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:32.760737 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:32.773244 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:32.864762 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:32.878169 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:32.931879 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:32.945171 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:33.015022 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:33.174565 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:33.192479 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:33.283235 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:33.295175 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:33.413250 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:33.431089 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:33.455822 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:33.467353 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:33.494698 2366349 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 613348 (id=15fe3af4, encoded size=56918050, 55 rocksdb batches, 189 log entries)
I170217 20:34:33.822324 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:34.063488 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:34.970895 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:35.163136 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:35.727766 14 storage/engine/rocksdb.go:1104  batch [2/694/0] commit took 1.119465477s (>500ms):
goroutine 14 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc425240000, 0x7f8a0822db00, 0xc4207a1a70, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4207e1880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc4260dac90, 0xc450034000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x10e)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.730297 67 storage/engine/rocksdb.go:1104  batch [2/712/0] commit took 1.126360899s (>500ms):
goroutine 67 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42a751dc0, 0x7f8a0822db00, 0xc4201f9e90, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420327c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc4260da810, 0xc4483fc600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x12b)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.731197 15 storage/engine/rocksdb.go:1104  batch [2/690/0] commit took 1.13011597s (>500ms):
goroutine 15 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422b56000, 0x7f8a0822db00, 0xc4260c30e0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc428709c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42614c3f0, 0xc420ec1970, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x1b)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.732018 68 storage/engine/rocksdb.go:1104  batch [2/694/0] commit took 1.12436779s (>500ms):
goroutine 68 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422b56380, 0x7f8a0822db00, 0xc420782720, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420788000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady(0xc420788000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2595 +0xd3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady(0xc4203b0700, 0x11a)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3406 +0x13a
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:223 +0x28c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.732343 68 storage/store.go:3419  [n2,s2,r282/2:/Table/51/1/{198618…-202195…},@c420788000] handle raft ready: 1.1s [processed=0]
W170217 20:34:35.735182 16 storage/engine/rocksdb.go:1104  batch [1/52/0] commit took 1.126891266s (>500ms):
goroutine 16 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc426101500, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420727180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42616faa0, 0xc450034450, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x132)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.736742 12 storage/engine/rocksdb.go:1104  batch [2/712/0] commit took 1.13060808s (>500ms):
goroutine 12 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc426101340, 0x7f8a0822db00, 0xc420732b40, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4203b4700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42616f140, 0xc42bd91b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x101)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.737910 66 storage/engine/rocksdb.go:1104  batch [2/760/0] commit took 1.13404459s (>500ms):
goroutine 66 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc426101180, 0x7f8a0822db00, 0xc420f697a0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420b47180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42616eea0, 0xc420ec1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x159)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:35.738047 2366349 storage/engine/rocksdb.go:1104  batch [191/81640/60] commit took 1.136167084s (>500ms):
goroutine 2366349 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc423aeac40, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc423bb9500, 0x7f8a0822dbe8, 0xc429c5c1b0, 0x784c459ff43afe15, 0xd192517631a84083, 0xc424252600, 0x37, 0x40, 0xc427e22400, 0xbd, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:670 +0xb30
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc423bb9500, 0x784c459ff43afe15, 0xd192517631a84083, 0xc424252600, 0x37, 0x40, 0xc427e22400, 0xbd, 0xc0, 0xc425335200, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2660 +0x12ea
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc424a56450, 0xc4253352c0, 0x784c459ff43afe15, 0xd192517631a84083, 0xc424252600, 0x37, 0x40, 0xc427e22400, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc4203b0700, 0xc425335200, 0x7f8a0347b218, 0xc4479bb3d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2795 +0x5a3
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x27f1260, 0xc4479bb3d0, 0xc42042bf80, 0x7f8a0822dbe8, 0xc424d51290, 0xc42091f738, 0xc42091f740)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x7f8a0822dbe8, 0xc424d51290)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203a5950, 0x1b6618f, 0x3, 0x1, 0x0, 0x0, 0xc424d512c0, 0x7f8a0822dbe8, 0xc424d51290)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:283 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:284 +0x14a
I170217 20:34:35.738369 2366349 storage/replica_raftstorage.go:604  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applied Raft snapshot in 2243ms [clear=899ms batch=204ms entries=0ms commit=1139ms]
W170217 20:34:35.790002 91 storage/node_liveness.go:290  [n2,hb] slow heartbeat took 1.1s
I170217 20:34:35.943465 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:36.182237 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:37.038937 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:37.141877 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:37.980134 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:38.122378 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:38.397084 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:38.476430 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:38.504484 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:38.668016 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:38.832115 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:38.872601 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:38.880993 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:38.910996 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:38.995167 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:39.045315 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:39.082136 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:39.124555 2375564 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 618220 (id=a58ce90c, encoded size=57091572, 55 rocksdb batches, 171 log entries)
W170217 20:34:39.141316 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:39.319673 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:39.420073 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:39.601286 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:39.654320 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:39.759396 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:39.806207 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:39.830701 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:39.866870 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.053665 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.193509 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:40.196677 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:40.246558 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.272216 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.313156 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.349991 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.394101 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.466122 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.477008 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.503101 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.513516 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.536341 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.547526 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.589931 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.601416 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.633138 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.673425 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.699351 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.710318 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.734610 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.746235 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.769683 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.780462 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.816074 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.838625 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.874920 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.886514 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.935967 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:40.947650 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:40.949546 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:40.973418 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:40.983689 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.008998 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.018320 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.044408 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.062746 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.116972 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.126786 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.154577 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.165169 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.191465 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.210114 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.219897 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:41.293141 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.304257 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.352519 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.364327 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.403798 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.438032 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
W170217 20:34:41.461464 67 storage/engine/rocksdb.go:1104  batch [2/1013/0] commit took 1.153893239s (>500ms):
goroutine 67 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422dcf880, 0x7f8a0822db00, 0xc420f697a0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420b47180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42739e480, 0xc422dac180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x159)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.461562 12 storage/engine/rocksdb.go:1104  batch [2/1001/0] commit took 1.154972161s (>500ms):
goroutine 12 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422dcf6c0, 0x7f8a0822db00, 0xc4208540f0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4209ca700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc427431c50, 0xc429edd670, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x155)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.464722 66 storage/engine/rocksdb.go:1104  batch [1/50/0] commit took 1.157782825s (>500ms):
goroutine 66 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc426fafa40, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420813880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b55dc50, 0xc424171e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xe)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.465119 13 storage/engine/rocksdb.go:1104  batch [3/230/3] commit took 1.176057327s (>500ms):
goroutine 13 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42bc5dc00, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc420888380, 0x7f8a0822dbe8, 0xc4208819b0, 0xc42efc2c68, 0x8, 0x0, 0x2921, 0x17da, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc420888380, 0x7f8a0822dbe8, 0xc4208819b0, 0xc42efc2c68, 0x8, 0x2921, 0x400000004, 0x4, 0xc425c77040, 0x17da, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420888380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc4273a0c90, 0xc42921ca50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x3b)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.465212 14 storage/engine/rocksdb.go:1104  batch [2/325/0] commit took 1.17591718s (>500ms):
goroutine 14 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc429f0ac40, 0x7f8a0822db00, 0xc42076b4d0, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420766a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42742cc30, 0xc449402c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x13d)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.465544 2375564 storage/engine/rocksdb.go:1104  batch [173/74477/60] commit took 1.176531234s (>500ms):
goroutine 2375564 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc423120fc0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc423bb9500, 0x7f8a0822dbe8, 0xc429c5c1b0, 0x8740bfee0ce98ca5, 0x8089d34734db5996, 0xc4240cd200, 0x37, 0x40, 0xc42da42400, 0xab, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:670 +0xb30
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc423bb9500, 0x8740bfee0ce98ca5, 0x8089d34734db5996, 0xc4240cd200, 0x37, 0x40, 0xc42da42400, 0xab, 0xc0, 0xc424a7efc0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2660 +0x12ea
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc425496960, 0xc424a7f080, 0x8740bfee0ce98ca5, 0x8089d34734db5996, 0xc4240cd200, 0x37, 0x40, 0xc42da42400, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc4203b0700, 0xc424a7efc0, 0x7f8a0347b218, 0xc421961f60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2795 +0x5a3
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x27f1260, 0xc421961f60, 0xc42042bf80, 0x7f8a0822dbe8, 0xc4260b86f0, 0xc42091e738, 0xc42091e740)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x7f8a0822dbe8, 0xc4260b86f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203a5950, 0x1b6618f, 0x3, 0x1, 0x0, 0x0, 0xc4260b9a40, 0x7f8a0822dbe8, 0xc4260b86f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:283 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:284 +0x14a
I170217 20:34:41.466303 2375564 storage/replica_raftstorage.go:604  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applied Raft snapshot in 2342ms [clear=970ms batch=193ms entries=0ms commit=1177ms]
W170217 20:34:41.466399 16 storage/engine/rocksdb.go:1104  batch [2/519/0] commit took 1.153818429s (>500ms):
goroutine 16 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc426fafc00, 0x7f8a0822db00, 0xc42248d890, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4211f6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b55def0, 0xc422dacf00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x157)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:41.467324 68 storage/engine/rocksdb.go:1104  batch [1/52/0] commit took 1.176110551s (>500ms):
goroutine 68 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc422dcf180, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420848a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc4274310e0, 0xc423393380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x13a)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
I170217 20:34:41.485249 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.639447 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.663180 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:41.680203 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:41.834428 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:41.841382 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:41.859988 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.088284 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:42.124207 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.281680 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:42.358255 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:42.381397 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.408789 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:42.512351 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.596592 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:42.677753 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.707281 61 server/status/runtime.go:228  runtime stats: 3.1 GiB RSS, 147 goroutines, 68 MiB/250 MiB/421 MiB GO alloc/idle/total, 2.0 GiB/2.6 GiB CGO alloc/total, 41736.69cgo/sec, 2.61/0.37 %(u/s)time, 0.00 %gc (21x)
I170217 20:34:42.758961 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:42.779437 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:42.851996 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:42.997440 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:43.048180 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:43.082317 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:43.094667 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:43.371629 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:43.933275 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:44.282197 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:44.938911 2385739 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 623070 (id=e97f3758, encoded size=57255573, 55 rocksdb batches, 131 log entries)
I170217 20:34:44.992975 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:45.118176 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:46.035046 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:46.196968 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:46.848802 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:47.106254 15 storage/engine/rocksdb.go:1104  batch [3/233/3] commit took 1.094133896s (>500ms):
goroutine 15 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc4255e1500, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc4207cae00, 0x7f8a0822dbe8, 0xc42b5ed920, 0xc43c414140, 0x8, 0x0, 0x2612, 0x1950, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc4207cae00, 0x7f8a0822dbe8, 0xc42b5ed920, 0xc43c414140, 0x8, 0x2612, 0x200000002, 0x2, 0xc427631770, 0x1950, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4207cae00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b583050, 0xc429b8e900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xaa)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.106392 16 storage/engine/rocksdb.go:1104  batch [3/230/3] commit took 1.09415007s (>500ms):
goroutine 16 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc428c26c40, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc420726000, 0x7f8a0822dbe8, 0xc42b5a6b70, 0xc43c416830, 0x8, 0x0, 0x39b1, 0x19c0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc420726000, 0x7f8a0822dbe8, 0xc42b5a6b70, 0xc43c416830, 0x8, 0x39b1, 0x200000002, 0x2, 0xc430a738b0, 0x19c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420726000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc429e5c810, 0xc423b6e180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xa)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.106462 14 storage/engine/rocksdb.go:1104  batch [2/694/0] commit took 1.072841065s (>500ms):
goroutine 14 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc4253ce700, 0x7f8a0822db00, 0xc424242d80, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc42366bc00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b583aa0, 0xc446f4de00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x11b)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090aea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.106947 68 storage/engine/rocksdb.go:1104  batch [1/51/0] commit took 1.092844433s (>500ms):
goroutine 68 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42537d340, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420253180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc429ef3650, 0xc420b58600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0xab)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.109411 67 storage/engine/rocksdb.go:1104  batch [1/51/0] commit took 1.075794639s (>500ms):
goroutine 67 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42537d6c0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4208cb500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc429ef3890, 0xc424aa9800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x6f)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090af20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.110292 12 storage/engine/rocksdb.go:1104  batch [2/712/0] commit took 1.07569279s (>500ms):
goroutine 12 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42537d500, 0x7f8a0822db00, 0xc420835d10, 0x27f2260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc42084ae00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2718 +0x51c
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc429ef37d0, 0xc424aa9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x138)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.110440 13 storage/engine/rocksdb.go:1104  batch [3/230/3] commit took 1.080092745s (>500ms):
goroutine 13 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc428c26e00, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand(0xc4208a8000, 0x7f8a0822dbe8, 0xc42089fe90, 0xc440a40ba0, 0x8, 0x0, 0x2e9b, 0x1cad, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3851 +0x518
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc4208a8000, 0x7f8a0822dbe8, 0xc42089fe90, 0xc440a40ba0, 0x8, 0x2e9b, 0x300000003, 0x3, 0xc4226551d0, 0x1cad, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3631 +0x622
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc4208a8000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2772 +0x886
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc42b583ec0, 0xc424aa8c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4203b0700, 0x2c)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3385 +0x17d
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4202bb550, 0xc4203a5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:234 +0x267
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:174 +0x33
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a5950, 0xc42090ae80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:209 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x66
W170217 20:34:47.111354 2385739 storage/engine/rocksdb.go:1104  batch [133/56619/60] commit took 1.09428002s (>500ms):
goroutine 2385739 [running]:
runtime/debug.Stack(0x1a13a00, 0x27e7c20, 0xc4200124e0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc423fda1c0, 0x0, 0x20, 0x7f8a0822dbe8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1104 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc423bb9500, 0x7f8a0822dbe8, 0xc429c5c1b0, 0x374c4d7158377fe9, 0xb3c9dc37466d5688, 0xc4276ac000, 0x37, 0x40, 0xc420daa000, 0x83, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:670 +0xb30
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc423bb9500, 0x374c4d7158377fe9, 0xb3c9dc37466d5688, 0xc4276ac000, 0x37, 0x40, 0xc420daa000, 0x83, 0x8a, 0xc426e658c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2660 +0x12ea
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc4203b0700, 0x7f8a0822dbe8, 0xc423103410, 0xc426e65980, 0x374c4d7158377fe9, 0xb3c9dc37466d5688, 0xc4276ac000, 0x37, 0x40, 0xc420daa000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3149 +0xcb0
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc4203b0700, 0xc426e658c0, 0x7f8a0347b218, 0xc44ab4b160, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2795 +0x5a3
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x27f1260, 0xc44ab4b160, 0xc42042bf80, 0x7f8a0822dbe8, 0xc427949d10, 0xc420021738, 0xc420021740)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x7f8a0822dbe8, 0xc427949d10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203a5950, 0x1b6618f, 0x3, 0x1, 0x0, 0x0, 0xc427949da0, 0x7f8a0822dbe8, 0xc427949d10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:283 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:284 +0x14a
I170217 20:34:47.111970 2385739 storage/replica_raftstorage.go:604  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applied Raft snapshot in 2173ms [clear=906ms batch=167ms entries=0ms commit=1100ms]
I170217 20:34:47.409362 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:47.534825 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:47.631473 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:47.670029 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:47.690017 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:47.719869 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:48.108142 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.172580 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.213386 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.274857 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.307871 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.446029 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.482290 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:48.521239 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
W170217 20:34:48.522801 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.608849 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.626858 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.652091 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.732866 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.793556 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.849170 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.859956 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:48.879524 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:48.908410 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:48.952405 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.050727 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.167542 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.206399 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.307437 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.397347 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.448144 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:49.476621 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.498216 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.520918 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.590380 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.696592 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.784885 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.886073 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
W170217 20:34:49.905584 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:49.932463 64 sql/backfill.go:620  index backfill: process [{Key:/Table/51/1/2377842537040075872/"97f8faf4-678c-4a78-a6ff-16c5f8864392"/749 EndKey:/Table/51/1/3086603066031747482/"e7a6b562-557f-498f-9950-4f8789abd3a8"/679} {Key:/Table/51/1/3126748045378381513/"fae243d5-d242-4dcc-a1f9-85fbdc81eb0e"/1400 EndKey:/Table/51/2}] spans
I170217 20:34:49.934449 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
W170217 20:34:49.982794 64 sql/schema_changer.go:835  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable
I170217 20:34:50.522250 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
I170217 20:34:50.915302 147 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.142.0.4:26257: getsockopt: connection refused"; Reconnecting to {rebal-3to5s-kickassjennings-cockroach-1:26257 <nil>}
I170217 20:34:51.049586 2394676 storage/replica_raftstorage.go:596  [n2,s2,r361/2:/Table/5{1/1/9213…-2},@c423bb9500] applying Raft snapshot at index 627826 (id=1f31403b, encoded size=57380615, 55 rocksdb batches, 2 log entries)
I170217 20:34:51.450038 40 vendor/google.golang.org/grpc/clientconn.go:766  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 35.185.33.52:26257: getsockopt: connection refused"; Reconnecting to {35.185.33.52:26257 <nil>}
^C

The text was updated successfully, but these errors were encountered:

vivekmenezes · 2017-03-05T21:33:24Z

this might have been resolved by #13822

vivekmenezes · 2017-03-07T18:59:43Z

Looks like this has been fixed by #13822 . I was able to reproduce this at will and am unable to do so now.

vivekmenezes · 2017-03-08T14:18:34Z

This might still be an issue. One schema change I see didn't terminate.

vivekmenezes · 2017-03-09T23:46:44Z

Closing this issue because the schema change did terminate.

vivekmenezes · 2017-03-10T21:11:59Z

One node on a cluster (node 1) panicked due to a bug in distsql

I170310 18:47:21.263516 545020 sql/distsqlrun/indexbackfiller.go:334  [n1,IndexBackfiller=51] completed processing of span: {Key:/Table/51/1/4401262457340330400/"a73ca9fb-b64f-4219-b8a8-a76e652b0190"/891 EndKey:/Table/51/1/4464456621538168855/"a71a96f9-e8a8-4917-b6ba-db814b4c8546"/3119}
I170310 18:47:21.264011 545020 sql/distsqlrun/indexbackfiller.go:339  [n1,IndexBackfiller=51] ckpt [{Key:/Table/51/1/2479350534246819265/"11879714-011e-41e4-a0da-9fe0485e6ccb"/2845 EndKey:/Table/51/1/2622619856395811436/"1b403a61-2130-4fd5-bf29-4b608da11807"/321} {Key:/Table/51/1/2689430885444500288/"25bc8686-bf8d-482d-a74b-1e96724f4c03"/1283 EndKey:/Table/51/1/2756612945354747666/"9bdaa9fb-6b7d-4eb9-9cff-2b7c1fe3f321"/2429} {Key:/Table/51/1/2900491007894638123/"99486abc-2c62-4d12-a12f-390c736754c0"/2725 EndKey:/Table/51/1/2972176666681397438/"7e540726-d30e-44b1-b1d1-2a4b200967dd"/2153} {Key:/Table/51/1/3239780956121557661/"a71a96f9-e8a8-4917-b6ba-db814b4c8546"/985 EndKey:/Table/51/1/3361407671418611930/"91f4033f-30a0-43f1-86b0-546bbd28bbb8"/771} {Key:/Table/51/1/3402367222481659435/"a73ca9fb-b64f-4219-b8a8-a76e652b0190"/490 EndKey:/Table/51/1/3482857642336534619/"d4e6eb07-a66b-4c0a-89b8-acd41ef2b545"/3631} {Key:/Table/51/1/3520298344897877430/"7282335d-1674-4545-b952-b564d217ec56"/2071 EndKey:/Table/51/1/3596221849763659454/"3dfaa635-020f-4f38-b55a-1feaf4999dd6"/1624} {Key:/Table/51/1/3631491913525932273/"99589645-3470-47e7-8aa5-ffbc16824b34"/2470 EndKey:/Table/51/1/3737987091483275596/"e8dd1cdd-9928-470f-8079-e3dc351164da"/354} {Key:/Table/51/1/3778293564327682961/"e1fb3e85-7eda-4ca8-afce-fa6eb0104e3b"/1287 EndKey:/Table/51/1/4401262457340330400/"a73ca9fb-b64f-4219-b8a8-a76e652b0190"/891} {Key:/Table/51/1/4464456621538168855/"a71a96f9-e8a8-4917-b6ba-db814b4c8546"/3119 EndKey:/Table/51/2}]
I170310 18:47:21.271078 545020 sql/distsqlrun/indexbackfiller.go:274  [n1,IndexBackfiller=51] exiting
panic: too many Close() calls

goroutine 559208 [running]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*MultiplexedRowChannel).Close(0xc421b084c0, 0x28b42c0, 0xc4242efec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:195 +0x1ab
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowRegistry).finishInboundStreamLocked(0xc420424920, 0xc420bc4c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_registry.go:274 +0xbf
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowRegistry).RegisterFlow.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_registry.go:157 +0x2d5
created by time.goFunc
	/usr/local/go/src/time/sleep.go:170 +0x44

But even with that panic distsql on node 2 still tries to talk to node 1

I170310 21:11:13.615246 116 sql/distsql_physical_planner.go:1336  creating plan diagram
I170310 21:11:13.615813 116 sql/distsql_physical_planner.go:1341  Plan diagram JSON:
{"nodeNames":["1","2"],"processors":[{"nodeIdx":0,"inputs":[],"core":{"title":"Backfiller","details":["blocks","Type: Index"]},"outputs":[]},{"nodeIdx":1,"inputs":[],"core":{"title":"No-op","details":[]},"outputs":[]},{"nodeIdx":1,"inputs":[],"core":{"title":"Response","details":[]},"outputs":[]}],"edges":[{"sourceProc":0,"sourceOutput":0,"destProc":1,"destInput":0},{"sourceProc":1,"sourceOutput":0,"destProc":2,"destInput":0}]}
I170310 21:11:13.615843 116 sql/distsql_physical_planner.go:1342  Plan diagram URL:
https://raduberinde.github.io/decode.html?eJykkDFLBDEQhXt_hbx6BHNlSrttThE7SbEmo4SLmZDJwsmx_102QTgttrGcN_O-N7wLsgQ-zp-ssK8wIBzgCKWKZ1WpmzyOpnCGvSfEXJa2yY7gpTLsBS22xLB4mP3pPabEFYTAbY6pc9-S-JOC8PJV2N5OOfAZbiXI0n5oK13lmP2co9xJ-RXxD9gza5GsvMtzBA4fPOpQWarnpyq-NzLGx37dhcDaxtaMYcpjtb11bTa75sMfs1tvvgMAAP__6qGQFw==
I170310 21:11:13.615858 116 sql/distsql_physical_planner.go:1346  running DistSQL plan
W170310 21:11:13.706741 116 sql/schema_changer.go:848  Error executing schema change: rpc error: code = 14 desc = grpc: the connection is unavailable

distsql ought to pick a node where the lease is

vivekmenezes · 2017-03-10T21:14:30Z

gossip on node 2 is not seeing node 1

{
  "infos": {
    "cluster-id": {
      "value": {
        "rawBytes": "pFznaAPYQO1P44VC0o63+t3I7ytx",
        "timestamp": {
          "wallTime": "1489179620794749142",
          "logical": 0
        }
      },
      "origStamp": "1489179620794751470",
      "ttlStamp": "9223372036854775807",
      "hops": 1,
      "nodeId": 3,
      "peerId": 3
    },
    "first-range": {
      "value": {
        "rawBytes": "Zkp0ZwMIARIAGg8EdXBkYXRlLWNsdXN0ZXIiBggFEAUYBCIGCAIQAhgCIgYIAxADGAUoBg==",
        "timestamp": {
          "wallTime": "1489179617009393297",
          "logical": 0
        }
      },
      "origStamp": "1489179617009395508",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "liveness:1": {
      "value": {
        "rawBytes": "JaKb+gMIARACGgwIsviyk5y3ptUUEAA=",
        "timestamp": {
          "wallTime": "1489171650143408292",
          "logical": 0
        }
      },
      "origStamp": "1489171650143410134",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "liveness:2": {
      "value": {
        "rawBytes": "Kx/RyAMIAhABGgwIj9KdzcCgqNUUEAA=",
        "timestamp": {
          "wallTime": "1489179656014385909",
          "logical": 0
        }
      },
      "origStamp": "1489179656014387382",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "liveness:3": {
      "value": {
        "rawBytes": "edT2AQMIAxABGgwImN2B+b2gqNUUEAA=",
        "timestamp": {
          "wallTime": "1489179655303459047",
          "logical": 0
        }
      },
      "origStamp": "1489179655303460936",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "liveness:4": {
      "value": {
        "rawBytes": "81+CBQMIBBABGgwIlK+LwMOgqNUUEAA=",
        "timestamp": {
          "wallTime": "1489179656794101023",
          "logical": 0
        }
      },
      "origStamp": "1489179656794102961",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "liveness:5": {
      "value": {
        "rawBytes": "HZGBEgMIBRABGgwI0PD/mcSgqNUUEAA=",
        "timestamp": {
          "wallTime": "1489179656980675157",
          "logical": 0
        }
      },
      "origStamp": "1489179656980676352",
      "ttlStamp": "9223372036854775807",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "node:2": {
      "value": {
        "rawBytes": "LOtqLAMIAhIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtMjoyNjI1NxoAIgA=",
        "timestamp": {
          "wallTime": "1489178297011217695",
          "logical": 0
        }
      },
      "origStamp": "1489178297011221500",
      "ttlStamp": "1489181897011217695",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "node:3": {
      "value": {
        "rawBytes": "mbyjygMIAxIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtMzoyNjI1NxoAIgA=",
        "timestamp": {
          "wallTime": "1489178300811843603",
          "logical": 0
        }
      },
      "origStamp": "1489178300811847609",
      "ttlStamp": "1489181900811843603",
      "hops": 1,
      "nodeId": 3,
      "peerId": 3
    },
    "node:4": {
      "value": {
        "rawBytes": "JfnS+gMIBBIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtNDoyNjI1NxoAIgA=",
        "timestamp": {
          "wallTime": "1489178414772748180",
          "logical": 0
        }
      },
      "origStamp": "1489178414772754512",
      "ttlStamp": "1489182014772748180",
      "hops": 2,
      "nodeId": 4,
      "peerId": 3
    },
    "node:5": {
      "value": {
        "rawBytes": "kK4bHAMIBRIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtNToyNjI1NxoAIgA=",
        "timestamp": {
          "wallTime": "1489178414977388728",
          "logical": 0
        }
      },
      "origStamp": "1489178414977393000",
      "ttlStamp": "1489182014977388728",
      "hops": 3,
      "nodeId": 5,
      "peerId": 3
    },
    "sentinel": {
      "value": {
        "rawBytes": "QEu03wPYQO1P44VC0o63+t3I7ytx",
        "timestamp": {
          "wallTime": "1489179617009379575",
          "logical": 0
        }
      },
      "origStamp": "1489179617009382684",
      "ttlStamp": "1489179737009379575",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "store:2": {
      "value": {
        "rawBytes": "UHepQQMIAhIAGjkIAhIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtMjoyNjI1NxoAIgAiEwiA4Ln7wwsQgMCeoNMKGMUBIEc=",
        "timestamp": {
          "wallTime": "1489179617011548905",
          "logical": 0
        }
      },
      "origStamp": "1489179617011552552",
      "ttlStamp": "1489179737011548905",
      "hops": 0,
      "nodeId": 2,
      "peerId": 0
    },
    "store:3": {
      "value": {
        "rawBytes": "ZgxrewMIAxIAGjkIAxIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtMzoyNjI1NxoAIgAiEwiA4Ln7wwsQgOCM1tMKGMkBIEY=",
        "timestamp": {
          "wallTime": "1489179620797422662",
          "logical": 0
        }
      },
      "origStamp": "1489179620797426454",
      "ttlStamp": "1489179740797422662",
      "hops": 1,
      "nodeId": 3,
      "peerId": 3
    },
    "store:4": {
      "value": {
        "rawBytes": "jcD3yQMIBBIAGjkIBBIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtNDoyNjI1NxoAIgAiEwiA4Ln7wwsQgMDUs9QKGNcBIEM=",
        "timestamp": {
          "wallTime": "1489179614787675550",
          "logical": 0
        }
      },
      "origStamp": "1489179614787680106",
      "ttlStamp": "1489179734787675550",
      "hops": 2,
      "nodeId": 4,
      "peerId": 3
    },
    "store:5": {
      "value": {
        "rawBytes": "J/7OTAMIBRIAGjkIBRIxCgN0Y3ASKnJlYmFsLTN0bzVzLWdyYXZlbGFtcG9ydC1jb2Nrcm9hY2gtNToyNjI1NxoAIgAiEwiA4Ln7wwsQgIDk79UKGNQBIEM=",
        "timestamp": {
          "wallTime": "1489179614977016757",
          "logical": 0
        }
      },
      "origStamp": "1489179614977020248",
      "ttlStamp": "1489179734977016757",
      "hops": 3,
      "nodeId": 5,
      "peerId": 3
    },
    "system-db": {
      "value": {
        "rawBytes": "7rL4fgMKLAoSiomIEmRhdGFibG9ja3MAAYuJEhYKBgqb0YEBZBIMCOjwz9ji4oGqFBAACigKDoqJiBJwaG90b3MAAYuJEhYKBjxqur4BaBIMCJr1hqbYq6bVFBAkCigKDoqJiBJzeXN0ZW0AAYuJEhYKBhJYYT4BAhIMCJOVgLnD24GqFBAACiwKEoqJiRJkZXNjcmlwdG9yAAGLiRIWCgaI7qzRAQYSDAiTlYC5w9uBqhQQAAoqChCKiYkSZXZlbnRsb2cAAYuJEhYKBiwVjAUBGBIMCJOVgLnD24GqFBAACiYKDIqJiRJqb2JzAAGLiRIWCgasbcghAR4SDAjcw8ut0qem1RQQAAonCg2KiYkSbGVhc2UAAYuJEhYKBs341fwBFhIMCJOVgLnD24GqFBAACisKEYqJiRJuYW1lc3BhY2UAAYuJEhYKBjy5H1QBBBIMCJOVgLnD24GqFBAACioKEIqJiRJyYW5nZWxvZwABi4kSFgoGyqOrWgEaEgwIk5WAucPbgaoUEAAKJAoKiomJEnVpAAGLiRIWCgZ798GBARwSDAiTlYC5w9uBqhQQAAonCg2KiYkSdXNlcnMAAYuJEhYKBl20GxgBCBIMCJOVgLnD24GqFBAACicKDYqJiRJ6b25lcwABi4kSFgoG6ftJewEKEgwIk5WAucPbgaoUEAAKKAoOiom6EmJsb2NrcwABi4kSFgoGjXs2AwFmEgwIstbT2uLigaoUEAAKKgoQiom8EmNvbW1lbnRzAAGLiRIWCgYeVmByAW4SDAiusa+Y2aum1RQQAAooCg6KibwScGhvdG9zAAGLiRIWCgaJfbUIAWwSDAjr5pH+2Kum1RQQAAonCg2KibwSdXNlcnMAAYuJEhYKBuReRVABahIMCJacmePYq6bVFBBqCjYKBYuJiYqJEi0KHbwY7UQDEhYKBnN5c3RlbRABGgoKCAoEcm9vdBAwEgwIk5WAucPbgaoUEAAKwAEKBYuJioqJErYBCqUB/sQ1mgMKnQEKCW5hbWVzcGFjZRgCIAEoATAAOgQIABAAQhgKCHBhcmVudElEEAEaBggBEAAYACAAMABCFAoEbmFtZRACGgYIBxAAGAAgADAAQhIKAmlkEAMaBggBEAAYACABMABIBFIlCgdwcmltYXJ5EAEYASIIcGFyZW50SUQiBG5hbWUwATACQABAAGACagoKCAoEcm9vdBAwgAEBiAEBmAEAEgwIk5WAucPbgaoUEAAKnAEKBYuJi4qJEpIBCoEB1BQMpgMKegoKZGVzY3JpcHRvchgDIAEoATAAOgQIABAAQhIKAmlkEAEaBggBEAAYACAAMABCGgoKZGVzY3JpcHRvchACGgYICBAAGAAgATAASANSFQoHcHJpbWFyeRABGAEiAmlkMAFAAGACagoKCAoEcm9vdBAwgAEBiAEBmAEAEgwIk5WAucPbgaoUEAAKqQEKBYuJjIqJEp8BCo4BN7p5NwMKhgEKBXVzZXJzGAQgASgBMAA6BAgAEABCGAoIdXNlcm5hbWUQARoGCAcQABgAIAAwAEIeCg5oYXNoZWRQYXNzd29yZBACGgYICBAAGAAgATAASANSGwoHcHJpbWFyeRABGAEiCHVzZXJuYW1lMAFAAGACagsKCQoEcm9vdBDwA4ABAYgBAZgBABIMCJOVgLnD24GqFBAACpMBCgWLiY2KiRKJAQp5omTGbQMKcgoFem9uZXMYBSABKAEwADoECAAQAEISCgJpZBABGgYIARAAGAAgADAAQhYKBmNvbmZpZxACGgYICBAAGAAgATAASANSFQoHcHJpbWFyeRABGAEiAmlkMAFAAGACagsKCQoEcm9vdBDwA4ABAYgBAZgBABIMCJOVgLnD24GqFBAACvoBCgWLiZOKiRLwAQrfAafIC4gDCtcBCgVsZWFzZRgLIAEoATAAOgQIABAAQhYKBmRlc2NJRBABGgYIARAAGAAgADAAQhcKB3ZlcnNpb24QAhoGCAEQABgAIAAwAEIWCgZub2RlSUQQAxoGCAEQABgAIAAwAEIaCgpleHBpcmF0aW9uEAQaBggFEAAYACAAMABIBVJCCgdwcmltYXJ5EAEYASIGZGVzY0lEIgd2ZXJzaW9uIgpleHBpcmF0aW9uIgZub2RlSUQwATACMAQwA0AAQABAAEAAYAJqCgoICgRyb290EAKAAQGIAQGYAQASDAiTlYC5w9uBqhQQAAr5AwoFi4mUiokS7wMK3gP8x7zIAwrWAwoIZXZlbnRsb2cYDCABKAIwADoMCLODlYPSp6bVFBAAQhkKCXRpbWVzdGFtcBABGgYIBRAAGAAgADAAQhkKCWV2ZW50VHlwZRACGgYIBxAAGAAgADAAQhgKCHRhcmdldElEEAMaBggBEAAYACAAMABCGwoLcmVwb3J0aW5nSUQQBBoGCAEQABgAIAAwAEIUCgRpbmZvEAUaBggHEAAYACABMABCIwoIdW5pcXVlSUQQBhoGCAgQABgAIAAqCXV1aWRfdjQoKTAASAdSOAoHcHJpbWFyeRABGAEiCXRpbWVzdGFtcCIIdW5pcXVlSUQwATAGQABAAEoKCAAQABoAIAAoAFoAYAJqCgoICgRyb290EAKAAQGIAQOYAQCyASYKB3ByaW1hcnkQABoJdGltZXN0YW1wGgh1bmlxdWVJRCABIAYoALIBIgoPZmFtXzJfZXZlbnRUeXBlEAIaCWV2ZW50VHlwZSACKAKyASAKDmZhbV8zX3RhcmdldElEEAMaCHRhcmdldElEIAMoA7IBJgoRZmFtXzRfcmVwb3J0aW5nSUQQBBoLcmVwb3J0aW5nSUQgBCgEsgEYCgpmYW1fNV9pbmZvEAUaBGluZm8gBSgFuAEGwgEAEgwIg5a8qtKnptUUEAAKxgIKBYuJlYqJErwCCqsCLVZw0wMKowIKCHJhbmdlbG9nGA0gASgBMAA6BAgAEABCGQoJdGltZXN0YW1wEAEaBggFEAAYACAAMABCFwoHcmFuZ2VJRBACGgYIARAAGAAgADAAQhcKB3N0b3JlSUQQAxoGCAEQABgAIAAwAEIZCglldmVudFR5cGUQBBoGCAcQABgAIAAwAEIcCgxvdGhlclJhbmdlSUQQBRoGCAEQABgAIAEwAEIUCgRpbmZvEAYaBggHEAAYACABMABCKAoIdW5pcXVlSUQQBxoGCAEQABgAIAAqDnVuaXF1ZV9yb3dpZCgpMABICFIqCgdwcmltYXJ5EAEYASIJdGltZXN0YW1wIgh1bmlxdWVJRDABMAdAAEAAYAJqCgoICgRyb290EAKAAQGIAQGYAQASDAiTlYC5w9uBqhQQAAqvAQoFi4mWiokSpQEKlAHAfNZmAwqMAQoCdWkYDiABKAEwADoECAAQAEITCgNrZXkQARoGCAcQABgAIAAwAEIVCgV2YWx1ZRACGgYICBAAGAAgATAAQhsKC2xhc3RVcGRhdGVkEAMaBggFEAAYACAAMABIBFIWCgdwcmltYXJ5EAEYASIDa2V5MAFAAGACagoKCAoEcm9vdBACgAEBiAEBmAEAEgwIk5WAucPbgaoUEAAKhwMKBYuJl4qJEv0CCuwCg0eqFAMK5AIKBGpvYnMYDyABKAEwADoECAAQAEIiCgJpZBABGgYIARAAGAAgACoOdW5pcXVlX3Jvd2lkKCkwAEIWCgZzdGF0dXMQAhoGCAcQABgAIAAwAEIeCgdjcmVhdGVkEAMaBggFEAAYACAAKgVub3coKTAAQhcKB3BheWxvYWQQBBoGCAgQABgAIAAwAEgFUiMKB3ByaW1hcnkQARgBIgJpZDABQABKCggAEAAaACAAKABaAFpGChdqb2JzX3N0YXR1c19jcmVhdGVkX2lkeBACGAAiBnN0YXR1cyIHY3JlYXRlZDACMAM4AUAAQABKCggAEAAaACAAKABaAGADagsKCQoEcm9vdBDwA4ABAYgBA5gBALIBSwofZmFtXzBfaWRfc3RhdHVzX2NyZWF0ZWRfcGF5bG9hZBAAGgJpZBoGc3RhdHVzGgdjcmVhdGVkGgdwYXlsb2FkIAEgAiADIAQoALgBAcIBABIMCNzDy63Sp6bVFBAACjoKBYuJuoqJEjEKIRiXraIDEhoKCmRhdGFibG9ja3MQMhoKCggKBHJvb3QQAhIMCOjwz9ji4oGqFBAACoAJCgWLibuKiRL2CArlCNGf2goDCt0ICgZibG9ja3MYMyAyKAUwADoMCOnRlKWwtKbVFBAAQhgKCGJsb2NrX2lkEAEaBggBEAAYACAAMABCGQoJd3JpdGVyX2lkEAIaBggHEAAYACAAMABCGQoJYmxvY2tfbnVtEAMaBggBEAAYACAAMABCGQoJcmF3X2J5dGVzEAQaBggIEAAYACAAMABCMQoGbmV3Y29sEAUaBggDEAAYACABKhkoJzEuNCc6OjpTVFJJTkc6OkRFQ0lNQUwpMABIBlJHCgdwcmltYXJ5EAEYASIIYmxvY2tfaWQiCXdyaXRlcl9pZCIJYmxvY2tfbnVtMAEwAjADQABAAEAASgoIABAAGgAgACgAWgBgA2oKCggKBHJvb3QQAnLRBRIpCgNmb28QAhgAIghibG9ja19pZDABOAI4A0AASgoIABAAGgAgACgAWgAYAiABKAIybho1u4n9LhCCT+EpjggSMWE5ZjgwM2UtYTQ0YS00YmY1LWJjOGUtYmZhMjlmMjI4Y2QwAAH3BgIiNbuJ/S6mHne4LejaEjkxZjQwMzNmLTMwYTAtNDNmMS04NmIwLTU0NmJiZDI4YmJiOAAB9wMDMm4aNbuJ/TT5MQ5uKCz4EjFhY2M4NzY3LTU1N2EtNDU0NS04MmQ2LTczYjhmYTFmN2E0ZgAB9wdHIjW7if01fqCeiFC44xI5N2Y4ZmFmNC02NzhjLTRhNzgtYTZmZi0xNmM1Zjg4NjQzOTIAAfcB7zJuGjW7if02lJ+qHycFWRIxZDQ5OTNhYy1kOWVmLTQ2MGMtYTk4OC1hMDc1YzU5M2JmNzUAAfcFqSI1u4n9NxbVpECY9wwSZDIzNmQzMjctYzNlMi00NzkzLThiN2ItNWQ4YTU2NmIxY2EwAAH3BtEybho2u4n9S8FyLqdOpq4SMTU0OTYyYzctZjE5MS00ZTRmLWE4ZWUtYzhkMjY4MjI1MTdlAAH3YUmIIjS7if1MApObjQ5wFxJjNGZjM2JlZS1jMGE1LTRhMTMtYjY4OC1jNmUyNDYyNmJiOGMAAfbPMm4aNbuJ/WbA/xUGFm3IEmQ0ZTZlYjA3LWE2NmItNGMwYS04OWI4LWFjZDQxZWYyYjU0NQAB9wOJIjW7if1nRQmyUCk3IRJlN2E2YjU2Mi01NTdmLTQ5OGYtOTk1MC00Zjg3ODlhYmQzYTgAAfcC0DJuGjW7if1rcbKOPue/JhJlN2E2YjU2Mi01NTdmLTQ5OGYtOTk1MC00Zjg3ODlhYmQzYTgAAfcFkyI1u4n9bIe32SU1lZISNmYxODM1ZjMtNDFjMC00ZWMyLTkwYTEtMDE3MGNmMDAzZTlhAAH3A6WAAQOIAQOYAQCyAT0KB3ByaW1hcnkQABoIYmxvY2tfaWQaCXdyaXRlcl9pZBoJYmxvY2tfbnVtGgZuZXdjb2wgASACIAMgBSgFsgEiCg9mYW1fNF9yYXdfYnl0ZXMQBBoJcmF3X2J5dGVzIAQoBLgBBcIBABIMCLCf2ZahoKjVFBABCjYKBYuJvIqJEi0KHUDqd30DEhYKBnBob3RvcxA0GgoKCAoEcm9vdBACEgwImvWGptirptUUECQKwAIKBYuJvYqJErYCCqUCTk3u0wMKnQIKBXVzZXJzGDUgNCgBMAA6BAgAEABCEgoCaWQQARoGCAEQABgAIAAwAEIaCgpwaG90b0NvdW50EAIaBggBEAAYACABMABCHAoMY29tbWVudENvdW50EAMaBggBEAAYACABMABCFAoEbmFtZRAEGgYIBxAAGAAgATAAQhcKB2FkZHJlc3MQBRoGCAcQABgAIAEwAEgGUiMKB3ByaW1hcnkQARgBIgJpZDABQABKCggAEAAaACAAKABaAGACagoKCAoEcm9vdBACgAEBiAEDmAEAsgFECgdwcmltYXJ5EAAaAmlkGgpwaG90b0NvdW50Ggxjb21tZW50Q291bnQaBG5hbWUaB2FkZHJlc3MgASACIAMgBCAFKAC4AQHCAQASDAiWnJnj2Kum1RQQagrXAwoFi4m+iokSzQMKvAOIreKgAwq0AwoGcGhvdG9zGDYgNCgBMAA6BAgAEABCHQoCaWQQARoGCAgQABgAIAAqCXV1aWRfdjQoKTAAQhYKBnVzZXJJRBACGgYIARAAGAAgATAAQhwKDGNvbW1lbnRDb3VudBADGgYIARAAGAAgATAAQhcKB2NhcHRpb24QBBoGCAcQABgAIAEwAEIYCghsYXRpdHVkZRAFGgYIAhAAGAAgATAAQhkKCWxvbmdpdHVkZRAGGgYIAhAAGAAgATAAQhkKCXRpbWVzdGFtcBAHGgYIBRAAGAAgATAASAhSIwoHcHJpbWFyeRABGAEiAmlkMAFAAEoKCAAQABoAIAAoAFoAWjkKCGJ5VXNlcklEEAIYASIGdXNlcklEIgl0aW1lc3RhbXAwAjAHOAFAAEAASgoIABAAGgAgACgAWgBgA2oKCggKBHJvb3QQAoABAYgBA5gBALIBXgoHcHJpbWFyeRAAGgJpZBoGdXNlcklEGgxjb21tZW50Q291bnQaB2NhcHRpb24aCGxhdGl0dWRlGglsb25naXR1ZGUaCXRpbWVzdGFtcCABIAIgAyAEIAUgBiAHKAC4AQHCAQASDAjr5pH+2Kum1RQQAAqhAwoFi4m/iokSlwMKhgNCtmuAAwr+AgoIY29tbWVudHMYNyA0KAEwADoECAAQAEIXCgdwaG90b0lEEAEaBggIEAAYACAAMABCJAoJY29tbWVudElEEAIaBggIEAAYACAAKgl1dWlkX3Y0KCkwAEIWCgZ1c2VySUQQAxoGCAEQABgAIAEwAEIXCgdtZXNzYWdlEAQaBggHEAAYACABMABCGQoJdGltZXN0YW1wEAUaBggFEAAYACAAMABIBlJGCgdwcmltYXJ5EAEYASIHcGhvdG9JRCIJdGltZXN0YW1wIgljb21tZW50SUQwATAFMAJAAEAAQABKCggAEAAaACAAKABaAGACagoKCAoEcm9vdBACgAEBiAEDmAEAogEnChRsZW5ndGgocGhvdG9JRCkgPSAxNhINY2hlY2tfcGhvdG9JRBgAsgFHCgdwcmltYXJ5EAAaB3Bob3RvSUQaCWNvbW1lbnRJRBoGdXNlcklEGgdtZXNzYWdlGgl0aW1lc3RhbXAgASACIAMgBCAFKAC4AQHCAQASDAiusa+Y2aum1RQQAAozCgWNiYiKiRIqChp4q+KqAwoACgAKABCAgEAYgICAICIECICjBRIMCJOVgLnD24GqFBAA",
        "timestamp": {
          "wallTime": "1489179657149432792",
          "logical": 0
        }
      },
      "origStamp": "1489179657149440741",
      "ttlStamp": "9223372036854775807",
      "hops": 2,
      "nodeId": 4,
      "peerId": 3
    }
  }
}

cuongdo · 2017-03-10T21:24:40Z

panic: too many Close() calls looks a lot like #13989.

cc @asubiotto @andreimatei

andreimatei · 2017-03-10T23:20:29Z

The "too many Close() calls" panic I think is being fixed by #13570 which avoids a connection timeout that (erroneously) resulted in this panic (and it'll also fix the panic). Streams used to connect only when the first results were being produced (which, I guess, for schema changes might be way late and beyond the timeout). Now we'll establish the stream early.

Vivek, you implied something about distsql connecting to the wrong node after that panic but I didn't understand. What's that about?

vivekmenezes · 2017-03-11T02:19:56Z

In the cluster the distsql gateway running on node 2 keeps trying to reach node 1 that has died with a panic. My current theory is that distsql uses the range descriptor cache to make this decision, and when node 1 dies, the node down event seen through the liveness gossip event, doesn't trigger a cleanup of the range descriptor cache. So we keep retrying the same node 1.

vivekmenezes · 2017-03-15T01:39:03Z

index backfill is now working fine, but I haven't tested what happens when a node fails. I'm pretty sure it will still fail. Adding andrei to this bug because I don't intend to work on the topic of distsql rerouting in the event of node failure, in the immediate future. Please ping the bug if anyone starts working on this.

dianasaur323 · 2017-04-18T14:28:46Z

@andreimatei is this something we care about finishing in 1.0? If so, we should move the milestone over so that we don't let it slip through the cracks.

RaduBerinde · 2017-04-18T17:21:37Z

This issue tracks implementing a mechanism for detecting and avoiding dead/unhealthy nodes during planning.

vivekmenezes self-assigned this Feb 17, 2017

vivekmenezes modified the milestones: 1.0, Q1 2017 Feb 20, 2017

knz mentioned this issue Feb 25, 2017

sql: schema updates have problematic shortcomings #13804

Closed

22 tasks

knz added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Feb 25, 2017

vivekmenezes closed this as completed Mar 7, 2017

vivekmenezes reopened this Mar 8, 2017

vivekmenezes closed this as completed Mar 9, 2017

vivekmenezes reopened this Mar 10, 2017

vivekmenezes assigned andreimatei Mar 15, 2017

RaduBerinde mentioned this issue Mar 22, 2017

distsql: tracking issue for enabling DistSQL by default #13666

Closed

18 tasks

andreimatei changed the title ~~sql: index backfill not terminating~~ distsql: index backfill not terminating Mar 22, 2017

andreimatei assigned RaduBerinde Apr 18, 2017

andreimatei modified the milestones: 1.0, Q1 2017 Apr 18, 2017

vivekmenezes removed their assignment Apr 18, 2017

RaduBerinde changed the title ~~distsql: index backfill not terminating~~ distsql: avoid dead nodes Apr 18, 2017

cuongdo unassigned andreimatei Apr 18, 2017

vivekmenezes mentioned this issue Apr 19, 2017

sql: ALTER TABLE kills cluster #15026

Closed

RaduBerinde mentioned this issue Apr 29, 2017

distsql: avoid unhealthy nodes during planning #15368

Merged

RaduBerinde closed this as completed in #15368 May 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

distsql: avoid dead nodes #13655

distsql: avoid dead nodes #13655

vivekmenezes commented Feb 17, 2017

vivekmenezes commented Mar 5, 2017

vivekmenezes commented Mar 7, 2017

vivekmenezes commented Mar 8, 2017

vivekmenezes commented Mar 9, 2017

vivekmenezes commented Mar 10, 2017

vivekmenezes commented Mar 10, 2017

cuongdo commented Mar 10, 2017

andreimatei commented Mar 10, 2017

vivekmenezes commented Mar 11, 2017

vivekmenezes commented Mar 15, 2017

dianasaur323 commented Apr 18, 2017

RaduBerinde commented Apr 18, 2017

distsql: avoid dead nodes #13655

distsql: avoid dead nodes #13655

Comments

vivekmenezes commented Feb 17, 2017

vivekmenezes commented Mar 5, 2017

vivekmenezes commented Mar 7, 2017

vivekmenezes commented Mar 8, 2017

vivekmenezes commented Mar 9, 2017

vivekmenezes commented Mar 10, 2017

vivekmenezes commented Mar 10, 2017

cuongdo commented Mar 10, 2017

andreimatei commented Mar 10, 2017

vivekmenezes commented Mar 11, 2017

vivekmenezes commented Mar 15, 2017

dianasaur323 commented Apr 18, 2017

RaduBerinde commented Apr 18, 2017