Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: testrace/TestScatter #15032

Closed
cockroach-teamcity opened this issue Apr 18, 2017 · 1 comment
Closed

teamcity: failed tests on master: testrace/TestScatter #15032

cockroach-teamcity opened this issue Apr 18, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#227229:

--- FAIL: testrace/TestScatter (0.000s)
Test ended in panic.

------- Stdout: -------
E170418 13:29:17.376682 430767 storage/replica_proposal.go:457  [n1,s1,r2/1:/{System/-Max},@c423c55500] could not load SystemConfig span: must retry later due to intent on SystemConfigSpan
E170418 13:29:17.547453 430795 storage/replica_proposal.go:457  [n1,s1,r4/1:/{System/tse-Max},@c420a94900] could not load SystemConfig span: must retry later due to intent on SystemConfigSpan
E170418 13:29:25.842520 435062 storage/replica.go:2586  [n3,s3,r27/5:/Table/51/1/{30-40},@c4244e4000] received invalid ChangeReplicasTrigger trigger:<change_type:REMOVE_REPLICA replica:<node_id:3 store_id:3 replica_id:5 > updated_replicas:<node_id:1 store_id:1 replica_id:6 > updated_replicas:<node_id:4 store_id:4 replica_id:2 > updated_replicas:<node_id:2 store_id:2 replica_id:4 > next_replica_id:7 >  to remove leaseholder replica {RaftAppliedIndex:40 LeaseAppliedIndex:25 Desc:r27:/Table/51/1/{30-40} [r5(n3,s3), r2(n4,s4), r4(n2,s2), r6(n1,s1), next=7] Lease:repl={3 3 5} start=1492522165.767188432,0 epo=1 pro=1492522165.767199732,0 TruncatedState:index:10 term:5  GCThreshold:0.000000000,0 Stats:{ContainsEstimates:false LastUpdateNanos:1492522165611136283 IntentAge:0 GCBytesAge:0 LiveBytes:240 LiveCount:10 KeyBytes:170 KeyCount:10 ValBytes:70 ValCount:10 IntentBytes:0 IntentCount:0 SysBytes:1252 SysCount:11} TxnSpanGCThreshold:0.000000000,0}
F170418 13:29:25.884938 435072 storage/replica.go:966  [n3,s3,r28/3:/Table/51/1/{40-50},@c42ada4c00] lease repl={4 4 2} start=1492522165.803460882,0 epo=1 pro=1492522165.803472982,0 owned by replica {NodeID:4 StoreID:4 ReplicaID:2} that no longer exists
goroutine 435072 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x410bb01, 0x34bf1845, 0x3743e20, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:837 +0xc4
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3745400, 0xc400000004, 0x2f85a69, 0x12, 0x3c6, 0xc422f3bad0, 0xc6)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:709 +0x660
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x7f47f5fd18d0, 0xc425a11350, 0x4, 0x2, 0x247a133, 0x33, 0xc4231e3d88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:146 +0x313
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x7f47f5fd18d0, 0xc425a11350, 0x1, 0x4, 0x247a133, 0x33, 0xc4231e3d88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:67 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x7f47f5fd18d0, 0xc425a11350, 0x247a133, 0x33, 0xc4231e3d88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:169 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease.func1(0xc42ada4c00, 0xc4231e4170, 0x14b681008cee8c9a, 0x0, 0x7f47f5fd18d0, 0xc425a11350, 0xc4231e4150, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:966 +0xe3a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease(0xc42ada4c00, 0x7f47f5fd18d0, 0xc425a11350, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1032 +0x1ed
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc42ada4c00, 0x7f47f5fd18d0, 0xc425a11350, 0x14b681008c5cd553, 0x0, 0x300000003, 0x3, 0x1c, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2184 +0x1217
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc42ada4c00, 0x7f47f5fd18d0, 0xc425a11350, 0x14b681008c5cd553, 0x0, 0x300000003, 0x3, 0x1c, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2070 +0xa9
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc42ada4c00, 0x7f47f5fd18d0, 0xc425a11350, 0x14b681008c5cd553, 0x0, 0x300000003, 0x3, 0x1c, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1398 +0x28e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4259a2000, 0x7f47f5fd18d0, 0xc425a110b0, 0x14b681008c5cd553, 0x0, 0x300000003, 0x3, 0x1c, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2558 +0x9d3
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc4225f2480, 0x7f47f5fd18d0, 0xc425a11050, 0x0, 0x0, 0x300000003, 0x3, 0x1c, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24b
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:843 +0x227
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc42b5a88c0, 0xc4231e5698, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:263 +0x125
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc42ade78c0, 0x7f47f5fd18d0, 0xc425a10ff0, 0xc427187490, 0xc425a10ff0, 0x0, 0x6db864)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:854 +0x1ca
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc42ade78c0, 0x7f47f5fd18d0, 0xc425a10ff0, 0xc427187490, 0xc42ade78c0, 0x7f47f3e6de98, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:871 +0xb8
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x23f8140, 0xc42ade78c0, 0x7f47f5fd18d0, 0xc425a10de0, 0xc427187420, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:1868 +0x35b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc424140460, 0x3315080, 0xc426ce3260, 0xc4232da900, 0xc420d7eab0, 0x32d6e10, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:738 +0xf68
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc424140460, 0x3315080, 0xc426ce3260, 0xc4232da900, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:932 +0x122b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42c5e70d0, 0xc424140460, 0x3315080, 0xc426ce3260, 0xc4232da900)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:497 +0xb7
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:498 +0xb9

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420075110, 0x243106c, 0xb, 0x24a6eb8, 0xc4204b3b01)
	/usr/local/go/src/testing/testing.go:698 +0x582
testing.runTests.func1(0xc420075110)
	/usr/local/go/src/testing/testing.go:882 +0xab
testing.tRunner(0xc420075110, 0xc4204b3d10)
	/usr/local/go/src/testing/testing.go:657 +0x108
testing.runTests(0xc4202577c0, 0x32eeea0, 0x8a, 0x8a, 0x2f8d60b)
	/usr/local/go/src/testing/testing.go:888 +0x4e1
testing.(*M).Run(0xc42073df28, 0x410b880)
	/usr/local/go/src/testing/testing.go:822 +0x1c4
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc4204b3f28)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:227 +0x18a
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/_test/_testmain.go:344 +0x210

goroutine 17 [syscall, 8 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 19 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3745400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1003 +0x85
created by github.com/cockroachdb/cockroach/pkg/util/log.init.1
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:566 +0x13a

goroutine 34 [syscall, 8 minutes]:
os/signal.signal_recv(0x6ff371)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 430787 [semacquire]:
sync.runtime_notifyListWait(0xc4204d3050, 0xc4000008dc)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4204d3040)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4200ba5a0, 0xc4228e0140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc4227eae20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431647 [semacquire]:
sync.runtime_notifyListWait(0xc4205cb250, 0xc4000006d9)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4205cb240)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4200ba510, 0xc42b5a88c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc4244db5e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431954 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc42a18e3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 471 [select, locked to thread]:
runtime.gopark(0x24a9090, 0x0, 0x242a447, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0xfd
runtime.selectgoImpl(0xc42003bf50, 0xc42003bf38, 0x0)
	/usr/local/go/src/runtime/select.go:423 +0x138a
runtime.selectgo(0xc42003bf50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x1ea
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 431511 [IO wait]:
net.runtime_pollWait(0x7f47f3e66f98, 0x72, 0x72db5a)
	/usr/local/go/src/runtime/netpoll.go:164 +0x5e
net.(*pollDesc).wait(0xc42899a148, 0x72, 0x32f3d68, 0xffffffffffffffff)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x5b
net.(*pollDesc).waitRead(0xc42899a148, 0xc423c56000, 0x400)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x42
net.(*netFD).Read(0xc42899a0e0, 0xc423c56000, 0x400, 0x400, 0x0, 0x32fe600, 0x32f3d68)
	/usr/local/go/src/net/fd_unix.go:250 +0x22b
net.(*conn).Read(0xc428552010, 0xc423c56000, 0x400, 0x400, 0x102, 0x6db864, 0x1c48e60)
	/usr/local/go/src/net/net.go:181 +0x97
net.(*TCPConn).Read(0xc428552010, 0xc423c56000, 0x400, 0x400, 0xc4223a96e8, 0xa4b5d6, 0xc4223a96e8)
	<autogenerated>:72 +0x73
crypto/tls.(*block).readFromUntil(0xc426461950, 0x7f47f22fe768, 0xc428552010, 0x5, 0xc428552010, 0x6e50f0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x114
crypto/tls.(*Conn).readRecord(0xc423c38000, 0x24a9117, 0xc423c38120, 0x6b2f22)
	/usr/local/go/src/crypto/tls/conn.go:590 +0x13d
crypto/tls.(*Conn).Read(0xc423c38000, 0xc426456000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x190
bufio.(*Reader).Read(0xc424644000, 0xc42c42a1f8, 0x9, 0x9, 0x0, 0x0, 0xc4223a9be0)
	/usr/local/go/src/bufio/bufio.go:213 +0x7e7
io.ReadAtLeast(0x32f5a00, 0xc424644000, 0xc42c42a1f8, 0x9, 0x9, 0x9, 0xc4203cf520, 0xffffffff, 0xc4297d2eb0)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x32f5a00, 0xc424644000, 0xc42c42a1f8, 0x9, 0x9, 0xd90268, 0xc42b6ac730, 0x0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42c42a1f8, 0x9, 0x9, 0x32f5a00, 0xc424644000, 0x0, 0xc400000000, 0xc42b0b7560, 0x32f6100)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0xa3
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42c42a1c0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x118
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc42bbcbad0, 0xc42b804fc0, 0xc42b804f00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x53
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc42b6ac600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1043 +0x15b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:251 +0x1294

goroutine 431736 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x4b3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc42bcaed40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431716 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1226 +0x19d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc4244dbc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432049 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1226 +0x19d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc424c10180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431656 [semacquire]:
sync.runtime_notifyListWait(0xc4205cb250, 0xc4000006a4)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4205cb240)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4200ba510, 0xc42b5a88c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc4244db700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432032 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000628)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431243 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:497 +0xad
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc420d69050)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431346 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc424e0ab90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431600 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:98 +0x8e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc42ab90a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431322 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc424e0a9f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431789 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:155 +0x74
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432118 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc42b3b3800)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1116 +0x912
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:281 +0x1954

goroutine 431877 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4242e04e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:859 +0x7cf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4242e04e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:604 +0x267
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:605 +0xa4d

goroutine 431987 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000650)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773940)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431330 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc424e0aa70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432679 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc429b9c500, 0x3316ca0, 0xc42197ced0, 0x3300480, 0xc429b9c500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:329 +0x2cd
github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x23f8300, 0xc429b9c500, 0x3314120, 0xc4250e4aa0, 0xc423cb35d8, 0xc42b937c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:332 +0xd6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc424140b60, 0x3315080, 0xc4238a48c0, 0xc42b937c20, 0xc421ea9a40, 0x32e1a00, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:842 +0xe22
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc424140b60, 0x3315080, 0xc4238a48c0, 0xc42b937c20, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:936 +0x11c2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc421db3850, 0xc424140b60, 0x3315080, 0xc4238a48c0, 0xc42b937c20)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:497 +0xb7
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:498 +0xb9

goroutine 435019 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Header(0xc4224bb560, 0x24a8408, 0xc42a9295b0, 0x3315020)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:252 +0x220
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvResponse(0x7f47f5fd18d0, 0xc42a9d2ed0, 0x0, 0x0, 0x330d640, 0x410be28, 0x0, 0x0, 0x3302700, 0x410be28, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:63 +0xc9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x7f47f5fd18d0, 0xc42a9d2ed0, 0x2454839, 0x21, 0x24088a0, 0xc42a44c8e8, 0x237a660, 0xc4258efb00, 0xc425d73680, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:265 +0xdf9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.Invoke(0x7f47f5fd18d0, 0xc42a9d2ed0, 0x2454839, 0x21, 0x24088a0, 0xc42a44c8e8, 0x237a660, 0xc4258efb00, 0xc425d73680, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:146 +0x217
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc427824048, 0x7f47f5fd18d0, 0xc42a9d2ed0, 0xc42a44c8e8, 0x0, 0x0, 0x0, 0x1, 0x410bda0, 0xc421f42b28)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:1845 +0x107
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1.1(0xc425958550, 0xc42a44c8c0, 0x7f47f5fd18d0, 0xc42a9d2ed0, 0x128b71d, 0x7f47f5fd18d0, 0xc421e9c540)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:196 +0x253
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1(0xc425958550, 0xc42a44c8c0, 0x7f47f5fd18d0, 0xc42a9d2ed0, 0x33197a0, 0x410be28, 0xc43977cc60)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:205 +0xce
created by github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:209 +0x224

goroutine 431790 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000637)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc4227738a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432028 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000641)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432106 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc42b038880, 0xc400000001, 0xc424a4a360, 0xc422332200, 0x3316c40, 0xc420c2f230, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:495 +0x977
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc42b038880, 0x1, 0x7f47f5fd18d0, 0xc4291c02d0, 0x0, 0x0, 0xc429aeeb40, 0xc424a4a360, 0xc422332200, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:439 +0x3d1
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc42b038880, 0x7f47f5fd18d0, 0xc4291c02d0, 0x7f4700000001, 0xc424a4a360, 0xc422332200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:440 +0x103
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:575 +0xfe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc428c3dc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432181 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func3(0x3315020, 0xc42b3b3800, 0xc429182a20, 0xc4258caf20)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:236 +0x52e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:256 +0x1389

goroutine 431963 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc42a18e490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432030 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000631)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432759 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42ac3ab80, 0xc4259d7b90, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc42ae40ea0, 0xc4259d7b90, 0x5, 0x5, 0x6af9fa, 0xc420030800, 0x30)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x32fa8c0, 0xc42ae40ea0, 0xc4259d7b90, 0x5, 0x5, 0x5, 0x5090abf3f615fc20, 0xc428c37ac8, 0x6ac61b)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x32fa8c0, 0xc42ae40ea0, 0xc4259d7b90, 0x5, 0x5, 0x13e36e9, 0xc428c37ac8, 0x8)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4259d7b80, 0x7fffffff, 0xc428c37c10, 0x6cbf1b, 0xc420023300, 0xc420023300, 0x1, 0x6dbd32)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4259d7b80, 0x330d640, 0x410be28, 0xc42ae40ea0, 0x3302700, 0x410be28, 0x2358c60, 0xc42429b7d0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42baa8c60, 0x2358c60, 0xc42429b7d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc42b33a460, 0x7f47f5fd18d0, 0xc42429b740, 0xc42881f920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:283 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x3316c40, 0xc42b33a460, 0xc4211cf740, 0xc429b9c500, 0x7f47f5fd18d0, 0xc421ea8c30, 0xc42c2e1798, 0x6cb842)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:467 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:484 +0x113
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc424e76300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431235 [chan receive]:
database/sql.(*DB).connectionOpener(0xc4222fe5a0)
	/usr/local/go/src/database/sql/sql.go:837 +0x6e
created by database/sql.Open
	/usr/local/go/src/database/sql/sql.go:582 +0x318

goroutine 431655 [semacquire]:
sync.runtime_notifyListWait(0xc4205cb250, 0xc4000006db)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4205cb240)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4200ba510, 0xc42b5a88c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc4244db6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431512 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc42b6ac600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1116 +0x912
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:281 +0x1954

goroutine 431992 [semacquire]:
sync.runtime_notifyListWait(0xc42d1ded10, 0xc400000620)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42d1ded00)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4282e4c60, 0xc42c45b860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc422773a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431400 [semacquire]:
sync.runtime_notifyListWait(0xc42c3ac9d0, 0xc400000575)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42c3ac9c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc428ff6090, 0xc4222fe640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc423356740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431213 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:260 +0x2fc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc426078000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431735 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x4b3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc42bcaed20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 435070 [chan receive]:
github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).lookupRangeDescriptorInternal(0xc429ffd3b0, 0x7f47f5fd18d0, 0xc425a10840, 0xc429d1a693, 0x3, 0xd, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:345 +0x8c6
github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).LookupRangeDescriptor(0xc429ffd3b0, 0x7f47f5fd18d0, 0xc425a10840, 0xc429d1a693, 0x3, 0xd, 0x0, 0x989600, 0x3b9aca00, 0x3ff8000000000000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:243 +0xb4
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).getDescriptor(0xc420335b00, 0x7f47f5fd18d0, 0xc425a10840, 0xc429d1a693, 0x3, 0xd, 0x0, 0xc4226c8800, 0x989680, 0x3b9aca00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:446 +0xca
github.com/cockroachdb/cockroach/pkg/kv.(*RangeIterator).Seek(0xc4235a1df8, 0x7f47f5fd18d0, 0xc425a10840, 0xc429d1a693, 0x3, 0xd, 0x700300)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_iter.go:163 +0x2f8
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc420335b00, 0x7f47f5fd18d0, 0xc425a10840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc420335b00, 0x7f47f5fd18d0, 0xc425a10840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:632 +0x3ca
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc42bdf3ce0, 0x7f47f5fd18d0, 0xc425a10840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:438 +0x490
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc42af931c0, 0x7f47f60128b0, 0xc424665560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:528 +0x228
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x7f47f60128b0, 0xc424665560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:464 +0x97
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x7f47f60128b0, 0xc424665560, 0xc422080dd8, 0xc424295500, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:436 +0x13b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc42af931c0, 0x7f47f60128b0, 0xc424665560, 0xc424295500, 0x2, 0xc4206a5840)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:464 +0xf8
github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).maybePushTransactions(0xc42d5b06c0, 0x7f47f60128b0, 0xc424665560, 0xc4232da360, 0x2, 0x2, 0x14b681008c11506e, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:202 +0xfbe
github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).processIntents(0xc42d5b06c0, 0x7f47f6012140, 0xc42006cd90, 0xc423991500, 0x33166a0, 0xc421d97280, 0xc4232da360, 0x2, 0x2, 0x14b681008c11506e, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:286 +0x1aa
github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).processIntentsAsync.func1(0x7f47f6012140, 0xc42006cd90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:258 +0xfd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func1(0xc42b5a88c0, 0x2f85e75, 0x1a, 0x103, 0xc424fda900, 0x0, 0x0, 0xc425a10750, 0x7f47f6012140, 0xc42006cd90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:348 +0x11a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x2c2

goroutine 430601 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:111 +0x4b1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc420a08760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430796 [semacquire]:
sync.runtime_notifyListWait(0xc4204d3050, 0xc4000008e8)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4204d3040)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4200ba5a0, 0xc4228e0140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc4227eaf60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432131 [runnable]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*quotaPool).add(0xc420957300, 0x1ec23b)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/control.go:122 +0x251
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).Write(0xc42b3b3800, 0xc429182a20, 0xc42a9e5c20, 0x47, 0x47, 0xc420c2e3b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:683 +0x11a3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc4258caf20, 0x2358b80, 0xc4209573e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:356 +0x40d
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Send(0xc4223b0c50, 0xc4209573e0, 0xc428710d80, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:278 +0x6c
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc420382280, 0x7f4700000004, 0xc428710d80, 0xc42aae9080, 0x3316c40, 0xc4223b0c50, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:518 +0x638
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc420382280, 0x4, 0x7f47f5fd18d0, 0xc4250d40c0, 0x1, 0x0, 0xc420cde0a0, 0xc428710d80, 0xc42aae9080, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:439 +0x3d1
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc420382280, 0x7f47f5fd18d0, 0xc4250d40c0, 0x7f4700000004, 0xc428710d80, 0xc42aae9080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:440 +0x103
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:575 +0xfe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc42d5197d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430606 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420a08d00, 0xc420258060, 0x225eb80, 0x32d5c20, 0x2340900)
	<autogenerated>:9 +0x9f
net/http.(*Server).Serve(0xc423ee26e0, 0x330d040, 0xc420a08d00, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x2a8
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:513 +0x54
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc420a08d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431585 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc4295be480)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1162 +0x6e6
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:283 +0x1a4b

goroutine 431812 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42ab90c40, 0xc423ac3b30, 0x225eb80, 0x32d5c20, 0x2340900)
	<autogenerated>:9 +0x9f
net/http.(*Server).Serve(0xc42aad9ce0, 0x330d040, 0xc42ab90c40, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x2a8
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:513 +0x54
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c45b860, 0xc42ab90ca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432661 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).controller(0xc420c328c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:742 +0xa23
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:161 +0xdf2

goroutine 432708 [IO wait]:
net.runtime_pollWait(0x7f478ed47978, 0x72, 0x72db5a)
	/usr/local/go/src/runtime/netpoll.go:164 +0x5e
net.(*pollDesc).wait(0xc42331aae8, 0x72, 0x32f3d68, 0xffffffffffffffff)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x5b
net.(*pollDesc).waitRead(0xc42331aae8, 0xc423fce800, 0x400)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x42
net.(*netFD).Read(0xc42331aa80, 0xc423fce800, 0x400, 0x400, 0x0, 0x32fe600, 0x32f3d68)
	/usr/local/go/src/net/fd_unix.go:250 +0x22b
net.(*conn).Read(0xc42c466ca8, 0xc423fce800, 0x400, 0x400, 0x102, 0x6db864, 0x1c48e60)
	/usr/local/go/src/net/net.go:181 +0x97
net.(*TCPConn).Read(0xc42c466ca8, 0xc423fce800, 0x400, 0x400, 0xc4251a16e8, 0xa4b5d6, 0xc4251a16e8)
	<autogenerated>:72 +0x73
crypto/tls.(*block).readFromUntil(0xc423fdeae0, 0x7f47f22fe768, 0xc42c466ca8, 0x5, 0xc42c466ca8, 0x6e50f0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x114
crypto/tls.(*Conn).readRecord(0xc4225af500, 0x24a9117, 0xc4225af620, 0x6b2f22)
	/usr/local/go/src/crypto/tls/conn.go:590 +0x13d
crypto/tls.(*Conn).Read(0xc4225af500, 0xc4256c2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x190
bufio.(*Reader).Read(0xc4227ee2a0, 0xc42411a8f8, 0x9, 0x9, 0x0, 0x0, 0xc4251a1be0)
	/usr/local/go/src/bufio/bufio.go:213 +0x7e7
io.ReadAtLeast(0x32f5a00, 0xc4227ee2a0, 0xc42411a8f8, 0x9, 0x9, 0x9, 0xc420c48b60, 0xffffffff, 0xc42377a500)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x32f5a00, 0xc4227ee2a0, 0xc42411a8f8, 0x9, 0x9, 0xd90268, 0xc42324b330, 0x0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42411a8f8, 0x9, 0x9, 0x32f5a00, 0xc4227ee2a0, 0x0, 0xc400000000, 0xc42056de60, 0x32f6100)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0xa3
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42411a8c0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x118
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc423dcdce0, 0xc422f2f740, 0xc422f2f700, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x53
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc42324b200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1043 +0x15b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:251 +0x1294

goroutine 431415 [semacquire]:
sync.runtime_notifyListWait(0xc42c3ac9d0, 0xc400000581)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42c3ac9c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc428ff6090, 0xc4222fe640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc423356960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430593 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x4b3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4228e0140, 0xc42c4a04a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431519 [IO wait]:
net.runtime_pollWait(0x7f47f2317400, 0x72, 0x72db5a)
	/usr/local/go/src/runtime/netpoll.go:164 +0x5e
net.(*pollDesc).wait(0xc4248723e8, 0x72, 0x32f3d68, 0xffffffffffffffff)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x5b
net.(*pollDesc).waitRead(0xc4248723e8, 0xc424df9000, 0x1000)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x42
net.(*netFD).Read(0xc424872380, 0xc424df9000, 0x1000, 0x1000, 0x0, 0x32fe600, 0x32f3d68)
	/usr/local/go/src/net/fd_unix.go:250 +0x22b
net.(*conn).Read(0xc42c4661c8, 0xc424df9000, 0x1000, 0x1000, 0xd19e31, 0x0, 0x24a8f80)
	/usr/local/go/src/net/net.go:181 +0x97
net.(*TCPConn).Read(0xc42c4661c8, 0xc424df9000, 0x1000, 0x1000, 0x1, 0x32f6100, 0xc42006c040)
	<autogenerated>:72 +0x73
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc429aee820, 0xc424df9000, 0x1000, 0x1000, 0xc429af34d0, 0xa4b5d6, 0xc429af34d0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x183
crypto/tls.(*block).readFromUntil(0xc425f1d320, 0x7f47f22fe7e0, 0xc429aee820, 0x5, 0xc429aee820, 0xc429af3538)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x114
crypto/tls.(*Conn).readRecord(0xc42b1f4e00, 0x24a9117, 0xc42b1f4f20, 0x6cbcea)
	/usr/local/go/src/crypto/tls/conn.go:590 +0x13d
crypto/tls.(*Conn).Read(0xc42b1f4e00, 0xc423cc6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x190
bufio.(*Reader).Read(0xc420d6d980, 0xc42197eab8, 0x9, 0x9, 0x7f479026a5c8, 0xc42091cff8, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x7e7
io.ReadAtLeast(0x32f5a00, 0xc420d6d980, 0xc42197eab8, 0x9, 0x9, 0x9, 0xc429af3a48, 0x71ed23, 0xc420d469c0)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x32f5a00, 0xc420d6d980, 0xc42197eab8, 0x9, 0x9, 0xc42cf2d6d0, 0x0, 0x1)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42197eab8, 0x9, 0x9, 0x32f5a00, 0xc420d6d980, 0x0, 0xc400000000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0xa3
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42197ea80, 0x0, 0x32f6100, 0xc42006c040, 0xc426b2d400)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x118
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc422110b70, 0xc42091d380, 0xc42091d380, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x53
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc42197ec40, 0xc422110cc0, 0x24a8340)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:307 +0x3c4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc424140b60, 0x3315080, 0xc42197ec40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:505 +0x1e8
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc424140b60, 0x3315bc0, 0xc42b1f4e00, 0x32fda00, 0xc4246dbe40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:486 +0x5a6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc424140b60, 0x3316dc0, 0xc429aee820)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:457 +0x5f4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:426 +0x69c

goroutine 431410 [semacquire]:
sync.runtime_notifyListWait(0xc42c3ac9d0, 0xc400000569)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42c3ac9c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc428ff6090, 0xc4222fe640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc4233568a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431344 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc424e0ab70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431371 [semacquire]:
sync.runtime_notifyListWait(0xc42c3ac9d0, 0xc400000588)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42c3ac9c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc428ff6090, 0xc4222fe640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4222fe640, 0xc42bfb3c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431623 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42b5a88c0, 0xc42006cbc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroac

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Apr 18, 2017
@a-robinson
Copy link
Contributor

#15003

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@a-robinson @cockroach-teamcity