Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: testrace/TestScatter #15167

Closed
cockroach-teamcity opened this issue Apr 19, 2017 · 1 comment
Closed

teamcity: failed tests on master: testrace/TestScatter #15167

cockroach-teamcity opened this issue Apr 19, 2017 · 1 comment
Assignees
@a-robinson
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#228520:

--- FAIL: testrace/TestScatter (0.000s)
Test ended in panic.

------- Stdout: -------
F170419 17:45:20.816601 432658 storage/replica.go:966  [n1,s1,r5/5:/Table/51/1/{40-50},@c42652d500] lease repl={4 4 3} start=1492623920.789179643,0 epo=1 pro=1492623920.789191344,0 owned by replica {NodeID:4 StoreID:4 ReplicaID:3} that no longer exists
goroutine 432658 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x3db8801, 0x30ac5ada, 0x3405d40, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:837 +0xc4
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3407320, 0xc400000004, 0x2cd2821, 0x12, 0x3c6, 0xc4290d60d0, 0xc5)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:709 +0x660
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x7fa69990ea00, 0xc4239ae120, 0x4, 0x2, 0x21e6b5c, 0x33, 0xc421b59ed0, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:146 +0x313
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x7fa69990ea00, 0xc4239ae120, 0x1, 0x4, 0x21e6b5c, 0x33, 0xc421b59ed0, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:68 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x7fa69990ea00, 0xc4239ae120, 0x21e6b5c, 0x33, 0xc421b59ed0, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:162 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease.func1(0xc42652d500, 0xc421b5a2b8, 0x14b6dd8c377c7c42, 0x0, 0x7fa69990ea00, 0xc4239ae120, 0xc421b5a298, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:966 +0xe3a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease(0xc42652d500, 0x7fa69990ea00, 0xc4239ae120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1032 +0x1ed
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc42652d500, 0x7fa69990ea00, 0xc4239ae120, 0x14b6dd8c37731463, 0x0, 0x100000001, 0x1, 0x5, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2184 +0x1217
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc42652d500, 0x7fa69990ea00, 0xc4239ae120, 0x14b6dd8c37731463, 0x0, 0x100000001, 0x1, 0x5, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2070 +0xa9
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc42652d500, 0x7fa69990ea00, 0xc4239ae120, 0x14b6dd8c37731463, 0x0, 0x100000001, 0x1, 0x5, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1398 +0x28e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc421eda400, 0x7fa69990ea00, 0xc4239ae0f0, 0x14b6dd8c37731463, 0x0, 0x100000001, 0x1, 0x5, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2558 +0x9d3
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc423c6c180, 0x7fa69990ea00, 0xc4239ae090, 0x0, 0x0, 0x100000001, 0x1, 0x5, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24b
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:843 +0x227
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc4239b1d60, 0xc421b5b7e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:263 +0x125
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc42a2b22c0, 0x7fa69990ea00, 0xc4239ae060, 0xc425524668, 0xc4239ae060, 0x12d0100, 0x21bee50)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:854 +0x1ca
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc42a2b22c0, 0x7fa69990ea00, 0xc4239ae060, 0xc425524668, 0x1f, 0x305d3182, 0xc42dda8b98)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:871 +0xb8
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1.1(0xc4244428c0, 0xc425524640, 0x7fa69990ea00, 0xc428cf7fb0, 0x119893d, 0x7fa699907988, 0xc42006cd68)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:192 +0x73d
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1(0xc4244428c0, 0xc425524640, 0x7fa69990ea00, 0xc428cf7fb0, 0x2fdb720, 0x3db8a88, 0xc421211920)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:205 +0xce
created by github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:209 +0x224

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200752b0, 0x219da7b, 0xb, 0x2213848, 0xc4204a7b01)
	/usr/local/go/src/testing/testing.go:698 +0x582
testing.runTests.func1(0xc4200752b0)
	/usr/local/go/src/testing/testing.go:882 +0xab
testing.tRunner(0xc4200752b0, 0xc4204a7d10)
	/usr/local/go/src/testing/testing.go:657 +0x108
testing.runTests(0xc4201b35a0, 0x2fb0e40, 0x8a, 0x8a, 0x2cda194)
	/usr/local/go/src/testing/testing.go:888 +0x4e1
testing.(*M).Run(0xc42025bf28, 0x3db84e0)
	/usr/local/go/src/testing/testing.go:822 +0x1c4
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc4204a7f28)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:227 +0x18a
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/_test/_testmain.go:344 +0x210

goroutine 17 [syscall, 8 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 19 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3407320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1003 +0x85
created by github.com/cockroachdb/cockroach/pkg/util/log.init.1
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:566 +0x13a

goroutine 20 [syscall, 8 minutes]:
os/signal.signal_recv(0x60c191)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 431511 [semacquire]:
sync.runtime_notifyListWait(0xc424089d90, 0xc400000073)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc424089d80)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42646bd40, 0xc42c9d2d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42b985840)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431433 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42148f190)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 491 [select, locked to thread]:
runtime.gopark(0x2215a20, 0x0, 0x2196e4f, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0xfd
runtime.selectgoImpl(0xc420174750, 0xc420174738, 0x0)
	/usr/local/go/src/runtime/select.go:423 +0x138a
runtime.selectgo(0xc420174750)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x1ea
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 432147 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc421ef9a80, 0xc400000004, 0xc42b72b440, 0xc42ae94100, 0x2fd8bc0, 0xc4227f07b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:495 +0x977
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc421ef9a80, 0x4, 0x7fa69990ea00, 0xc423966900, 0x0, 0x0, 0xc42611fe00, 0xc42b72b440, 0xc42ae94100, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:439 +0x3d1
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc421ef9a80, 0x7fa69990ea00, 0xc423966900, 0x7fa600000004, 0xc42b72b440, 0xc42ae94100)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:440 +0x103
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:575 +0xfe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc423d33890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430772 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4226e6000, 0xc42245a480, 0xc427b3e230, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:356 +0xb5b
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:297 +0x86
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc427b89650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431952 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc422c15000, 0xc4245ed370, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc42919f0e0, 0xc4245ed370, 0x5, 0x5, 0x1b55a30, 0x7fa684333348, 0xc4261547f8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc42919f0e0, 0xc4245ed370, 0x5, 0x5, 0x5, 0xc4261547f8, 0x60d31c, 0x62bb43)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc42919f0e0, 0xc4245ed370, 0x5, 0x5, 0xc4261547f8, 0x4, 0x60d202)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4245ed360, 0x7fffffff, 0x34, 0x34, 0xc42b28dd48, 0x5d8d3b, 0x5bffcd, 0xc41fea1ea1)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4245ed360, 0x2fcf5c0, 0x3db8a88, 0xc42919f0e0, 0x2fc4680, 0x3db8a88, 0x20c5540, 0xc422bc2bc0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc4267b6be0, 0x20c5540, 0xc422bc2bc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:613 +0x1d5
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc4239dc8d0, 0x7fa69990ea00, 0xc422d89bc0, 0xc4213a0180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:351 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2fd8c20, 0xc4239dc8d0, 0xc421ef9a80, 0xc42919f108, 0xc420b76fa8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:300 +0x61
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:323 +0x61
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc422d89bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432188 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4240885c0, 0xc4227efbb0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc429f33440, 0xc4227efbb0, 0x5, 0x5, 0x1b55a30, 0x7fa6801db1c8, 0xc42646bd58)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc429f33440, 0xc4227efbb0, 0x5, 0x5, 0x5, 0xc42027bb78, 0x5ed781, 0xc421ff0340)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc429f33440, 0xc4227efbb0, 0x5, 0x5, 0x62b6d8, 0xc424089d90, 0xc42027bbf0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4227efba0, 0x7fffffff, 0x34, 0x34, 0xc42027bd48, 0x5d8d3b, 0x5bffcd, 0xc41fd73bf3)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4227efba0, 0x2fcf5c0, 0x3db8a88, 0xc429f33440, 0x2fc4680, 0x3db8a88, 0x20c5540, 0xc425188180, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc4255a6960, 0x20c5540, 0xc425188180, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:613 +0x1d5
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc42b7df010, 0x7fa69990ea00, 0xc42271ea80, 0xc423ac3200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:351 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2fd8c20, 0xc42b7df010, 0xc428ef9b80, 0xc429f33468, 0xc421b337a8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:300 +0x61
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:323 +0x61
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42271eab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431234 [semacquire]:
sync.runtime_notifyListWait(0xc42245a810, 0xc400000081)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42245a800)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425794000, 0xc42a92e640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc423dbab20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431168 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).controller(0xc427528700)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:742 +0xa23
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:161 +0xdf2

goroutine 431937 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc423279e00, 0xc42383ec50, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4210f4900, 0xc42383ec50, 0x5, 0x5, 0x1b55a30, 0x7fa68024cb88, 0xc42bb36018)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc4210f4900, 0xc42383ec50, 0x5, 0x5, 0x5, 0xc420279b78, 0x5ed781, 0xc429567380)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc4210f4900, 0xc42383ec50, 0x5, 0x5, 0x62b6d8, 0xc4255d5310, 0xc420279bf0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42383ec40, 0x7fffffff, 0x34, 0x34, 0xc420279d48, 0x5d8d3b, 0x5bffcd, 0xc41ff18728)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42383ec40, 0x2fcf5c0, 0x3db8a88, 0xc4210f4900, 0x2fc4680, 0x3db8a88, 0x20c5540, 0xc421cf1ae0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc42611e780, 0x20c5540, 0xc421cf1ae0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:613 +0x1d5
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc4267996a0, 0x7fa69990ea00, 0xc42cd492c0, 0xc423ac2f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:351 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2fd8c20, 0xc4267996a0, 0xc423c6c100, 0xc4210f4928, 0xc42c983fa8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:300 +0x61
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:323 +0x61
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc42cd492f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431108 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1247 +0xf95
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc4206e5b30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430754 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc42c9d1560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430928 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc426844860, 0x2214c98, 0xc42001aee0, 0x2fd8d40, 0xc420357c20)
	<autogenerated>:9 +0x9f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc42001aee0, 0x2fcefc0, 0xc426844860, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:396 +0x1f8
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:535 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc4268449c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431243 [semacquire]:
sync.runtime_notifyListWait(0xc42245a810, 0xc400000099)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42245a800)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425794000, 0xc42a92e640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc423dbac60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430973 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:761 +0x1c61
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc42034abe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431173 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:260 +0x2fc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc425892e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432555 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc42d99e300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1116 +0x912
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:281 +0x1954

goroutine 431264 [semacquire]:
sync.runtime_notifyListWait(0xc42245a810, 0xc40000008c)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42245a800)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425794000, 0xc42a92e640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc423dbafa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430690 [semacquire]:
sync.runtime_notifyListWait(0xc4255d5310, 0xc4000001eb)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4255d5300)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42bb36000, 0xc4239b1d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc424c38ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432034 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc422c15280, 0xc4245ed430, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc42919f320, 0xc4245ed430, 0x5, 0x5, 0x1b55a30, 0x7fa679199908, 0xc42bb36018)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc42919f320, 0xc4245ed430, 0x5, 0x5, 0x5, 0xc421c93b78, 0x5ed781, 0xc423b3a4e0)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc42919f320, 0xc4245ed430, 0x5, 0x5, 0x62b6d8, 0xc4255d5310, 0xc421c93bf0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4245ed420, 0x7fffffff, 0x68, 0x68, 0xc421c93d48, 0x5d8d3b, 0x5bffcd, 0xc41fbd6796)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4245ed420, 0x2fcf5c0, 0x3db8a88, 0xc42919f320, 0x2fc4680, 0x3db8a88, 0x20c5540, 0xc428530d20, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc4267b6d20, 0x20c5540, 0xc428530d20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:613 +0x1d5
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc4239dc940, 0x7fa69990ea00, 0xc422d89dd0, 0xc42b3f3070)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:351 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2fd8c20, 0xc4239dc940, 0xc423c6c100, 0xc42c3b97a8, 0xc42c3b97a8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:300 +0x61
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:323 +0x61
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc422d89e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431024 [semacquire]:
sync.runtime_notifyListWait(0xc428e134d0, 0xc4000000aa)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc428e134c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4261547e0, 0xc4290a5f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc4225b01c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431965 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42b892d80, 0xc426844310, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc42a6977a0, 0xc426844310, 0x5, 0x5, 0x34073c0, 0x0, 0x9990ea00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc42a6977a0, 0xc426844310, 0x5, 0x5, 0x5, 0x7fa69990ea00, 0xc4240198f0, 0x12d0100)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc42a6977a0, 0xc426844310, 0x5, 0x5, 0xc4231b6f20, 0x13e551d, 0xc42a2b22c0)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc426844300, 0x7fffffff, 0x7, 0xb0000000006, 0xc420020c00, 0x5bc21c, 0x5bcb1e, 0x7fa67d45a098)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc426844300, 0x2fcf5c0, 0x3db8a88, 0xc42a6977a0, 0x2fc4680, 0x3db8a88, 0x20c5620, 0xc428d08570, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42c0b2dc0, 0x20c5620, 0xc428d08570, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc426325520, 0x5b943b, 0x5abe7f, 0xc424a57ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:283 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2fd8bc0, 0xc426325520, 0xc42b263780, 0xc429551180, 0x7fa69990ea00, 0xc4206e4ae0, 0xc424a57f98, 0x5d8662)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:467 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:484 +0x113
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc428d08540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431455 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42148f320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431296 [semacquire]:
sync.runtime_notifyListWait(0xc42245a810, 0xc40000007e)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc42245a800)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425794000, 0xc42a92e640)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc423dbb400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430766 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:114 +0x61c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc421be9520)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431527 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3475 +0x481
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc421b23ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431215 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc4208dcf30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431328 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1355 +0x68b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc425206cd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430771 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc427e11860)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:859 +0x7cf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc427e11860)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:604 +0x267
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:605 +0xa4d

goroutine 432336 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4226c8ea0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:859 +0x7cf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4226c8ea0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:604 +0x267
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:605 +0xa4d

goroutine 430671 [semacquire]:
sync.runtime_notifyListWait(0xc4255d5310, 0xc400000210)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4255d5300)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42bb36000, 0xc4239b1d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc424c38880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431126 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc43f228240, 0xc424a3ad00, 0xc4203d92b0, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:356 +0xb5b
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:297 +0x86
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc4213a2e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430524 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc42b3920b0, 0x7fa69990ea00, 0xc420702060)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:219 +0xfce
github.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:203 +0xc0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc42c9d0440)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432616 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42c103fc0, 0xc426e42f90, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4202b5e60, 0xc426e42f90, 0x5, 0x5, 0x10, 0x10, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc4202b5e60, 0xc426e42f90, 0x5, 0x5, 0x5, 0x24, 0x4, 0x36)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc4202b5e60, 0xc426e42f90, 0x5, 0x5, 0xc421377958, 0xc42c0ceae0, 0xc420480400)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc426e42f80, 0x7fffffff, 0x3, 0x5bc21c, 0x5bcb1e, 0x5bc1be, 0x7fa64c2c8b48, 0xc42a796000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc426e42f80, 0x2fcf5c0, 0x3db8a88, 0xc4202b5e60, 0x2fc4680, 0x3db8a88, 0x20c5620, 0xc42d0335c0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42baf4420, 0x20c5620, 0xc42d0335c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc42c92bc80, 0x5b943b, 0x5abe7f, 0xc42c3ba6d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:283 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2fd8bc0, 0xc42c92bc80, 0xc42324d980, 0xc428ef9b80, 0x7fa69990ea00, 0xc420e6aa80, 0xc42c3ba798, 0x5d8662)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:467 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:484 +0x113
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42d033500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431018 [semacquire]:
sync.runtime_notifyListWait(0xc428e134d0, 0xc4000000a3)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc428e134c0)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4261547e0, 0xc4290a5f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc4225b0100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432169 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc420bcc600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1116 +0x912
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:281 +0x1954

goroutine 431067 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc42baa20d0, 0x7fa6999116f8, 0xc424a3ac80, 0xc42a43a480, 0x2fd7ba0, 0xc4266234c0, 0xc42a92e640, 0xc427930a40, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:325 +0x4f9
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:132 +0x6e5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc42a252640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432171 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42a149d40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:859 +0x7cf
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:592 +0x9fe

goroutine 430701 [semacquire]:
sync.runtime_notifyListWait(0xc4255d5310, 0xc4000001f5)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc4255d5300)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42bb36000, 0xc4239b1d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc424c38c40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431228 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42ae94900, 0xc42c565bd0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc424fb78c0, 0xc42c565bd0, 0x5, 0x5, 0xc42aa53a68, 0x0, 0xc42a937e70)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc424fb78c0, 0xc42c565bd0, 0x5, 0x5, 0x5, 0xc42aa53b30, 0x5b33cd, 0xc423cacc30)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc424fb78c0, 0xc42c565bd0, 0x5, 0x5, 0x5dbc64, 0xc42aa53b00, 0x608190)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42c565bc0, 0x7fffffff, 0x1b55a30, 0x7fa67bd7c708, 0xc424819440, 0x0, 0x0, 0xc42aa53bf0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42c565bc0, 0x2fcf5c0, 0x3db8a88, 0xc424fb78c0, 0x2fc4680, 0x3db8a88, 0x20bcf60, 0xc42403a320, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc424794580, 0x20bcf60, 0xc42403a320, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc42148e800, 0x7fa6999116f8, 0xc426c29200, 0xc42b5cd200)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:194 +0x86
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2fd7ba0, 0xc42148e800, 0xc42339d6c0, 0x7fa6999116f8, 0xc426c29200, 0xc42b5cd200, 0x30, 0xc427a51e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:313 +0x43
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:321 +0xf2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42b11f090)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430755 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc42c9d1570)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431787 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4226e6000, 0xc42335ecc0, 0xc429dc3530, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:356 +0xb5b
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:297 +0x86
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc4218e6240)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 430743 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239b1d60, 0xc42c9d1490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432186 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc424088400, 0xc4227efa30, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc429f33320, 0xc4227efa30, 0x5, 0x5, 0x1b56f30, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc429f33320, 0xc4227efa30, 0x5, 0x5, 0x5, 0x48, 0x0, 0x7fa673e6b6e8)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc429f33320, 0xc4227efa30, 0x5, 0x5, 0x5ba508, 0x20b5840, 0x2134f80)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4227efa20, 0x7fffffff, 0x3, 0x5bc21c, 0x5bcb1e, 0x7fa6962c93e0, 0xc41fedd25b, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4227efa20, 0x2fcf5c0, 0x3db8a88, 0xc429f33320, 0x2fc4680, 0x3db8a88, 0x20c5620, 0xc4219304e0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc425f74dc0, 0x20c5620, 0xc4219304e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc42b7deef0, 0x5b943b, 0x5abe7f, 0xc42422ced0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:283 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2fd8bc0, 0xc42b7deef0, 0xc42ae94140, 0xc421ef9a80, 0x7fa69990ea00, 0xc423966900, 0xc42422cf98, 0x5d8662)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:467 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:484 +0x113
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc42271e840)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431076 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x4b3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc42c565c40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432556 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc42d99e300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1162 +0x6e6
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:283 +0x1a4b

goroutine 431197 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:82 +0x1cb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc4208dcdf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431075 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x4b3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4290a5f40, 0xc42c565c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 432140 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc428e13bc0, 0xc42ca0f0f0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:143 +0x70a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc42674e000, 0xc42ca0f0f0, 0x5, 0x5, 0x1f, 0x5f25b0, 0xc427466a10)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:333 +0x7f
io.ReadAtLeast(0x2fbc840, 0xc42674e000, 0xc42ca0f0f0, 0x5, 0x5, 0x5, 0x2a, 0x0, 0x0)
	/usr/local/go/src/io/io.go:307 +0xb7
io.ReadFull(0x2fbc840, 0xc42674e000, 0xc42ca0f0f0, 0x5, 0x5, 0xc4224fea50, 0xc427466ad8, 0x6a4ab4)
	/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42ca0f0e0, 0x7fffffff, 0x0, 0x0, 0x0, 0x5bc1be, 0x7fa6999099b8, 0xc4204ce000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:243 +0xa0
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42ca0f0e0, 0x2fcf5c0, 0x3db8a88, 0xc42674e000, 0x2fc4680, 0x3db8a88, 0x20c5620, 0xc4277f0690, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:339 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc428b994a0, 0x20c5620, 0xc4277f0690, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:386 +0x20e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc421be9690, 0x5b943b, 0x5abe7f, 0xc42d155ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:283 +0x86
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2fd8bc0, 0xc421be9690, 0xc423670a80, 0xc429551180, 0x7fa69990ea00, 0xc4206e4ae0, 0xc42d155f98, 0x5d8662)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:467 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:484 +0x113
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42a92e640, 0xc423078690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431084 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc426dc4480)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1116 +0x912
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:281 +0x1954

goroutine 432183 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4218c4340)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:859 +0x7cf
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:592 +0x9fe

goroutine 431524 [semacquire]:
sync.runtime_notifyListWait(0xc424089d90, 0xc40000007a)
	/usr/local/go/src/runtime/sema.go:298 +0x11a
sync.(*Cond).Wait(0xc424089d80)
	/usr/local/go/src/sync/cond.go:57 +0x97
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42646bd40, 0xc42c9d2d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xe2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42b9859e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x74

goroutine 431232 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:701 +0x344
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42c9d2d20, 0xc42227e5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Apr 19, 2017
@a-robinson a-robinson self-assigned this Apr 25, 2017
@a-robinson
Copy link
Contributor

Same as #15003

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@a-robinson a-robinson

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@a-robinson @cockroach-teamcity