Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl: TestCCLLogic seems to be stuck on a schema change #83664

Closed
yuzefovich opened this issue Jun 30, 2022 · 3 comments
Closed

ccl: TestCCLLogic seems to be stuck on a schema change #83664

yuzefovich opened this issue Jun 30, 2022 · 3 comments
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@yuzefovich
Copy link
Member

yuzefovich commented Jun 30, 2022

This is similar to #83630, but I just got a timeout on a schema change statement CREATE UNIQUE INDEX ON t_to_be_hashed (b) USING HASH;; and have the corresponding goroutine dump with this possibly related goroutine:

goroutine 456527 [select, 12 minutes]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc003414000)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:324 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00f5a8e70, {0x4e72b40, 0xc00b518800}, 0x66d02d0)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:969 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x65)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc00c8938c0, 0xc017705770, 0xc017705740)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:692 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc00c8938c0, {0x4e72b40, 0xc00b518800})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x11f
google.golang.org/grpc.invoke({0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x2}, {0x4f90ac0, 0xc010905000}, {0x4e72b40, 0xc00b518800}, 0x2, {0xc0110a44c0, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0xd7
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2({0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x7f42cbea7878}, {0x4f90ac0, 0xc010905000}, {0x4e72b40, 0xc00b518800}, 0x5b71a70cb1c6a20, 0x52cd558, ...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:255 +0x379
google.golang.org/grpc.(*ClientConn).Invoke(0x3127360e0b71fc88, {0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x0}, {0x4f90ac0, 0xc010905000}, {0x4e72b40, 0xc00b518800}, {0x0, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x223
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc005f99378, {0x6732dc8, 0xc00d5a1020}, 0x0, {0x0, 0x0, 0x0})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9361 +0xc9
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.TracingInternalClient.Batch({{0x67d94e8, 0xc005f99378}}, {0x6732dc8, 0xc00d5a1020}, 0xc010905000, {0x0, 0x0, 0x0})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:325 +0x186
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(_, {_, _}, _, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, ...}, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:210 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x9, 0x9, 0x3, ...}, ...}, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x145
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2055 +0x126f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1595 +0xd05
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1197 +0x426
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:823 +0x645
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xd8
github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWithAdmission({0x6732dc8, 0xc01b54bce0}, {0x66832c0, _}, {{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, ...}, ...}, ...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:458 +0x18d
github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:439
github.com/cockroachdb/cockroach/pkg/kv.SendWrapped({0x6732dc8, 0xc01b54bce0}, {0x66832c0, 0xc011f35c88}, {0x67daea8, 0xc0153b6370})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:475 +0xa5
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminScatter(0xc011f35c20, {0x6732dc8, 0xc01b54bce0}, {0xc01618c9d8, 0x8, 0x8}, 0x0)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:617 +0x23d
github.com/cockroachdb/cockroach/pkg/sql.splitAndScatter({0x6732dc8, 0xc01b54bce0}, 0x6801a38, {0xc01618c9d8, 0x8, 0x8}, {0xc005287500, 0x16, 0x1})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:3220 +0xe5
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).preSplitHashShardedIndexRanges.func1({0x6732dc8, 0xc01b54bce0}, 0x1800000000, 0x66c0610)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:3191 +0x6dd
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2({0x6732dc8, 0xc01b54bce0}, 0xc01a77d560)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:84 +0x216
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x6732dc8, 0xc01b54bce0}, 0x4b0b33)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:966 +0x27
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc018878be0, {0x6732dc8, 0xc01b54bce0}, 0xc017708870)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:959 +0xae
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x6732dc8, 0xc01b54bce0}, 0xc011f35c20, 0x44ffc7)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:965 +0x5a
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0xc01ba5e9c8, {0x6732dc8, 0xc01b54bce0}, 0x6732dc8, 0x0, 0xc01b54bce0)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:928 +0x89
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:907
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn(0xc00de38070, {0x6732dc8, 0xc01b54bce0}, {0x67d4498, 0xc00f06b020}, 0xc011f35c20, 0xc017708a30)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:79 +0x196
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txn(0xc01ba5ea60, {0x6732dc8, 0xc01b54bce0}, 0x6732dc8)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2520 +0x97
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).preSplitHashShardedIndexRanges(0xc01ba5ea90, {0x6732dc8, 0xc01b54bce0})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:3112 +0x3f
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill(0xc01b560580, {0x6732dc8, 0xc01b54bce0})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1949 +0x4d
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec(0xc01b560580, {0x6732dc8, 0xc01a77d560})
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:849 +0xa91
github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1(0x71, 0x2, 0x0)
	/home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2662 +0x508
github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume({0xc01ba5f5c0}, {0x6732dc8, 0xc01a77d560}, {0x4d953e0, 0xc001052338})

ccl.zip

Jira issue: CRDB-17198

@yuzefovich yuzefovich added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jun 30, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Jun 30, 2022
@ajwerner
Copy link
Contributor

ajwerner commented Jul 1, 2022

We're blocked in AdminScatter, in a weird redirection loop maybeTransferLeaseDuringLeaveJoint in. This is going to KV.

goroutine 458183 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc0159e8f60)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:127 +0x14a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1555 +0x7a5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1197 +0x426
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:823 +0x645
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xd8
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:1002 +0x14d
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:985
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x6732dc8, 0xc00ee61da0}, 0x8, 0xc0098d3080)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:847 +0xfc
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc00dec2750, {0x6732dc8, 0xc00ee61da0}, 0x7)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:870 +0x74
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminTransferLease(0xc016a89c18, {0x6732dc8, 0xc00ee61da0}, {0x4cc4080, 0xc004efb6f8}, 0x1c9e500)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:656 +0x76
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeTransferLeaseDuringLeaveJoint(0xc009fd5800, {0x6732dc8, 0xc00ee61da0}, 0xc0097bb880)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:1235 +0x305
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeLeaveAtomicChangeReplicas(0xc009fd5800, {0x6732dc8, 0xc00ee61da0}, 0xc0097bb880)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:1266 +0xec
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).changeReplicasImpl(0xc009fd5800, {0x6732dc8, 0xc00ee61da0}, 0x10, 0xcbea81d8, {0x4fe063d, 0x9}, {0xc01a2f6f30, 0
x8d}, {0xc0080e7488, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:1092 +0x5d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).changeReplicas(0xc011c3cf00, {0x6732dc8, 0xc00ee61da0}, 0x50470f4, {0xc0080e7488, 0x2, 0xc006e4f440}, 0xc
006e4f4a0, 0x6e4f500, {0x4fe063d, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicate_queue.go:1595 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).considerRebalance(0xc011c3cf00, {0x6732dc8, 0xc00ee61da0}, 0xc009fd5800, {0xc01167f140, 0x17, 0x4}, {0x0,
 0x0, 0x0}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicate_queue.go:1384 +0xc0a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChange(0xc011c3cf00, {0x6732dc8, 0xc00ee61da0}, 0x3b9aca00, 0x4000000000000000, 0x5, 0x0)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicate_queue.go:710 +0x11b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).adminScatter(_, {_, _}, {{{0xc002ce4c70, 0x8, 0x8}, {0xc002ce4c80, 0x9, 0x10}, 0x0}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:3526 +0x485
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch(0xc009fd5800, {0x6732dc8, 0xc00ee61ce0}, 0xc00b181f00)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:969 +0x103e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithoutRangeID(0xc009fd5800, {0x6732dc8, 0xc00ee61cb0}, 0xc00b181f00)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:183 +0x4aa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
 goroutine 456527 [select, 12 minutes]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc003414000)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:324 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00f5a8e70, {0x4e72b40, 0xc00b518800}, 0x66d02d0)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:969 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x65)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc00c8938c0, 0xc017705770, 0xc017705740)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:692 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc00c8938c0, {0x4e72b40, 0xc00b518800})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x11f
google.golang.org/grpc.invoke({0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x2}, {0x4f90ac0, 0xc010905000}, {0x4e72b40, 0xc00b518800}, 0x2, {0xc0110a44c0, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0xd7
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2({0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x7f42cbea7878}, {0x4f90ac0, 0xc010905000},
 {0x4e72b40, 0xc00b518800}, 0x5b71a70cb1c6a20, 0x52cd558, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:255 +0x379
google.golang.org/grpc.(*ClientConn).Invoke(0x3127360e0b71fc88, {0x6732dc8, 0xc00d5a1020}, {0x505dade, 0x0}, {0x4f90ac0, 0xc010905000}, {0x4e72b40, 0xc00b518800}, {0x0, ...
})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x223
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc005f99378, {0x6732dc8, 0xc00d5a1020}, 0x0, {0x0, 0x0, 0x0})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9361 +0xc9
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.TracingInternalClient.Batch({{0x67d94e8, 0xc005f99378}}, {0x6732dc8, 0xc00d5a1020}, 0xc010905000, {0x0, 0x0, 0x0})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:325 +0x186
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(_, {_, _}, _, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:210 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x9, 0x9, 0x3, ...}, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x145
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2055 +0x126f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1595 +0xd05
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1197 +0x426
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:823 +0x645
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xd8
github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWithAdmission({0x6732dc8, 0xc01b54bce0}, {0x66832c0, _}, {{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, ...}, ...}, ...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:458 +0x18d
github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(...)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:439
github.com/cockroachdb/cockroach/pkg/kv.SendWrapped({0x6732dc8, 0xc01b54bce0}, {0x66832c0, 0xc011f35c88}, {0x67daea8, 0xc0153b6370})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:475 +0xa5
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminScatter(0xc011f35c20, {0x6732dc8, 0xc01b54bce0}, {0xc01618c9d8, 0x8, 0x8}, 0x0)
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:617 +0x23d
github.com/cockroachdb/cockroach/pkg/sql.splitAndScatter({0x6732dc8, 0xc01b54bce0}, 0x6801a38, {0xc01618c9d8, 0x8, 0x8}, {0xc005287500, 0x16, 0x1})
        /home/yuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:3220 +0xe5

@ajwerner
Copy link
Contributor

ajwerner commented Jul 1, 2022

Seems like this is going to be #83687

@nvanbenschoten
Copy link
Member

Solved by 139dc42.

@healthy-pod healthy-pod added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Development

No branches or pull requests

4 participants