Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

execinfra: panic decoding row in under high verbosity #54226

Closed
andreimatei opened this issue Sep 10, 2020 · 2 comments · Fixed by #54386
Closed

execinfra: panic decoding row in under high verbosity #54226

andreimatei opened this issue Sep 10, 2020 · 2 comments · Fixed by #54386
Assignees
@yuzefovich
Labels
A-sql-execution Relating to SQL execution. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Comments

@andreimatei
Copy link
Contributor

I got the following crash repeatedly under logspy on clusters running the restore2TB roachtest.
It's kind of a biggie; marking it as a release blocker.

panic: mismatched types ([int string timestamp bytes bytes string int bytes int int string timestamp bytes bytes string int bytes int]) and row ([{0 [253 8 43 230 49 202 28       128 1] <nil>} {0 [] <nil>} {0 [] <nil>} {0 [] <nil>} {0 [] <nil>} {0 [] <nil>} {0 [] <nil>} {2 [6 16 243 183 254 166 216 109 69 168 177 181 166 95 133 69 113 248] '\xf3b7fea      6d86d45a8b1b5a65f854571f8'} {2 [19 2] <nil>}])

goroutine 2438298 [running]:
panic(0x4c9e660, 0xc00f16d330)
	/usr/local/go/src/runtime/panic.go:722 +0x2c2 fp=0xc012cf9328 sp=0xc012cf9298 pc=0x7b15c2
github.com/cockroachdb/cockroach/pkg/sql/rowenc.EncDatumRow.stringToBuf(0xc0095708c0, 0x9, 0x9, 0xc010919170, 0x12, 0x12, 0xc012cf9410, 0xc012cf93e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/encoded_datum.go:388 +0x232 fp=0xc012cf93a8 sp=0xc012cf9328 pc=0x1a765f2
github.com/cockroachdb/cockroach/pkg/sql/rowenc.EncDatumRow.String(0xc0095708c0, 0x9, 0x9, 0xc010919170, 0x12, 0x12, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/encoded_datum.go:413 +0xf8 fp=0xc012cf9688 sp=0xc012cf93a8 pc=0x1a767d8
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcOutputHelper).ProcessRow(0xc0008cac18, 0x616f720, 0xc00c285100, 0xc0095708c0, 0x9, 0x9, 0xc0096fea00, 0x616f720, 0xc00c285100, 0xc0095708c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:294 +0x57e fp=0xc012cf9740 sp=0xc012cf9688 pc=0x21271ee
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).ProcessRowHelper(0xc0008cac00, 0xc0095708c0, 0x9, 0x9, 0x9, 0x0, 0x616f720)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:731 +0x75 fp=0xc012cf97c8 sp=0xc012cf9740 pc=0x2128105
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinReader).Next(0xc0008cac00, 0x0, 0x0, 0x0, 0xc008001b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinreader.go:411 +0xae fp=0xc012cf9830 sp=0xc012cf97c8 pc=0x236c0be
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run(0x616f720, 0xc00c285100, 0x619d4e0, 0xc0008cac00, 0x6147ca0, 0xc000ac4e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:170 +0x35 fp=0xc012cf9870 sp=0xc012cf9830 pc=0x2121595
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc0008cac00, 0x616f720, 0xc00c285100)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:763 +0x92 fp=0xc012cf98b0 sp=0xc012cf9870 pc=0x21282a2
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc00e295b00, 0x616f720, 0xc00c285100, 0x569e468, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:380 +0x24d fp=0xc012cf9960 sp=0xc012cf98b0 pc=0x233c46d
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000b51080, 0xc00976f500, 0xc00c38f9e0, 0xc0091e0800, 0xc002805500, 0xc0009f1510, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x5d9 fp=0xc012cf9f00 sp=0xc012cf9960 pc=0x33bdb19
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000b51080, 0x616f7e0, 0xc00ee13ad0, 0xc0009f1510, 0xc00976f500, 0xc00c38f9e0, 0x61721a0, 0xc0096fe140, 0x0, 0xc002805500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:991 +0x1d1 fp=0xc012cf9f78 sp=0xc012cf9f00 pc=0x33c0ea1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0009f1000, 0x616f7e0, 0xc00ee13ad0, 0xc0009f1420, 0x3, 0x61e6700, 0xc00c730b40, 0xc012cfa200, 0xc00c730bf8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1058 +0x3c0 fp=0xc012cfa0a8 sp=0xc012cf9f78 pc=0x335b400
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0009f1000, 0x616f7e0, 0xc00ee13ad0, 0xc0009f1420, 0x61e6700, 0xc00c730b40, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:929 +0x6ff fp=0xc012cfa358 sp=0xc012cfa0a8 pc=0x335a5df
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0009f1000, 0x616f7e0, 0xc00ee13ad0, 0x618e560, 0xc005e191d0, 0x5200678, 0x74, 0x4, 0x1, 0xc007c50c00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:609 +0xb58 fp=0xc012cfaee8 sp=0xc012cfa358 pc=0x33571f8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0009f1000, 0x616f7e0, 0xc00ee13ad0, 0x618e560, 0xc005e191d0, 0x5200678, 0x74, 0x4, 0x1, 0xc007c50c00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:116 +0x909 fp=0xc012cfb518 sp=0xc012cfaee8 pc=0x3355df9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc0009f1000, 0x616f7e0, 0xc00ee13a70, 0xc003a24690, 0xc00a4fa540, 0x4, 0x4, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:205 +0x27e fp=0xc012cfb6a8 sp=0xc012cfb518 pc=0x33564ae
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0009f1000, 0x616f720, 0xc00a4fa6c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1515 +0x10a7 fp=0xc012cfbe78 sp=0xc012cfb6a8 pc=0x334af87
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0009f1000, 0x616f7e0, 0xc017c81740, 0xc000e96e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1388 +0x1f2 fp=0xc012cfbf38 sp=0xc012cfbe78 pc=0x3349c92
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc0009f1000, 0x616f7e0, 0xc017c81740, 0xc000c30000, 0xc0019adc80, 0x0, 0xc00e24ed30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:175 +0x64 fp=0xc012cfbfa8 sp=0xc012cfbf38 pc=0x34d72c4
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc012cfbfb0 sp=0xc012cfbfa8 pc=0x7e1d91
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:174 +0x2f0
@andreimatei andreimatei added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-sql-execution Relating to SQL execution. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 10, 2020
@andreimatei andreimatei mentioned this issue Sep 10, 2020
Closed
@dt dt added branch-release-20.2 branch-master Failures and bugs on the master branch. labels Sep 14, 2020
@aayushshah15
Copy link
Contributor

@adityamaru and I ran into this while running TPCC 15k. I'm attaching my logs here in case it helps the investigation.
mismatched_types_panic.zip

@yuzefovich
Copy link
Member

I'm able to easily reproduce it when running with vmodule=processorsbase=4 with TPCH query 6, will dig into it.

@yuzefovich yuzefovich mentioned this issue Sep 15, 2020
Merged
@craig craig bot closed this as completed in 48e7f5a Sep 16, 2020
@yuzefovich yuzefovich mentioned this issue Sep 16, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
A-sql-execution Relating to SQL execution. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rowexec: fix column mapping for index join yuzefovich/cockroach
4 participants
@dt @andreimatei @yuzefovich @aayushshah15