Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cockroach version panicked #1282

Closed
davepacheco opened this issue Jun 26, 2022 · 4 comments
Closed

cockroach version panicked #1282

davepacheco opened this issue Jun 26, 2022 · 4 comments

Comments

@davepacheco
Copy link
Collaborator

cockroach version panicked:

fatal error: panic while printing panic value

goroutine 1 [running]:
runtime.throw(0x5a32586, 0x20)
        /opt/ooce/go-1.16/src/runtime/panic.go:1117 +0x72 fp=0xc00103f040 sp=0xc00103f010 pc=0x10a9512
runtime.preprintpanics.func1()
        /opt/ooce/go-1.16/src/runtime/panic.go:661 +0x56 fp=0xc00103f068 sp=0xc00103f040 pc=0x10d9a96
runtime.call16(0x0, 0x608e030, 0x0, 0x0)
        /opt/ooce/go-1.16/src/runtime/asm_amd64.s:550 +0x3e fp=0xc00103f088 sp=0xc00103f068 pc=0x10e1abe
runtime.reflectcallSave(0xc00103f1c0, 0x608e030, 0x0, 0x0)
        /opt/ooce/go-1.16/src/runtime/panic.go:877 +0x58 fp=0xc00103f0b8 sp=0xc00103f088 pc=0x10a8ad8
runtime.runOpenDeferFrame(0xc000000180, 0xc0010a8000, 0xc00103f208)
        /opt/ooce/go-1.16/src/runtime/panic.go:851 +0x62d fp=0xc00103f140 sp=0xc00103f0b8 pc=0x10a87ad
panic(0x52cd740, 0x6ee49d0)
        /opt/ooce/go-1.16/src/runtime/panic.go:965 +0x1b9 fp=0xc00103f208 sp=0xc00103f140 pc=0x10a8d59
runtime.panicmakeslicecap(...)
        /opt/ooce/go-1.16/src/runtime/slice.go:31
runtime.makeslice(0x5164c20, 0x0, 0x6f726b636f632fb5, 0x107c0c5)
        /opt/ooce/go-1.16/src/runtime/slice.go:95 +0xe5 fp=0xc00103f238 sp=0xc00103f208 pc=0x10c2125
strings.(*Builder).grow(...)
        /opt/ooce/go-1.16/src/strings/builder.go:68
strings.(*Builder).Grow(...)
        /opt/ooce/go-1.16/src/strings/builder.go:82
strings.Join(0xc0010a4000, 0x18, 0x1a, 0x59bb460, 0x3, 0x52ff600, 0x526d300)
        /opt/ooce/go-1.16/src/strings/strings.go:434 +0x4e7 fp=0xc00103f310 sp=0xc00103f238 pc=0x113d667
gopkg.in/yaml%2ev2.(*TypeError).Error(0xc000800048, 0x526d300, 0xc000800048)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/gopkg.in/yaml.v2/yaml.go:294 +0x59 fp=0xc00103f368 sp=0xc00103f310 pc=0x1b366f9
runtime.preprintpanics(0xc00103f448)
        /opt/ooce/go-1.16/src/runtime/panic.go:667 +0xa9 fp=0xc00103f3c8 sp=0xc00103f368 pc=0x10a7e69
panic(0x526d300, 0xc000800048)
        /opt/ooce/go-1.16/src/runtime/panic.go:1063 +0x552 fp=0xc00103f490 sp=0xc00103f3c8 pc=0x10a90f2
github.com/cockroachdb/cockroach/pkg/util/log/logconfig.DefaultConfig(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/logconfig/config.go:74 +0x105 fp=0xc00103f4f0 sp=0xc00103f490 pc=0x1b3a6c5
github.com/cockroachdb/cockroach/pkg/cli.(*logConfigFlags).reset(0xc000efe900)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/cli/log_flags.go:289 +0x37 fp=0xc00103f800 sp=0xc00103f4f0 pc=0x4d72e57
github.com/cockroachdb/cockroach/pkg/cli.setCliContextDefaults()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/cli/context.go:221 +0x16a fp=0xc00103f820 sp=0xc00103f800 pc=0x4d4478a
github.com/cockroachdb/cockroach/pkg/cli.initCLIDefaults()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/cli/context.go:51 +0x193 fp=0xc00103f8a8 sp=0xc00103f820 pc=0x4d43153
github.com/cockroachdb/cockroach/pkg/cli.init.8()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/cli/flags.go:281 +0x37 fp=0xc00103fce8 sp=0xc00103f8a8 pc=0x4d620d7
runtime.doInit(0x9277c00)
        /opt/ooce/go-1.16/src/runtime/proc.go:6315 +0xec fp=0xc00103fe38 sp=0xc00103fce8 pc=0x10b99ec
runtime.doInit(0x92220c0)
        /opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc00103ff88 sp=0xc00103fe38 pc=0x10b9972
runtime.main()
        /opt/ooce/go-1.16/src/runtime/proc.go:208 +0x205 fp=0xc00103ffe0 sp=0xc00103ff88 pc=0x10abd25
runtime.goexit()
        /opt/ooce/go-1.16/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00103ffe8 sp=0xc00103ffe0 pc=0x10e35a1

goroutine 35 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000119e10)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 34 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000119d40)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 36 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000119ee0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 37 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a000)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 38 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a0d0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 39 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a1a0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 40 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a270)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 41 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a340)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 42 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a410)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 43 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a4e0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 44 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a5b0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 45 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a680)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 46 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a750)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 47 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a820)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 48 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a8f0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 49 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00027a9c0)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173

goroutine 50 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x74
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35

goroutine 51 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d

goroutine 53 [syscall]:
os/signal.signal_recv(0x0)
        /opt/ooce/go-1.16/src/runtime/sigqueue.go:168 +0xa5
os/signal.loop()
        /opt/ooce/go-1.16/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
        /opt/ooce/go-1.16/src/os/signal/signal.go:151 +0x45

goroutine 117 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:165 +0x16b
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:157 +0x35

goroutine 119 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00087bd00)
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:276 +0xcd
created by go.opencensus.io/stats/view.init.0
        /ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68

How I got here: I tried to reproduce #1279 in hopes that it might be a simpler reproduction of #1130. I ran this:

for ((i = 0; ; i++)) { echo "ATTEMPT $i"; cockroach version || break; }

(in retrospect, the problem with that is that you won't have the exit status of cockroach version -- $? is 0 at the end of the loop)

I think it took close a couple of days and close to a million iterations (I interrupted it at one point and didn't note how many there were), but it eventually panicked with the output above.
@davepacheco
Copy link
Collaborator Author

This is on helios-1.0.21004. cockroach version:

Build Tag:        v21.2.9
Build Time:       2022/04/28 04:02:42
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.10
C Compiler:       gcc 10.3.0
Build Commit ID:  11787edfcfc157a0df951abc34684e4e18b3ef20
Build Type:       release

This is the stock build we've been using -- so jemalloc (not stdalloc).

@davepacheco davepacheco mentioned this issue Jul 1, 2022
Closed
@davepacheco
Copy link
Collaborator Author

Found a different panic today:

panic: invalid predicate with signature *fmt.pp != func(*scpb.Column, scplan.Params) bool for *scpb.Column[0]

goroutine 1 [running]:
github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan.buildSchemaChangeOpGenFunc(0x703b2a0, 0x0, 0xc000e0aff0, 0xc000e0b050, 0xc000e08c18)
	/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/declarative.go:171 +0xd93
github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan.buildSchemaChangePlanner(0xc000e0ab40, 0xc000e0ab40)
	/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/declarative.go:60 +0x125
github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan.init()
	/ws/gc/cockroach/cache/gopath/src/github.co

and filed these as cockroachdb/cockroach#83706.

@davepacheco
Copy link
Collaborator Author

I put the script that I used to try to reproduce this, plus some output, onto catacomb:

dap@catacomb /data/staff/core/issues $ ls -l omicron-1282
total 60
drwxrwxr-x+  2 dap      staff         10 Jul  1 15:31 attempt_2
drwxrwxr-x+  2 dap      staff         10 Jul  6 11:38 attempt_3
drwxrwxr-x+  2 dap      staff         10 Jul  7 20:27 attempt_4
-rwxrwxr-x+  1 dap      staff        583 Jul  6 15:23 try_repro_cockroach_version.sh
dap@catacomb /data/staff/core/issues $

@davepacheco davepacheco mentioned this issue Dec 16, 2022
Closed
@davepacheco
Copy link
Collaborator Author

These failures are covered by the CockroachDB bug I mentioned above and the follow-up cockroachdb/cockroach#83864.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@davepacheco