cmd/compile/internal/ssa: "panic during regalloc" on linux/ppc64*

[bcmills]

#!watchflakes
post <- goarch ~ `ppc64.*` && log ~ `panic during regalloc` && log ~ `^\s*cmd/compile/internal/ssa`

greplogs -l -e '(?ms)panic during regalloc .*cmd/compile/internal/ssa\.\(\*sparseSet\)\.add'
2022-05-13T21:33:25-4170084/linux-ppc64le-power9osu

(attn @randall77 @dr2chase @cherrymui; CC @golang/ppc64)

[bcmills]

# encoding/json [encoding/json.test]
go/src/encoding/json/decode_test.go:2438:70: internal compiler error: 'TestUnmarshalRescanLiteralMangledUnquote': panic during regalloc while compiling TestUnmarshalRescanLiteralMangledUnquote:

runtime error: invalid memory address or nil pointer dereference

goroutine 46 [running]:
cmd/compile/internal/ssa.Compile.func1()
	/workdir/go/src/cmd/compile/internal/ssa/compile.go:50 +0x7c
panic({0x85e800, 0xebf7c0})
	/workdir/go/src/runtime/panic.go:882 +0x238
cmd/compile/internal/ssa.(*sparseSet).add(...)
	/workdir/go/src/cmd/compile/internal/ssa/sparseset.go:39
cmd/compile/internal/ssa.(*sparseSet).addAll(...)
	/workdir/go/src/cmd/compile/internal/ssa/sparseset.go:45
cmd/compile/internal/ssa.(*stackAllocState).computeLive(0xc0029394a0?, {0xc000f50c00?, 0x38?, 0x203030203030302?})
	/workdir/go/src/cmd/compile/internal/ssa/stackalloc.go:367 +0xe38
cmd/compile/internal/ssa.(*stackAllocState).init(0xc0029394a0, 0xc0010101c0, {0xc000f50c00, 0x38, 0x38})
	/workdir/go/src/cmd/compile/internal/ssa/stackalloc.go:126 +0x150
cmd/compile/internal/ssa.stackalloc(0xc0010101c0, {0xc000f50c00, 0x38, 0x38})
	/workdir/go/src/cmd/compile/internal/ssa/stackalloc.go:88 +0x124
cmd/compile/internal/ssa.(*regAllocState).regalloc(0xc001154180, 0xc0010101c0)
	/workdir/go/src/cmd/compile/internal/ssa/regalloc.go:1833 +0x1f8
cmd/compile/internal/ssa.regalloc(0xc0010101c0?)
	/workdir/go/src/cmd/compile/internal/ssa/regalloc.go:148 +0x50
cmd/compile/internal/ssa.Compile(0xc0010101c0)
	/workdir/go/src/cmd/compile/internal/ssa/compile.go:98 +0x928
cmd/compile/internal/ssagen.buildssa(0xc0012717c0, 0x0)
	/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:574 +0x1468
cmd/compile/internal/ssagen.Compile(0xc0012717c0, 0x0?)
	/workdir/go/src/cmd/compile/internal/ssagen/pgen.go:183 +0x48
cmd/compile/internal/gc.compileFunctions.func4.1(0x0?)
	/workdir/go/src/cmd/compile/internal/gc/compile.go:153 +0x4c
cmd/compile/internal/gc.compileFunctions.func3.1()
	/workdir/go/src/cmd/compile/internal/gc/compile.go:140 +0x54
created by cmd/compile/internal/gc.compileFunctions.func3
	/workdir/go/src/cmd/compile/internal/gc/compile.go:138 +0x8c

[bcmills]

There have been several changes in stackalloc.go during the Go 1.19 cycle, so marking as release-blocker at least until we can determine whether this is a regression.

[dr2chase]

It doesn't fail using the obvious cross-compilation:

GOOS=linux GOARCH=ppc64le go test -c encoding/json

which means, something wrong in that compiler?
About to try it on a gomote....

[bcmills]

It only failed once in the builder — looks like a nondeterministic failure.

My guesses would be either a race or (given the ssa package) a runtime bug involving a mishandled OOM condition.

[dr2chase]

Oh joy. I've already verified that it doesn't reproduce (once) with

GOPPC64=power9 GOOS=linux GOARCH=ppc64le go test -c encoding/json

nor (once) on a plain ppc64le gomote.

[dr2chase]

On the failing builder, with a normally built compiler

for i in {1..100} ; do  go clean -cache ; go test -c encoding/json ; echo $i done ; done

over lunch worked fine.
Am about to retry with GOPPC64=power9 built compiler.

[ceseo]

I can't reproduce it either (POWER9, with either GOPPC=power9 and GOPPC=power8.

[cherrymui]

There have been several changes in stackalloc.go during the Go 1.19 cycle

I don't think we have recent changes in stackalloc.go. The most recent change I saw is https://go-review.googlesource.com/c/go/+/312292 , which is a year old.

I also built the compiler in race mode on PPC64LE and couldn't find a race or a crash in a reasonable amount of time.

Given the above, and it does not occur on a first class port, I think it is not a release blocker. Thanks.

[ianlancetaylor]

As far as I can see this has happened exactly once.

Rolling forward to 1.20, but if it doesn't happen again we can close this.

[bcmills]

greplogs -l -e '(?ms)panic during regalloc .*cmd/compile/internal/ssa' --since=2022-05-16
2022-08-31T21:10:41-52d9e7f/linux-ppc64-sid-buildlet

[bcmills]

(That time was in (*sparseMap).set, but it seems like a close enough fit.)

[randall77]

That last one looks like corruption to me. There's no way a negative number should appear in the slice involved (cmd/compile/internal/ssa.sparseMap.sparse). Entries start out 0 and every value assigned is known nonnegative.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `panic during regalloc` && log ~ `^\s*cmd/compile/internal/ssa`

2023-11-20 18:34 linux-ppc64-sid-buildlet tools@bd215c0c go@e1dc209b github.com/yuin/goldmark/util [build] (log)

../../../../pkg/mod/github.com/yuin/[email protected]/util/html5entities.go:18:53: internal compiler error: 'init': panic during regalloc while compiling init:

runtime error: index out of range [134432465] with length 262144

goroutine 37 [running]:
cmd/compile/internal/ssa.Compile.func1()
	/workdir/go/src/cmd/compile/internal/ssa/compile.go:49 +0x7c
panic({0x939de0, 0xc003e42000})
	/workdir/go/src/runtime/panic.go:884 +0x240
cmd/compile/internal/ssa.(*sparseMapPos).set(...)
...
	/workdir/go/src/cmd/compile/internal/gc/compile.go:171 +0x4c
cmd/compile/internal/gc.compileFunctions.func3.1()
	/workdir/go/src/cmd/compile/internal/gc/compile.go:153 +0x44
created by cmd/compile/internal/gc.compileFunctions.func3
	/workdir/go/src/cmd/compile/internal/gc/compile.go:152 +0x1f4



Please file a bug report including a short program that triggers the error.
https://go.dev/issue/new

— watchflakes

[mknyszek]

CC @golang/ppc64

[pmur]

I suspect the most recent flake here is another manifestation of #64290. The failed index is 0x80346d1, and if the high bit is cleared, the index is valid.