runtime: inappropriate deadlock detection on windows with syscall.NewCallback

[JackMordaunt]

What version of Go are you using (go version)?

# go version
go version go1.20.2 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

# go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\jack\AppData\Local\go-build
set GOENV=C:\Users\jack\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\jack\go\pkg\mod
set GONOPROXY=*
set GONOSUMDB=*
set GOOS=windows
set GOPATH=C:\Users\jack\go
set GOPRIVATE=*
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\jack\scoop\apps\go\current
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\jack\scoop\apps\go\current\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\jack\Src\go-toast\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\jack\AppData\Local\Temp\go-build1330827647=/tmp/go-build -gno-record-gcc-switches

What did you do?

When interacting with Windows COM, Go detects a "deadlock" while waiting for the Windows Runtime
to call the program back. In particular I have authored git.sr.ht/~jackmordaunt/go-toast which
implements Windows Toast Notifications. We create a "callback" COM object using a Vtable populated
with syscall.NewCallback function pointers.

The program issues the toast and waits for the Windows Runtime to invoke our syscall.NewCallback
wrapped Go function.

The Go runtime believes this is a deadlock, when in reality the Windows Runtime might call us back.

The following reproducer demonstrates the deadlock behavior. The non-deadlock function uses a noop goroutine to trick the Go runtime into not panicking. The reproduces invokes the git.sr.ht/~jackmordaunt/go-toast package for real because I don't know how to reproduce this without engaging the Windows Runtime.

Reproducer

package main

import (
	"flag"
	"fmt"
	"time"

	"git.sr.ht/~jackmordaunt/go-toast"
)

func main() {
	var (
		runDeadlock bool
	)

	flag.BoolVar(&runDeadlock, "deadlock", false, "run the deadlocking function")
	flag.Parse()

	if runDeadlock {
		deadlock()
	} else {
		noDeadlock()
	}
}

func noDeadlock() {
	done := make(chan struct{})

	toast.SetActivationCallback(func(args string, data []toast.UserData) {
		fmt.Printf("args %q, data %v\n", args, data)
		done <- struct{}{}
	})

	n := toast.Notification{
		AppID:          "deadlock test",
		Title:          "testing deadlock",
		Body:           "deadlock occurs when waiting on Windows Runtime",
		ActivationType: toast.Foreground,
	}

	n.Push()
        
        // Useless goroutine tricks the Go runtime to avoid deadlock panic. 
	go func() {
		for range time.NewTicker(time.Second).C {

		}
	}()

	<-done
}

func deadlock() {
	done := make(chan struct{})

	toast.SetActivationCallback(func(args string, data []toast.UserData) {
		fmt.Printf("args %q, data %v\n", args, data)
		done <- struct{}{}
	})

	n := toast.Notification{
		AppID:          "deadlock test",
		Title:          "testing deadlock",
		Body:           "deadlock occurs when waiting on Windows Runtime",
		ActivationType: toast.Foreground,
	}

	n.Push()

	<-done
}

What did you expect to see?

The program should wait until it is called back by via the syscall.NewCallback mechanism, rather than panicking.

What did you see instead?

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
main.deadlock()
        C:/Users/jack/Src/go-toast/examples/deadlock-bug/main.go:69 +0xb1
main.main()
        C:/Users/jack/Src/go-toast/examples/deadlock-bug/main.go:20 +0xbb
exit status 2

[prattmic]

I believe this is a duplicate of #55015