runtime: hangs on Windows due to McAfee's mfehcinj.dll apparently hijacking or suspending threads

[ybbus]

What version of Go are you using (go version)?

go version go1.15.4 windows/amd64

Does this issue reproduce with the latest release?

I can reproduce it very regularly at least on my system.

What operating system and processor architecture are you using (go env)?

go env Output

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Dev\Temp\go-build
set GOENV=C:\Users\Alex\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Dev\workdir\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Dev\workdir
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Dev\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=C:\Dev\Temp
set GOTOOLDIR=C:\Dev\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Dev\Temp\go-build995151114=/tmp/go-build -gno-record-gcc-switches

What did you do?

I experienced a lot of freezes for several days now, when using the command:
go generate ./..
in one of my projects.

go generate does trigger four different mockgen commands, thats all.
These mockgen commands in turn trigger some go list and go build commands, which freeze very often.

I don't think it has anything to do with go generate or mockgen. It just seems to trigger the issue more often as when I just do go build. But also go build as a standalone command sometimes triggered this issue.

What did you expect to see?

Successful execution of all go commands that were called from go generate --> mockgen.

What did you see instead?

The main issue here is, that go build and also go list that is executed by mockgen freezes very often.

In windows, by using a process monitor tool, I can see, that go generate called mockgenthat in turn called go build.
It happens very often, that the go build -o prog.bin.exe prog.goprocess is does not return and seems to be stuck.

I also found out, when setting GOMAXPROCS=1, go generate ./... works perfectly without a single one of this issues appearing.

Furthermore when go build was stuck I attached with dlv and got the following list of goroutines:

Goroutine 1 - User: c:/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x756e89)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
* Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 5 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 15 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 19 - User: c:/go/src/runtime/symtab.go:940 runtime.step (0x744a7e) (thread 30604)
  Goroutine 20 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 1267 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1268 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1269 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1270 - User: :0 ??? (0x7ffbb80dc144) (thread 21588)

The following goroutines are waiting at the provided code lines:

1    waits at: c:/go/src/cmd/go/internal/work/exec.go:188
1267 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1268 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1269 waits at: c:/go/src/cmd/go/internal/work/exec.go:169

I think the other goroutines are not that interesting, they look like this:

the other routines look something like this:
0  0x000000000072be3a in runtime.gopark
   at c:/go/src/runtime/proc.go:307
1  0x0000000000715929 in runtime.goparkunlock
   at c:/go/src/runtime/proc.go:312
2  0x0000000000715929 in runtime.bgsweep
   at c:/go/src/runtime/mgcsweep.go:182
3  0x000000000075b9a1 in runtime.goexit
   at c:/go/src/runtime/asm_amd64.s:1374

or this:
   0  0x000000000072be3a in runtime.gopark
   at c:/go/src/runtime/proc.go:307
1  0x000000000070e174 in runtime.gcBgMarkWorker
   at c:/go/src/runtime/mgc.go:1891
2  0x000000000075b9a1 in runtime.goexit
   at c:/go/src/runtime/asm_amd64.s:1374

As mentioned, this issue does not appear, when using GOMAXPROCS=1.
Otherwise it appears nearly always.

Any idea?

[ybbus]

I am not sure about this problem. It "freezes" on many different parts in go, e.g.

go list -e -json -compiled=true -test=true -export=false -deps=false -find=false -- C:\Dev\projects\XXX\pkg\config

With goroutines

Goroutine 1 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 5 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 6 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 28 - User: :0 ??? (0x7ffbb80dc4c4) (thread 5088)
  Goroutine 29 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 30 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 31 - User: c:/go/src/strings/builder.go:68 strings.Join (0x7fc3c7)
  Goroutine 32 - User: :0 ??? (0x7ffbb80dc7e4) (thread 28972)
  Goroutine 33 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 34 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 50 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 66 - User: :0 ??? (0x7ffbb80dc144) (thread 17908)
  Goroutine 67 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 98 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 99 - User: c:/go/src/strings/builder.go:68 strings.Join (0x7fc3c7)
  Goroutine 100 - User: :0 ??? (0x7ffbb80dc144) (thread 21604)
  Goroutine 101 - User: c:/go/src/os/stat_windows.go:64 os.stat (0x7e2a05)
  Goroutine 102 - User: c:/go/src/cmd/go/internal/imports/read.go:46 cmd/go/internal/imports.(*importReader).readByte (0xbbb20e)

or on command: go generate ./...
with goroutines

Goroutine 1 - User: c:/go/src/syscall/syscall_windows.go:46 syscall.UTF16FromString (0x7a736e)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 19 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 43 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 44 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 50 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
* Goroutine 51 - User: :0 ??? (0x7ffbb80dc144) (thread 16904)
  Goroutine 52 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 53 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 54 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 55 - User: c:/go/src/crypto/sha256/sha256.go:214 crypto/sha256.(*digest).Sum (0x92ac0e)
  Goroutine 56 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 57 - User: c:/go/src/sync/map.go:226 sync.(*Map).LoadOrStore (0x77511a)
  Goroutine 58 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 59 - User: :0 ??? (0x7ffbb80dc4c4) (thread 7224)
  Goroutine 60 - User: c:/go/src/bytes/buffer.go:229 bytes.makeSlice (0x80c6da)
  Goroutine 61 - User: c:/go/src/io/io.go:406 io.copyBuffer (0x7a045b)
  Goroutine 62 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 63 - User: c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62 cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1 (0xaf2c29)

I am quite confused with this behavior.

[bcmills]

@ybbus, unfortunately those goroutine traces are not particularly helpful: the goroutines waiting in src/cmd/go/internal/work/exec.go are waiting for something else to complete, but don't tell us what that something else is.

The later traces show a program that does not appear to be deadlocked.

---

  Goroutine 102 - User: c:/go/src/cmd/go/internal/imports/read.go:46 cmd/go/internal/imports.(*importReader).readByte (0xbbb20e)

is in a conditional after having successfully read a byte from a *bufio.Reader:

go/src/cmd/go/internal/imports/read.go

Line 46 in 1f2e58d

r.buf = append(r.buf, c)

---

  Goroutine 63 - User: c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62 cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1 (0xaf2c29)

is in a loop after having read a file to be hashed:

go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go

Line 62 in 1f2e58d

fmt.Fprintf(h, "%x %s\n", hf.Sum(nil), file)

---

So I'm not sure what to make of this: it looks like a running program, albeit possibly one that is running slowly. (Is it possible that you're seeing slowness due to, say, filesystem contention?)

[ybbus]

@bcmills
It is very hard for me to get some useful information out of this issue. :(

What I can tell you is, that the last two go-routines you mentioned are indeed "stuck".
I could leave the process for an hour and they would still be in the exact same position in code.
I checked this with "continue" command in delve and then break again after a while, with the exact same positions of the go routines.

I also cannot explain why this doesn't happen with GOMAXPROCS=1.

I would appreciate if someone has a hint for me how to better approach this problem.
I tried all kind of tools (windows sysinternals, dlv) but did not yet find any useful clue to figure out, why it gets stuck.

Could some anti-virus software be the problem here? blocking file-access / forbid memory allocation...?

[ybbus]

Looked a bit more into this.
Still the same: GOMAXPROCS=1 no problems, GOMAXPROCS=8, nearly every invoke freezes

What I also observed, that I cannot really nail it down to some code line. It seems to happen random on any line of code, and also on any part in the execution but mostly when go build is executing (maybe since this takes most of the runtime).

But what seems to be in common is, that the end of the stack on that goroutine lokks like this:

0  0x000000000072be3a in runtime.gopark
    at c:/go/src/runtime/proc.go:307
 1  0x00000000007106f1 in runtime.goparkunlock
    at c:/go/src/runtime/proc.go:312
 2  0x00000000007106f1 in runtime.gcParkAssist
    at c:/go/src/runtime/mgcmark.go:716
 3  0x00000000007100dd in runtime.gcAssistAlloc
    at c:/go/src/runtime/mgcmark.go:580
 4  0x00000000006fe59c in runtime.mallocgc
    at c:/go/src/runtime/malloc.go:952
##### up here is mostly the same every time
 5  0x00000000006fb7c5 in runtime.convTslice
    at c:/go/src/runtime/iface.go:385
 6  0x0000000000af2c29 in cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1
    at c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62

To be honest I am not very familiar with this low level stuff, but for me it seems like that a preemtion happens for the goroutine currently running, and it never ever wakes up later on.

Also not very deterministic:
Sometimes the stuck process has 0% CPU usage, and sometimes it uses at lot of CPU what looks like some busywaiting.
It don't think it is just slow. I've let this command run for over an hour without any progress and after that the goroutines stack still looked the same.
Using GOMAXPROCS=1, the command does run in under one minute.

The lower GOMAXPROCS is, the lower is the chance this issue happens, with 1 and 2 it did not happen yet.

[bcmills]

Could some anti-virus software be the problem here?

I think the answer to that is always “maybe”. 😅
(Do you have any anti-virus software installed?)

blocking file-access / forbid memory allocation...?

The go command does use file-locking, but in that case I would expect to see some goroutine blocked in the cmd/go/internal/lockedfile or cmd/go/internal/lockedfile/internal/filelock packages, and that doesn't seem to be the case here — none of the goroutine stacks you've posted appear to be in I/O at all.

[bcmills]

A couple more diagnostic questions:

• Do other Go programs exhibit similar hangs on your system, or just cmd/go?
• How did you install the Go toolchain? (From source, from an installer at https://golang.org/dl, something else..?)

[ybbus]

Bryan:
I havent experienced this yet on other go programs, but to be honest I do not run a lot of go programs locally, just developing and therefore doing a lot of build, test and generate.

When I run other programs locally there was no issue yet.

For installing I always download the windows archive and extract it to my machine.

Yes, MCAfee is installed, but I was allowed to completely disable it (corporate machine) for testing, and it did not change anything.

Is ther an easy way to check with dlv, what the program is really doing at the moment or why (for what) it is waiting?
It seems like it just stopped doing stuff without any good reason.

[davecheney]

try passing the -x flag to go build with both the GOMAXPROCS 1 and 8 scenarios. Please provide the entire output

[ybbus]

Hi Dave,
I will try that out.
In the meantime I also checked a bit more using Sysinternals Process explorer.

It seems that even though I disabled most of the McAfee stuff, it still tries to be part of the problem I think.
I did not even know that is is going to directly inject itself in other processes, but that is what you can see on the screenshot.

The stack of all threads of the hanging go processes look the same and seem to "WaitForSingleObject". The only stack that looks different is the one that also seems to have something injected from McAfee.

I know too little of the low level windows os stuff to have some opinion about this, but for me it actually looks like some issue with mcafee.

I just don't get, why there are tons of programs running on my machine that don't have a similar problems with McAffe. Seems to be something go specific? Also it is quite strange that GOMAXPROCS seems to make a huge difference here.

[ybbus]

The other thing that may happen is that there is one thread that is running all the time with high CPU, but still the go command does never return:

[ybbus]

Dave:
MAXPROCS=8 with go build -x does not get very far :)

C:\XXX\projects\XXX\XXX\dlt-service\pkg\config\gomock_reflect_037446375>go build -x -o prog.bin.exe prog.go
WORK=C:\XXX\Temp\go-build997381258

otherwise with MAXPROCS=1

MAXPROCS=1 Output

go build -x -o prog.bin.exe prog.go
WORK=C:\XXXXX\Temp\go-build585078346
mkdir -p $WORK\b001\
cat >$WORK\b001\_gomod_.go << 'EOF' # internal
package main
import _ "unsafe"
//go:linkname __debug_modinfo__ runtime.modinfo
var __debug_modinfo__ = "0w\xaf\f\x92t\b\x02A\xe1\xc1\a\xe6\xd6\x18\xe6path\tcommand-line-arguments\nmod\tzzzzz/yyyyy/dlt-service\t(devel)\t\ndep\tgithub.com/fsnotify/fsnotify\tv1.4.7\th1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=\ndep\tgithub.com/golang/mock\tv1.4.4\th1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=\ndep\tgithub.com/hashicorp/hcl\tv1.0.0\th1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=\ndep\tgithub.com/magiconair/properties\tv1.8.1\th1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=\ndep\tgithub.com/mitchellh/mapstructure\tv1.1.2\th1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=\ndep\tgithub.com/pelletier/go-toml\tv1.2.0\th1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=\ndep\tgithub.com/spf13/afero\tv1.1.2\th1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=\ndep\tgithub.com/spf13/cast\tv1.3.0\th1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=\ndep\tgithub.com/spf13/jwalterweatherman\tv1.0.0\th1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=\ndep\tgithub.com/spf13/pflag\tv1.0.3\th1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=\ndep\tgithub.com/spf13/viper\tv1.7.1\th1:pM5oEahlgWv/WnHXpgbKz7iLIxRf65tye2Ci+XFK5sk=\ndep\tgithub.com/subosito/gotenv\tv1.2.0\th1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=\ndep\tgolang.org/x/text\tv0.3.2\th1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=\ndep\tgopkg.in/ini.v1\tv1.51.0\th1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=\ndep\tgopkg.in/yaml.v2\tv2.2.4\th1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=\n\xf92C1\x86\x18 r\x00\x82B\x10A\x16\xd8\xf2"
        EOF
cat >$WORK\b001\importcfg << 'EOF' # internal
# import config
packagefile encoding/gob=C:\XXXXX\go\pkg\windows_amd64\encoding\gob.a
packagefile flag=C:\XXXXX\go\pkg\windows_amd64\flag.a
packagefile fmt=C:\XXXXX\go\pkg\windows_amd64\fmt.a
packagefile github.com/golang/mock/mockgen/model=C:\XXXXX\Temp\go-build\10\109e2e69560a3543b230cdf986befcb2e6290529c978a3b0eabb2f18094dab72-d
packagefile zzzzz/yyyyy/dlt-service/pkg/config=C:\XXXXX\Temp\go-build\58\5887bb7485e88fbcc15fb62384273e3026e1e63106530bb84f5c50f386972a1b-d
packagefile os=C:\XXXXX\go\pkg\windows_amd64\os.a
packagefile path=C:\XXXXX\go\pkg\windows_amd64\path.a
packagefile reflect=C:\XXXXX\go\pkg\windows_amd64\reflect.a
packagefile runtime=C:\XXXXX\go\pkg\windows_amd64\runtime.a
EOF
cd C:\XXXXX\projects\zzzzz\yyyyy\dlt-service\pkg\config\gomock_reflect_037446375
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\compile.exe" -o "$WORK\\b001\\_pkg_.a" -trimpath "$WORK\\b001=>" -p main -lang=go1.15 -complete -buildid JJ3Jf0mSmjshQOlN5BMT/JJ3Jf0mSmjshQOlN5BMT -goversion go1.15.4 -D _/C_/XXXXX/projects/zzzzz/yyyyy/dlt-service/pkg/config/gomock_reflect_037446375 -importcfg "$WORK\\b001\\importcfg" -pack -c=4 "C:\\XXXXX\\projects\\zzzzz\\yyyyy\\dlt-service\\pkg\\config\\gomock_reflect_037446375\\prog.go" "$WORK\\b001\\_gomod_.go"
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\buildid.exe" -w "$WORK\\b001\\_pkg_.a" # internal
cp "$WORK\\b001\\_pkg_.a" "C:\\XXXXX\\Temp\\go-build\\4f\\4fa3da2ae4df62905dff257ffe93933cf830e556c79d91891177273d2ff58241-d" # internal
cat >$WORK\b001\importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK\b001\_pkg_.a
packagefile encoding/gob=C:\XXXXX\go\pkg\windows_amd64\encoding\gob.a
packagefile flag=C:\XXXXX\go\pkg\windows_amd64\flag.a
packagefile fmt=C:\XXXXX\go\pkg\windows_amd64\fmt.a
packagefile github.com/golang/mock/mockgen/model=C:\XXXXX\Temp\go-build\10\109e2e69560a3543b230cdf986befcb2e6290529c978a3b0eabb2f18094dab72-d
packagefile zzzzz/yyyyy/dlt-service/pkg/config=C:\XXXXX\Temp\go-build\58\5887bb7485e88fbcc15fb62384273e3026e1e63106530bb84f5c50f386972a1b-d
packagefile os=C:\XXXXX\go\pkg\windows_amd64\os.a
packagefile path=C:\XXXXX\go\pkg\windows_amd64\path.a
packagefile reflect=C:\XXXXX\go\pkg\windows_amd64\reflect.a
packagefile runtime=C:\XXXXX\go\pkg\windows_amd64\runtime.a
packagefile bufio=C:\XXXXX\go\pkg\windows_amd64\bufio.a
packagefile encoding=C:\XXXXX\go\pkg\windows_amd64\encoding.a
packagefile encoding/binary=C:\XXXXX\go\pkg\windows_amd64\encoding\binary.a
packagefile errors=C:\XXXXX\go\pkg\windows_amd64\errors.a
packagefile io=C:\XXXXX\go\pkg\windows_amd64\io.a
packagefile math=C:\XXXXX\go\pkg\windows_amd64\math.a
packagefile math/bits=C:\XXXXX\go\pkg\windows_amd64\math\bits.a
packagefile sync=C:\XXXXX\go\pkg\windows_amd64\sync.a
packagefile sync/atomic=C:\XXXXX\go\pkg\windows_amd64\sync\atomic.a
packagefile unicode=C:\XXXXX\go\pkg\windows_amd64\unicode.a
packagefile unicode/utf8=C:\XXXXX\go\pkg\windows_amd64\unicode\utf8.a
packagefile sort=C:\XXXXX\go\pkg\windows_amd64\sort.a
packagefile strconv=C:\XXXXX\go\pkg\windows_amd64\strconv.a
packagefile strings=C:\XXXXX\go\pkg\windows_amd64\strings.a
packagefile time=C:\XXXXX\go\pkg\windows_amd64\time.a
packagefile internal/fmtsort=C:\XXXXX\go\pkg\windows_amd64\internal\fmtsort.a
packagefile github.com/golang/mock/gomock=C:\XXXXX\Temp\go-build\60\601f44e01f55f73609660ecdc59c862e261d2b809533709aae14e00ea27598b3-d
packagefile github.com/spf13/viper=C:\XXXXX\Temp\go-build\a7\a7b8d2703b14f15e33bab09c30715eda783308d3a003e3056c81f6a79229feac-d
packagefile internal/oserror=C:\XXXXX\go\pkg\windows_amd64\internal\oserror.a
packagefile internal/poll=C:\XXXXX\go\pkg\windows_amd64\internal\poll.a
packagefile internal/syscall/execenv=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\execenv.a
packagefile internal/syscall/windows=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows.a
packagefile internal/testlog=C:\XXXXX\go\pkg\windows_amd64\internal\testlog.a
packagefile syscall=C:\XXXXX\go\pkg\windows_amd64\syscall.a
packagefile unicode/utf16=C:\XXXXX\go\pkg\windows_amd64\unicode\utf16.a
packagefile internal/unsafeheader=C:\XXXXX\go\pkg\windows_amd64\internal\unsafeheader.a
packagefile internal/bytealg=C:\XXXXX\go\pkg\windows_amd64\internal\bytealg.a
packagefile internal/cpu=C:\XXXXX\go\pkg\windows_amd64\internal\cpu.a
packagefile runtime/internal/atomic=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\atomic.a
packagefile runtime/internal/math=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\math.a
packagefile runtime/internal/sys=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\sys.a
packagefile bytes=C:\XXXXX\go\pkg\windows_amd64\bytes.a
packagefile internal/reflectlite=C:\XXXXX\go\pkg\windows_amd64\internal\reflectlite.a
packagefile internal/race=C:\XXXXX\go\pkg\windows_amd64\internal\race.a
packagefile internal/syscall/windows/registry=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows\registry.a
packagefile context=C:\XXXXX\go\pkg\windows_amd64\context.a
packagefile encoding/csv=C:\XXXXX\go\pkg\windows_amd64\encoding\csv.a
packagefile encoding/json=C:\XXXXX\go\pkg\windows_amd64\encoding\json.a
packagefile github.com/fsnotify/fsnotify=C:\XXXXX\Temp\go-build\1e\1ed0f1c46c880e2c6cc8148dfadead67b4eefa80e7d6cf6afa0004941457ea80-d
packagefile github.com/hashicorp/hcl=C:\XXXXX\Temp\go-build\32\32b7a1ffbacd01fcaa242286bd2bb7cccbc72e3149bd109b95fe93080a2a5ab6-d
packagefile github.com/hashicorp/hcl/hcl/printer=C:\XXXXX\Temp\go-build\01\016804adc7b57984933ab80850831574c5d4a3fae5ccc856216d17941a0487dd-d
packagefile github.com/magiconair/properties=C:\XXXXX\Temp\go-build\23\239a0de4f71fd1234ed89705587cf6cc7bb55785f60d91401ab243cc62d5a92b-d
packagefile github.com/mitchellh/mapstructure=C:\XXXXX\Temp\go-build\ab\ab3935b0e07b7438fb20e119b33efbe700696c812df23b86c536586cb7987071-d
packagefile github.com/pelletier/go-toml=C:\XXXXX\Temp\go-build\5d\5dba111cfc102063714dede76bafc32c66264998beb25d6c7129ebf2cb170a41-d
packagefile github.com/spf13/afero=C:\XXXXX\Temp\go-build\93\931e58331c1b802787f5ff389434883280786eda5634890e6d793dae50229626-d
packagefile github.com/spf13/cast=C:\XXXXX\Temp\go-build\11\11b966d35db068c18238ff0f4e4784c9c865ba958cc28ef3e5297c535ba1b91a-d
packagefile github.com/spf13/jwalterweatherman=C:\XXXXX\Temp\go-build\7f\7fb7dd87189e3144cb19036b43bdac0ebf73d9de0fd355d48d2eaf0ac6b4923d-d
packagefile github.com/spf13/pflag=C:\XXXXX\Temp\go-build\4b\4b57e69250b14274268a29feb79b19458e1c6438f29d8713c479fdddb93f9c0b-d
packagefile github.com/subosito/gotenv=C:\XXXXX\Temp\go-build\ca\ca08d89fb013a51f590f0e5cce4e83b9b863444f9ba7399b1865893e8ecfe20f-d
packagefile gopkg.in/ini.v1=C:\XXXXX\Temp\go-build\0c\0cfc989b448b54874061740e25f59372de99a858b2661ed0c22822e1f2fc7117-d
packagefile gopkg.in/yaml.v2=C:\XXXXX\Temp\go-build\63\63770eb6a562fbcbedc0853b62fe1e376fb059ae340fe38254689a3b646dfeb8-d
packagefile log=C:\XXXXX\go\pkg\windows_amd64\log.a
packagefile path/filepath=C:\XXXXX\go\pkg\windows_amd64\path\filepath.a
packagefile internal/syscall/windows/sysdll=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows\sysdll.a
packagefile encoding/base64=C:\XXXXX\go\pkg\windows_amd64\encoding\base64.a
packagefile github.com/hashicorp/hcl/hcl/ast=C:\XXXXX\Temp\go-build\9b\9bf532129a40b204feb1fb47fdb7f4e066a4eb7fd51b1cfe00b6a598202a9cd1-d
packagefile github.com/hashicorp/hcl/hcl/parser=C:\XXXXX\Temp\go-build\19\196933d4932413c85c778a58ce4b5e8f86517cadec2239b1a6a4263d157224cc-d
packagefile github.com/hashicorp/hcl/hcl/token=C:\XXXXX\Temp\go-build\1e\1ec247c5ba207a59d6b55e2ff205c846939f18168ca5d283c5e39087a2638ef8-d
packagefile github.com/hashicorp/hcl/json/parser=C:\XXXXX\Temp\go-build\72\7297703cbb80ca4ca59f806edcdaa73bd3a27f4723cc483ac5f8a0f85ab60aa9-d
packagefile text/tabwriter=C:\XXXXX\go\pkg\windows_amd64\text\tabwriter.a
packagefile io/ioutil=C:\XXXXX\go\pkg\windows_amd64\io\ioutil.a
packagefile net/http=C:\XXXXX\go\pkg\windows_amd64\net\http.a
packagefile regexp=C:\XXXXX\go\pkg\windows_amd64\regexp.a
packagefile net=C:\XXXXX\go\pkg\windows_amd64\net.a
packagefile github.com/spf13/afero/mem=C:\XXXXX\Temp\go-build\a1\a11301268a1e43bd44623a80d8249cdd50058bf05206b0a64c2e3385add04d92-d
packagefile golang.org/x/text/transform=C:\XXXXX\Temp\go-build\ab\ab0f407746c6f50ddb771ed1df9249f300d7be5afcada3a81d3a79297ed9bef7-d
packagefile golang.org/x/text/unicode/norm=C:\XXXXX\Temp\go-build\8e\8e5b7fe9c16570f265818d417ba9a107b37842c4d5e44f0fb2ab76cef19ff992-d
packagefile html/template=C:\XXXXX\go\pkg\windows_amd64\html\template.a
packagefile encoding/hex=C:\XXXXX\go\pkg\windows_amd64\encoding\hex.a
packagefile github.com/hashicorp/hcl/hcl/scanner=C:\XXXXX\Temp\go-build\36\361f78b7d31e4fdea0c92df8f5f7eb4517a8eab090fef6d9c31a5e4d058fdb75-d
packagefile github.com/hashicorp/hcl/hcl/strconv=C:\XXXXX\Temp\go-build\ea\eae329e2ca698227192663f9a745192dbbced4855404c946c1e43a5e87da62fb-d
packagefile github.com/hashicorp/hcl/json/scanner=C:\XXXXX\Temp\go-build\a7\a7464a1a8b7c1b4547b3a71818191b63f77d136a32c1542b80d8f593e8186945-d
packagefile github.com/hashicorp/hcl/json/token=C:\XXXXX\Temp\go-build\44\44180e6fb1d7f7fccbffe65b7a5e64c5c7c92e65e876a6bc38a044e3911146a0-d
packagefile compress/gzip=C:\XXXXX\go\pkg\windows_amd64\compress\gzip.a
packagefile container/list=C:\XXXXX\go\pkg\windows_amd64\container\list.a
packagefile crypto/rand=C:\XXXXX\go\pkg\windows_amd64\crypto\rand.a
packagefile crypto/tls=C:\XXXXX\go\pkg\windows_amd64\crypto\tls.a
packagefile vendor/golang.org/x/net/http/httpguts=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http\httpguts.a
packagefile vendor/golang.org/x/net/http/httpproxy=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http\httpproxy.a
packagefile vendor/golang.org/x/net/http2/hpack=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http2\hpack.a
packagefile vendor/golang.org/x/net/idna=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\idna.a
packagefile math/rand=C:\XXXXX\go\pkg\windows_amd64\math\rand.a
packagefile mime=C:\XXXXX\go\pkg\windows_amd64\mime.a
packagefile mime/multipart=C:\XXXXX\go\pkg\windows_amd64\mime\multipart.a
packagefile net/http/httptrace=C:\XXXXX\go\pkg\windows_amd64\net\http\httptrace.a
packagefile net/http/internal=C:\XXXXX\go\pkg\windows_amd64\net\http\internal.a
packagefile net/textproto=C:\XXXXX\go\pkg\windows_amd64\net\textproto.a
packagefile net/url=C:\XXXXX\go\pkg\windows_amd64\net\url.a
packagefile regexp/syntax=C:\XXXXX\go\pkg\windows_amd64\regexp\syntax.a
packagefile vendor/golang.org/x/net/dns/dnsmessage=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\dns\dnsmessage.a
packagefile internal/nettrace=C:\XXXXX\go\pkg\windows_amd64\internal\nettrace.a
packagefile internal/singleflight=C:\XXXXX\go\pkg\windows_amd64\internal\singleflight.a
packagefile html=C:\XXXXX\go\pkg\windows_amd64\html.a
packagefile text/template=C:\XXXXX\go\pkg\windows_amd64\text\template.a
packagefile text/template/parse=C:\XXXXX\go\pkg\windows_amd64\text\template\parse.a
packagefile compress/flate=C:\XXXXX\go\pkg\windows_amd64\compress\flate.a
packagefile hash/crc32=C:\XXXXX\go\pkg\windows_amd64\hash\crc32.a
packagefile math/big=C:\XXXXX\go\pkg\windows_amd64\math\big.a
packagefile crypto=C:\XXXXX\go\pkg\windows_amd64\crypto.a
packagefile crypto/aes=C:\XXXXX\go\pkg\windows_amd64\crypto\aes.a
packagefile crypto/cipher=C:\XXXXX\go\pkg\windows_amd64\crypto\cipher.a
packagefile crypto/des=C:\XXXXX\go\pkg\windows_amd64\crypto\des.a
packagefile crypto/ecdsa=C:\XXXXX\go\pkg\windows_amd64\crypto\ecdsa.a
packagefile crypto/ed25519=C:\XXXXX\go\pkg\windows_amd64\crypto\ed25519.a
packagefile crypto/elliptic=C:\XXXXX\go\pkg\windows_amd64\crypto\elliptic.a
packagefile crypto/hmac=C:\XXXXX\go\pkg\windows_amd64\crypto\hmac.a
packagefile crypto/md5=C:\XXXXX\go\pkg\windows_amd64\crypto\md5.a
packagefile crypto/rc4=C:\XXXXX\go\pkg\windows_amd64\crypto\rc4.a
packagefile crypto/rsa=C:\XXXXX\go\pkg\windows_amd64\crypto\rsa.a
packagefile crypto/sha1=C:\XXXXX\go\pkg\windows_amd64\crypto\sha1.a
packagefile crypto/sha256=C:\XXXXX\go\pkg\windows_amd64\crypto\sha256.a
packagefile crypto/sha512=C:\XXXXX\go\pkg\windows_amd64\crypto\sha512.a
packagefile crypto/subtle=C:\XXXXX\go\pkg\windows_amd64\crypto\subtle.a
packagefile crypto/x509=C:\XXXXX\go\pkg\windows_amd64\crypto\x509.a
packagefile encoding/pem=C:\XXXXX\go\pkg\windows_amd64\encoding\pem.a
packagefile vendor/golang.org/x/crypto/chacha20poly1305=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\chacha20poly1305.a
packagefile vendor/golang.org/x/crypto/cryptobyte=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\cryptobyte.a
packagefile vendor/golang.org/x/crypto/curve25519=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\curve25519.a
packagefile vendor/golang.org/x/crypto/hkdf=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\hkdf.a
packagefile hash=C:\XXXXX\go\pkg\windows_amd64\hash.a
packagefile vendor/golang.org/x/text/secure/bidirule=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\secure\bidirule.a
packagefile vendor/golang.org/x/text/unicode/bidi=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\unicode\bidi.a
packagefile vendor/golang.org/x/text/unicode/norm=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\unicode\norm.a
packagefile mime/quotedprintable=C:\XXXXX\go\pkg\windows_amd64\mime\quotedprintable.a
packagefile crypto/internal/subtle=C:\XXXXX\go\pkg\windows_amd64\crypto\internal\subtle.a
packagefile crypto/internal/randutil=C:\XXXXX\go\pkg\windows_amd64\crypto\internal\randutil.a
packagefile vendor/golang.org/x/crypto/cryptobyte/asn1=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\cryptobyte\asn1.a
packagefile crypto/ed25519/internal/edwards25519=C:\XXXXX\go\pkg\windows_amd64\crypto\ed25519\internal\edwards25519.a
packagefile crypto/dsa=C:\XXXXX\go\pkg\windows_amd64\crypto\dsa.a
packagefile crypto/x509/pkix=C:\XXXXX\go\pkg\windows_amd64\crypto\x509\pkix.a
packagefile encoding/asn1=C:\XXXXX\go\pkg\windows_amd64\encoding\asn1.a
packagefile vendor/golang.org/x/crypto/chacha20=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\chacha20.a
packagefile vendor/golang.org/x/crypto/internal/subtle=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\internal\subtle.a
packagefile vendor/golang.org/x/crypto/poly1305=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\poly1305.a
packagefile vendor/golang.org/x/sys/cpu=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\sys\cpu.a
packagefile vendor/golang.org/x/text/transform=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\transform.a
EOF
mkdir -p $WORK\b001\exe\
cd .
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\link.exe" -o "$WORK\\b001\\exe\\a.out.exe" -importcfg "$WORK\\b001\\importcfg.link" -buildmode=pie -buildid=cLYmR2vpJ0BXtT-SxR4k/JJ3Jf0mSmjshQOlN5BMT/7qbI6Ur-PSPn5ckbZQJl/cLYmR2vpJ0BXtT-SxR4k -extld=gcc "$WORK\\b001\\_pkg_.a"
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\buildid.exe" -w "$WORK\\b001\\exe\\a.out.exe" # internal
cp $WORK\b001\exe\a.out.exe prog.bin.exe
rm -r $WORK\b001\

[ybbus]

Just another thought:
It seems that mcafee acquires one or two threads in the running go process.
Maybe the following could be an explanation:

MAXPROCS=1
mcafee does not even have a chance to start a thread and do strange stuff --> everything works

MAXPROCS=8
enough threads (time) to start up the mcafee hook. since mcafee wants to check the binary before execution, it blocks evrything else but needs some other threads, that it does not get because of the MAXPROCS=8, so it waits forever and also blocks all others forever
go runtime also waits for a thread, since it does not really know that there is some "stranger" also requesting threads and so go also blocks because go runtime is not aware of threads beeing stolen.

That is my guess and would fit to my following observation:

The build again runs smoothly, when setting GOMAXPROCS to a high value like 256.
At least 90% of the time, unfortunately it also gets stuck time by time

Maybe this is enough on the one hand for go itself and on the other hand also for mcafee to do whatever it does...

64 and 128 for example still does not seem to be enough --> again always hanging

256 only works when directly invoking go build.
When trying this with go generate ./.. it again hangs after sometime. At least later in time.
When setting to 1024 , then also the go generate runs without problems (most of the time)

Any thoughts on that?
I can just guess here since I have no real clue.

[davecheney]

Seems this is a bug in mcafee av

[bcmills]

It is possible that the McAfee DLL is hijacking the threads when the Go runtime makes a system call under the assumption that every thread “probably” makes a system call eventually.

Some Go threads essentially never make system calls, so they would not be hijackable. That may be what leads to the observed deadlock.

If that is correct, then a workaround on the Go side might look like the workaround for #18717 (CL 35494), in which threads controlled by the Go runtime periodically make an inexpensive but otherwise needless call to an arbitrary function in the C standard library in order to allow hooks embedded in that library to run on the thread.
(CC @aclements @ianlancetaylor @alexbrainman)

That said, even if this hypothesis is correct I would rather have McAfee fix their faulty assumption than us work around it.

[bcmills]

(And by the way, @ybbus, thanks for the detailed investigation of what's running on those threads!)

[bcmills]

Ah, one other experiment you could try (via #36494). @ybbus, what happens if you set GODEBUG=asyncpreemptoff=1 in the environment, still with GOMAXPROCS=8? (Does that change the frequency of the hangs?)

[networkimprov]

I'd imagine you can configure McAfee to ignore your program.

This problem is not unique to McAfee; we've heard reports where the Go toolchain runs fine, but the binaries it builds fail due to AV interference.

[alexbrainman]

If that is correct, then a workaround on the Go side might look like the workaround for #18717 (CL 35494)

Sorry I don't know enough to comment.

what happens if you set GODEBUG=asyncpreemptoff=1 in the environment,

I agree. This might help. I have similar unresolved issue #36492. @aclements thinks it is my buggy antivirus program. Mind you, it is different antivirus vendor.

Alex

[ybbus]

@networkimprov
I'll try to discuss that, but this seems not to be very easy in a big company. Also go does generate a lot random name binaries that you do not know upfront, and if the problem also exists there (not sure yet), you would need to exclude a lot bunch of folders from the scan. Nothing the security guys like to see, but I will try to address this.

@bcmills
Indeed, setting asyncpreemtoff solves the problem. I was not able to reproduce the issue with MAXPROCS=8 over several runs.

What are the implications of using this flag in my daily development? Is it ok to let this enabled all the time?
Or let's phrase it like this:
What is the lesser evil: GOMAXPROCS=1 or asyncpreemptoff

Never the less I will ask the IT guys to forward this to McAfee, since the should really care about behaving nicely to foreign processes.

[networkimprov]

You're probably better off setting asyncpreemptoff, which switches the runtime to cooperative scheduling of goroutines on available cores (GOMAXPROCS) -- it was the default until 1.14.

Since that fixes it, maybe McAfee (or your IT dept) has an update planned.

[ybbus]

IT dept allowed to uninstall McAfee temporary --> issue disappeard

Unfortunatly after reinstalling McAfee, issue is still gone. I will leave it with that, and if the issue again comes up ask the IT guys to talk to McAfee about that.

Thanks for all your help. I now have at least some good short term workaround options to continue working when the issue comes up again.

From my side, this issue may be closed and hopefully helps other devs to faster get to the root cause of this problem (took me quite a while to be honest)

[ianlancetaylor]

Thanks for following up. Closing because it doesn't seem that there is anything we can change in Go to fix this.

[alexbrainman]

Closing because it doesn't seem that there is anything we can change in Go to fix this.

I disagree.

I think Go should have asyncpreemptoff=1 by default on Windows.

None of our Windows users complained about preemption being a problem, but now @ybbus and I are suffering.

So far, we only know about developers are affected. But maybe this also affects users who run our programs.

And, while I agree that the problem is most likely with buggy antivirus software, that is the world we have to live in.

Alex

[networkimprov]

That's a good point. I wondered why async preemption was rolled out as the default, instead of an option.

To make matters worse, we can't disable it in a build, and my proposal to allow that was quickly rejected, #40870.

[ianlancetaylor]

Lets take async preemption on Windows to a different issue, not this one. Thanks.