-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: CreateThread fails with ERROR_ACCESS_DENIED #18253
Comments
Error code 5 is |
This seems impossible, but isn't isolated:
2016-10-17T18:53:22-0ba3c60/windows-386-gce:
2016-11-11T21:00:11-10d2efd/windows-amd64-gce:
2016-11-23T00:44:01-50c4dbc/windows-amd64-gce:
2016-12-07T16:59:25-1ea60c1/windows-386-gce:
So, it certainly seems to be unrelated to the particular test. It's interesting that it's always "have 5 already; errno=5". |
I do not have any bright ideas. CreateThread returns ERROR_ACCESS_DENIED makes me think that some pointers we pass to CreateThread might be invalid. Which leaves
I don't see why this is interesting - 5 is, probable, what Go program that does not starts goroutines have (at this moment). Alex |
MSDN says "Note that CreateThread may succeed even if lpStartAddress points to data, code, or is not accessible." which suggests that it's not a problem with the start PC, though it does only say "may". I'm surprised it would care about the lpParameter. I figured it would just pass that value through to the start function. Does it really care that that's a valid pointer to RW memory?
Possibly. It looks like GOMAXPROCS is 4 on these builders, so 5 is not an unreasonable number of threads. But something had to start them, and if the test only has a single goroutine, the only thing I can think that would encourage more threads to start would be the garbage collector. If that's the case, this means these tests have all gotten at least as far as the first GC cycle (versus crashing really early in execution). |
Looking at https://golang.org/test/fixedbugs/bug093.go you can see that it is very unlikely that any garbage collection occurred. |
You are right, I am wrong. lpParameter is just a uintptr - Windows does not care what is in it. I also tried calling CreateThread with lpStartAddress set to 0 and 123, and CreateThread succeeds regardless. |
This is what I see on my computer:
Alex |
Interesting. Thanks. Too bad, though; that would have been a plausible explanation. :)
Huh. That's a lot more threads than I expected. But at least it explains why mcount is 5. |
@aclements, how far back do your logs go? Is this failure new to Go 1.8? |
My logs go back to the beginning of 2015. I'm not sure how to best isolate this, but here's what I can find. I've crossed out the ones that seem less likely to be the same issue.
If this is right, it's been happening since before Go 1.8. |
@jstarks, any idea why _CreateThread might sometimes return ERROR_ACCESS_DENIED? |
Looking at Wine suggests that If that theory is correct, then one fix would to change It might be interesting if somebody with a Windows system could see how often this problem occurs, by simply running bug093 in a loop, so we can see whether that change actually fixes the problem. |
I built fixedbugs/bug093.go on Windows 10 (with CGO_ENABLED=0, though, if that matters) and ran the binary in a loop via another binary using os/exec.Command("./bug093.exe").CombinedOutput() in a loop. It's now been thousands of iterations, and no failure. But maybe Windows 10 doesn't have the problem. Our builders are much older. (Windows 2008 Server) Edit: it ran for about 100,000 iterations without failure overnight, before the computer went to sleep on its own. Still can't reproduce. Maybe I'll find time to try on a Windows 2008 Server VM next. |
I am travelling until after New Year. I only have my Chromebook with me. Alex |
CL https://golang.org/cl/34616 mentions this issue. |
On Windows, CreateThread occasionally fails with ERROR_ACCESS_DENIED. We're not sure why this is, but the Wine source code suggests that this can happen when there's a concurrent CreateThread and ExitProcess in the same process. Fix this by setting a flag right before calling ExitProcess and halting if CreateThread fails and this flag is set. Updates #18253 (might fix it, but we're not sure this is the issue and can't reproduce it on demand). Change-Id: I1945b989e73a16cf28a35bf2613ffab07577ed4e Reviewed-on: https://go-review.googlesource.com/34616 TryBot-Result: Gobot Gobot <[email protected]> Run-TryBot: Austin Clements <[email protected]> Reviewed-by: Alex Brainman <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>
I'm going to close this until we see it happen again. |
Reopening. Apparently Bazel is running into this occasionally. I'll let them add more details here. /cc @johnsonj |
I can confirm that ERROR_ACCESS_DENIED will be returned if you call CreateThread while the process is terminating. I don't immediately see another reason this would be returned. |
TensorFlow sometimes need buildozer from this repository on windows: cc @yifeif |
@gunan, can you provide any more details or repro steps? |
We will work on a smaller opensource example. |
@gunan, just to confirm: you're encountering this with a binary built with Go 1.8? |
@aclements yes, we built buildifier and buildozer with go 1.8 |
Happy to take a look at a repro |
Sorry for the delay, I was not able to get to creating a minimal example. |
@gunan, any progress on a minimal example? |
So far I have not been able to isolate this in buildozer. |
Recent failures:
Interestingly, several of these are cut off part way through the message, which sure smells like we're still having issues with calling |
Looks like we are calling Windows ExitProcess in syscall too:
That can easily account for this failure. I suppose we should just replace syscall.Exit with call to runtime.exit. No? Alex |
CL https://golang.org/cl/45115 mentions this issue. |
@alexbrainman, good call! In fact, it looks like the compiler always explicitly os.Exit's, even on success. Other build tools may, too. That could explain why we almost always see this in the test directory (it does a huge number of builds) and why it's scattered across different tests (it doesn't matter which test you're building, just the timing of when the compiler exits). |
Windows flake:
windows-386-gce at 1ea60c1
https://build.golang.org/log/f1487f7fe66b893aa63406d0c181dbdc1c72b3ae
Any ideas? @ianlancetaylor @aclements @alexbrainman
The text was updated successfully, but these errors were encountered: