-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: "fatal error: out of memory" on windows-arm64-11 #51019
Comments
So far I'm not seeing any recurrences on what I assume is a much higher number of runs. We can keep an eye on it but right now I'm inclined to leave it alone. |
Here's one running
|
From the sheer number of packages that failed in each of those logs, I suspect that the parallelism is being set to high. What's the CPU-to-RAM ratio for this builder? (Maybe we could scale down |
About 12G RAM for 8 cores, which seems pretty plausible to me? There isn't much precedent for tweaking ...are the crypto tests really that memory hungry though? Smells weird. |
Change https://go.dev/cl/385182 mentions this issue: |
I agree. It looks like the actual OOM happened while recompiling packages in |
The Windows 11 ARM64 builder is experiencing occasional OOMs while building tests. Reducing GOMAXPROCS will reduce the go command's parallelism and hopefully prevent them. For golang/go#51019. Change-Id: Ia4bfdddaca178c130b9b57087a66a54cff903a05 Reviewed-on: https://go-review.googlesource.com/c/build/+/385182 Trust: Heschi Kreinick <[email protected]> Run-TryBot: Heschi Kreinick <[email protected]> Auto-Submit: Heschi Kreinick <[email protected]> Reviewed-by: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
Unfortunately still OOMing even with GOMAXPROCS=4.
2022-02-15T14:54:27-76bd8ea/windows-arm64-11 It's not at all clear to me why this is happening for the |
Oh, hrm. The failure condition in that last one is a bit different — it OOMed during bootstrapping. 🤔 |
Three more OOMs over the weekend: one during bootstrapping in the main repo, and two during
2022-02-20T20:58:11-851ecea/windows-arm64-11 |
Still ongoing:
2022-03-15T13:54:34-6799a7a-e475cf2/windows-arm64-11 |
We got bit twice during the release during bootstrap too. But I have no idea what to do about it. |
Still happening quite frequently, but only on the
2022-04-11T15:41:56-32de2b0/windows-arm64-11 |
Yep, same qemu script. My best guess is some kind of OS issue/conflict with the emulator, but I have no idea how to prove or disprove that belief. |
I wonder if this is somehow related to #49564, in that they both involve unexpected OOM failures on Windows. |
@golang/release, is there a way to get the runtime to dump the current heap size when it fails with |
The above failure mode suggests that there is a problem with the builder itself, not (just) #52433, since that failure occurred during bootstrapping using the old and venerable |
|
Change https://go.dev/cl/408702 mentions this issue: |
For golang/go#52653. Updates golang/go#51019. Change-Id: Ie57f7b2c2b6d4c3cc4b5f5f886773dff2a36a61e Reviewed-on: https://go-review.googlesource.com/c/build/+/408702 TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Alex Rakoczy <[email protected]>
@qmuntal, this is one of the issues I think should block promoting windows/arm64 to a first class port. |
(You can find other However, it's not clear to me which of those should be blockers to making |
Tracking the new Microsoft-provided |
It seems the "fatal error: out of memory" problem hasn't been reported here for a while. One possible explanation is that we've updated the host OS version where the qemu emulator is running (one of the I'll close this issue optimistically; please feel free to reopen otherwise. |
Change https://go.dev/cl/418940 mentions this issue: |
I'm no longer sure this is a builder issue. I'm facing some OOM errors like this one |
Found new dashboard test flakes for:
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/evanw/esbuild/internal/js_parser [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/yuin/goldmark/util [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/texthtml [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/chromedp/cdproto/domstorage [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/chromedp/cdproto/network [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/chromedp/cdproto/domdebugger [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 github.com/chromedp/cdproto/css [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/api.test [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/backport/go/parser [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/cmd/versionprune.test [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/memcache [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/redirect.test [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/short.test [build] (log)
2022-12-05 14:51 windows-arm64-11 website@69512276 go@e727f419 x/website/internal/webtest.test [build] (log)
|
Found new dashboard test flakes for:
2022-12-02 14:23 windows-arm64-11 go@dadd80ae (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/authhandler.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f github.com/google/go-cmp/cmp/internal/function [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/google/downscope.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/google.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/google/internal/externalaccount [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/jira [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/jws.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/internal.test [build] (log)
2022-12-06 15:30 windows-arm64-11 oauth2@b177c21a go@1eb0465f x/oauth2/jwt [build] (log)
|
Found new dashboard test flakes for:
2022-12-06 05:12 windows-arm64-11 go@1eb0465f cmd/compile/internal/ssa [build] (log)
2022-12-07 14:20 windows-arm64-11 go@a4a86c7b bootstrap/cmd/compile/internal/ssa [build] (log)
2022-12-07 14:20 windows-arm64-11 go@a4a86c7b bootstrap/cmd/link/internal/ld [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/go/packages.TestAddImportInOverlay (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/go/packages.TestIssue56632 (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/go/packages.TestLoadDifferentPatterns (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/go/packages.TestAbsoluteFilenames (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/go/ssa.TestObjValueLookup (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/internal/typesinternal.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/internal/typeparams.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/refactor/eg_test.TestStdlib [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/present [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/analysis/fillstruct.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/analysis/simplifycompositelit.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/analysis/simplifyrange.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/lsppos.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/fake.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/source.test [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/source [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/source/completion [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/cache [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/tests.TestLSP [build] (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/analysis/fillreturns.Test (log)
2022-12-07 15:31 windows-arm64-11 tools@09fb680d go@dc04f3ba x/tools/gopls/internal/lsp/command.TestGenerated (log)
2022-12-12 13:51 windows-arm64-11 website@e35843a5 go@888047c3 github.com/chromedp/cdproto/runtime [build] (log)
2022-12-12 13:51 windows-arm64-11 website@e35843a5 go@888047c3 x/website/internal/web [build] (log)
2022-12-12 13:51 windows-arm64-11 website@e35843a5 go@888047c3 x/website/internal/tmplfunc.test [build] (log)
|
I can't reproduce this issue anymore, and there hasn't been any memory-related issues since the windows/arm64 builders were deployed on Azure instead of using qemu on Mac. I propose to close this now. |
Thanks. If there's new evidence of this occurring, we can reopen or file a new issue. |
greplogs --dashboard -md -l -e '(?ms)\Awindows-arm64.*^fatal error: out of memory' --since=2021-01-01
2022-02-04T14:02:15-25d2ab2-4afcc9f/windows-arm64-11
We may need to reconfigure the builder to either turn down the build/test parallelism or have more RAM available.
There is only one of these failures in the logs, but OTOH this builder has only ever run
x/tools
12 times — so that's an 8% failure rate for this repo so far. 😅(attn @golang/release)
The text was updated successfully, but these errors were encountered: