x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

bcmills · 2020-05-21T15:45:19Z

#!watchflakes
post <- builder == "windows-amd64-race" && pkg ~ `^golang\.org/x/tools` && log ~ `fatal error: out of memory`

x/tools often runs out of memory when testing golang.org/x/tools/go/packages, and occasionally when running other tests too.

We should either upgrade the builder (see also #33986, #33951) or reduce the memory footprint of the test (see also #36943, #14113, #32834).

CC @dmitshur @toothrot @cagedmantis @ianthehat

The text was updated successfully, but these errors were encountered:

dmitshur · 2020-05-21T19:24:16Z

The windows-amd64-race builder is already using n1-highcpu-16 machine type (16 vCPUs, 14.4 GB mem), same as the windows-amd64-longtest builder. So, there isn't an easy opportunity to increase the RAM of the -race builder to match that of -longtest builder.

One difference between them is that the -race builder is using Windows server 2008, while -longtest is using Windows server 2016. Would it be a good first step to update the -race builder to a newer version of Windows? Or is it considered a benefit that it's testing with an older version of Windows?

bcmills · 2020-05-21T21:02:27Z

I think it makes sense to upgrade the -race builder to the latest OS image, since it's really checking for bugs on the Go side anyway.

(I doubt that will fix this issue, but it's a fine first step to eliminate a variable.)

gopherbot · 2020-05-27T16:05:34Z

Change https://golang.org/cl/235419 mentions this issue: dashboard: update windows-amd64-race builder to Windows Server 2016 image

…mage The purpose of the -race builder is to check for bugs on the Go side, so use the newer available Windows version with the expectation that there should be fewer unfixed bugs left in it. Take this easy step to eliminate a variable for issue golang/go#39196. Also delete host-windows-amd64-2008-big, since it's no longer used. For golang/go#39196. Change-Id: I9095a73ab21a151e5972219bccd5a16b249765b4 Reviewed-on: https://go-review.googlesource.com/c/build/+/235419 Reviewed-by: Carlos Amedee <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>

JohnCClarke · 2020-06-11T15:05:12Z

My team sees a similar error very frequently when running lots of tests sequentially from Makefiles. Curiously adding a few seconds delay between tests appears to improve the stability (although, without hard statistics we might be fooling ourselves). Simply rerunning the tests is often enough for them to succeed.

This is using go version go1.14.2 windows/amd64 on Microsoft Windows [Version 10.0.18363.836], and my impression is that the situation has become worse over time. That could be either or both of newer releases of golang or Windows. I wonder if a Windows expert knows more details of how Windows allocates memory and/or address space, and whether there is a better way for golang, or golang users, to interact with Windows.

This issue is a real nuisance for me and my team. We are very willing to help investigate and/or try potential fixes if anyone needs. Thanks!

Update at 2020-06-22: We have discovered that golang executables running out of memory is specific to executing our long "make" runs under powershell. Executing the same "make" runs under cmd.exe works correctly.

IWBN to understand the root cause though.

dmitshur · 2020-07-06T22:31:32Z

I've looked at the last 2 weeks of results (this is after the builder change in CL 235419), and I'm not spotting OOM-related failures. There are frequent failures on windows-amd64-race builder (and other race builders), but many of all seem to be due to #39865.

bcmills · 2022-01-11T15:37:11Z

Only one OOM over the past year, but it's recent enough that I don't see any intervening changes that would have fixed it. 🤔

greplogs --dashboard -md -l -e '(?ms)\Awindows-amd64-race.*fatal error: out of memory.*^FAIL\s+golang\.org/x/tools' --since=2021-01-01

2021-12-01T18:48:35-3c63f30-6c4bf8f/windows-amd64-race

gopherbot · 2024-01-01T17:27:15Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 21, 2020

bcmills added this to the Backlog milestone May 21, 2020

gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label May 21, 2020

bcmills added this to Test Flakes Dec 1, 2023

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 1, 2023

gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 1, 2024

github-project-automation bot moved this to Done in Test Flakes Jan 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

bcmills commented May 21, 2020 •

edited

Loading

dmitshur commented May 21, 2020

bcmills commented May 21, 2020

gopherbot commented May 27, 2020

JohnCClarke commented Jun 11, 2020 •

edited

Loading

dmitshur commented Jul 6, 2020 •

edited

Loading

bcmills commented Jan 11, 2022 •

edited

Loading

gopherbot commented Jan 1, 2024

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

x/tools: frequent "out of memory" errors on windows-amd64-race builder #39196

Comments

bcmills commented May 21, 2020 • edited Loading

dmitshur commented May 21, 2020

bcmills commented May 21, 2020

gopherbot commented May 27, 2020

JohnCClarke commented Jun 11, 2020 • edited Loading

dmitshur commented Jul 6, 2020 • edited Loading

bcmills commented Jan 11, 2022 • edited Loading

gopherbot commented Jan 1, 2024

bcmills commented May 21, 2020 •

edited

Loading

JohnCClarke commented Jun 11, 2020 •

edited

Loading

dmitshur commented Jul 6, 2020 •

edited

Loading

bcmills commented Jan 11, 2022 •

edited

Loading