x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

dmitshur · 2020-01-07T22:58:30Z

The linux-amd64-race builder is currently configured with 15 GB RAM (via the n1-standard-4 machine type). CL 213557 has recently increased the RAM for windows-amd64-race builder to 14.4 GB in order to work around 7.2 GB not being sufficient, as described at #35186 (comment).

The freebsd-amd64-race builder is currently configured to use the n1-highcpu-4 machine type, which has 3.6 GB RAM. The freebsd-amd64-race builder is failing quite often, which may be connected to insufficient memory:

signal: killed
FAIL	golang.org/x/tools/go/packages	217.792s

*** Test killed with quit: ran too long (10m0s).
FAIL	golang.org/x/tools/internal/imports	600.067s

panic: test timed out after 10m0s
[...]
FAIL	golang.org/x/tools/go/packages	611.592s

[...]
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:29 Error:2020/01/07 18:44:29 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:35 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:35 Error:2020/01/07 18:44:35 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 18:44:36 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:36 Error:2020/01/07 18:44:36 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:42 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
2020/01/07 18:44:43 Error:2020/01/07 18:44:42 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
--- FAIL: TestCommandLine (240.28s)
    --- FAIL: TestCommandLine/Modules (194.21s)
        --- FAIL: TestCommandLine/Modules/Diagnostics (159.73s)
            --- FAIL: TestCommandLine/Modules/Diagnostics/testy_test (151.73s)
                check.go:62: missing diagnostic "testy/testy_test.go:6:6: x declared but not used", map[]
FAIL
FAIL	golang.org/x/tools/internal/lsp/cmd	240.922s

2020/01/07 20:54:34 no dep handle: no metadata for nosuchpkg
	package = nosuchpkg
2020/01/07 20:54:51 no dep handle: no metadata for golang.org/x/tools/internal/lsp/protocol
	package = golang.org/x/tools/internal/lsp/protocol
2020/01/07 20:54:51 no dep handle: no metadata for golang.org/x/xerrors
	package = golang.org/x/xerrors
FAIL	golang.org/x/tools/internal/lsp/source	246.068s

If there aren't objections, I plan to bump its up RAM to be more similar in size (without exceeding the current value of 15 GB RAM used both by Linux and now Windows -race builders) to see it helps with the test failures and issues like #34621.

/cc @bradfitz @bcmills @matloob @toothrot

The text was updated successfully, but these errors were encountered:

gopherbot · 2020-01-13T19:24:36Z

Change https://golang.org/cl/214433 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM

Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for the freebsd-amd64-race builder. The freebsd-amd64-race builder has produced good test results for the x/tools repo for a long time, but by now it has started to consistently fail for reasons that seem connected to it having only 3.6 GB memory. The Windows race builders needed to be bumped from 7.2 GB to 14.4 GB to run successfully, so this change makes a small incremental step to bring freebsd-amd64-race closer in line with other builders. If memory-related problems continue to occur with 7.2 GB, the next step will be to go up to 14.4 GB. The freebsd-amd64-race builder is using an older version of FreeBSD. We may want to start using a newer one for running tests with -race, but that should be a separate change so we can see the results of this change without another confounding variable. Also update all FreeBSD builders to use https in buildletURLTmpl, because it's expected to work fine and will be more consistent. Updates golang/go#36444 Updates golang/go#34621 Updates golang/go#29252 Updates golang/go#33986 Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac Reviewed-on: https://go-review.googlesource.com/c/build/+/214433 Run-TryBot: Carlos Amedee <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

dmitshur · 2020-01-14T19:51:24Z

CL 214433 was deployed this morning, and the freebsd-amd64-race builder has been passing on x/tools since then:

I'll give it more time to gather more data, and close this issue if the reliable build results continue.

dmitshur · 2020-01-16T18:46:08Z

The freebsd-amd64-race builder no longer failing consistently on x/tools. However, there are still occasional failures. Need to see if they are connected to limited memory, and if so, consider going up to 15 GB on the FreeBSD race builder to match the memory of Linux and Windows race builders.

Edit: Many of the failures are due to data races being detected. Filed issue #36605.

bcmills · 2020-02-20T14:23:48Z

The failures in https://build.golang.org/log/74eea9d200fd346319a2595c2f2b2ab3f25c2a07 appear to be due to timeouts rather than data races, although it is possible that a data race failure is being masked by failure to propagate logs somewhere higher up the stack.

dmitshur · 2020-04-07T15:35:37Z

I'll give it more time to gather more data, and close this issue if the reliable build results continue.

I think the build results are reliable, keeping the specific issues mentioned above in mind.

I'll close this since the original issue is resolved, and we can follow up with more specific issues as needed.

dmitshur · 2020-04-07T15:40:07Z

Whoops, I misread the issue and missed the "x/tools" prefix, thinking it was about the main Go repository.

This is still a problem for x/tools:

https://build.golang.org/?repo=golang.org%2fx%2ftools

I'll re-open and look into this more later.

gopherbot · 2020-04-10T16:01:02Z

Change https://golang.org/cl/227859 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 16 vCPUs, 14.4 GB mem

This is a followup to CL 214433. Start using n1-highcpu-16 machine type instead of n1-highcpu-8 for the freebsd-amd64-race builder. Increasing the RAM from 3.6 GB to 7.2 GB has helped golang/go#36444 significantly: the builder stopped failing consistently on x/tools and resulted in many data races being uncovered in golang/go#36605. However, by now, it has started to fail consistently again. This time it seems to be due to low performance, causing the tests in golang.org/x/tools/internal/lsp/regtest package to fail due with "context deadline exceeded" errors. FreeBSD is one of the ports that stays visible when "show only first- class ports" is checked on build.golang.org. The other -race builders have all been upgraded to the n1-highcpu-16 machine type by now out of necessity. It seems fair to provide the FreeBSD port with an equal amount of resources, even if the increased memory isn't strictly required yet. Once this change is applied, if the failures persist, we can be more confident that the problem is due to the code or the port, rather than due to this -race builder having 2𝗑 less CPU and RAM resources compared to other -race builders. An alternative is to increase timeout for this builder type, but I'm opting to defer exploring that after equalizing the machine type. For golang/go#36444. For golang/go#34621. For golang/go#29252. For golang/go#33986. Change-Id: I41f149365128c7bc6f576c778ac07618acc04612 Reviewed-on: https://go-review.googlesource.com/c/build/+/227859 Reviewed-by: Alexander Rakoczy <[email protected]>

dmitshur · 2020-04-10T22:49:23Z

CL 227859 has been deployed earlier today. The freebsd-amd64-race builder has started to pass on x/tools:

I'll consider this resolved. If something unexpected comes up, we can re-open this issue or file a new one.

dmitshur added OS-FreeBSD Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 7, 2020

dmitshur added this to the Unreleased milestone Jan 7, 2020

gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Jan 7, 2020

This was referenced Jan 7, 2020

x/tools/go/packages: test timeout on freebsd-amd64-race builder #34621

Closed

x/tools: build failing on windows-amd64-longtest #35186

Closed

x/tools/internal/lsp: frequent out-of-memory errors on windows-amd64-race builder #33951

Closed

dmitshur self-assigned this Jan 13, 2020

dmitshur mentioned this issue Jan 15, 2020

x/tools/go/internal/gcimporter: frequent failures with SIGSEGV on openbsd-386-64 builder #36563

Closed

bcmills mentioned this issue Feb 21, 2020

x/tools/internal/lsp/regtest: various flakes due to go.mod contention #37318

Closed

dmitshur closed this as completed Apr 7, 2020

dmitshur reopened this Apr 7, 2020

dmitshur changed the title ~~x/tools: freebsd-amd64-race builder is failing consistently~~ x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo Apr 7, 2020

dmitshur closed this as completed Apr 10, 2020

golang locked and limited conversation to collaborators Apr 10, 2021

gopherbot added the FrozenDueToAge label Apr 10, 2021

rsc unassigned dmitshur Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

dmitshur commented Jan 7, 2020

gopherbot commented Jan 13, 2020

dmitshur commented Jan 14, 2020

dmitshur commented Jan 16, 2020 •

edited

Loading

bcmills commented Feb 20, 2020

dmitshur commented Apr 7, 2020

dmitshur commented Apr 7, 2020

gopherbot commented Apr 10, 2020

dmitshur commented Apr 10, 2020 •

edited

Loading

x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

x/build: freebsd-amd64-race builder is failing consistently on golang.org/x/tools repo #36444

Comments

dmitshur commented Jan 7, 2020

gopherbot commented Jan 13, 2020

dmitshur commented Jan 14, 2020

dmitshur commented Jan 16, 2020 • edited Loading

bcmills commented Feb 20, 2020

dmitshur commented Apr 7, 2020

dmitshur commented Apr 7, 2020

gopherbot commented Apr 10, 2020

dmitshur commented Apr 10, 2020 • edited Loading

dmitshur commented Jan 16, 2020 •

edited

Loading

dmitshur commented Apr 10, 2020 •

edited

Loading