x/build: add a longtest+race builder

[bcmills]

We have -race builders that run tests with the race detector enabled, and -longtest builders that run tests without -short.
However, most non-trivial races occur in concurrent code, and tests with nontrivial concurrency tend to take longer. So the tests run by the -race builders miss a lot of the actual races.

It would be nice to have at least one builder that runs the full test suite (without -short) under the race detector.

[heschi]

cc @golang/release

[mknyszek]

The race detector builders currently have GO_DISABLE_OUTBOUND_NETWORK=1 while longtest builders have needsGoProxy: true. These seem in conflict, and I don't have a good idea of how they'll actually resolve. I guess I can just try and see what happens?

[heschi]

The long tests require network so you'll want to leave the network enabled.

[mknyszek]

Got it. Also @bcmills do you want this to run for subrepos? Or are you just concerned about the main repo at the moment?

(I figure I'll just start with the main repo since I suspect there will be some surprises...)

[bcmills]

Ideally subrepos too (especially x/sys and x/net), but the main repo is a good starting point.

[mknyszek]

I redeployed the coordinator and the builders are online! (For subrepos as well.) Hopefully the timeout scale is sufficient. We'll see soon.

[mknyszek]

I think the UI could also use some work. It's in the race section but it just says "linux" twice. I'll poke at that.

[gopherbot]

Change https://go.dev/cl/449196 mentions this issue: dashboard: add linux-amd64-longtest-race builder

[mknyszek]

Hm, the builds seem stuck on... what I have to guess is the index/suffixarray tests. On longtest those already take 47 seconds... I wonder if it's ballooning out on longtest x race.

[mknyszek]

For comparison, image/gif longtest seems to be around 6x slower with the race detector. So, assuming that slowdown applies here as well (maybe not) 47 seconds balloons out to around 5 minutes. That seems like a lot.

[mknyszek]

Worse than that actually, 600+ seconds. The runtime tests took 700+ seconds. Unfortunately the builds are incredibly slow. :( How should we proceed? Leave the builder but start disabling tests for this configuration? Is it fine to be this slow?

[mknyszek]

Reference: https://farmer.golang.org/temporarylogs?name=linux-amd64-longtest-race&rev=d2aa787f2a925a97dad11b1acb2a065f52b097f8&st=0xc000fb7dc0

[bcmills]

If the tests pass reliably (with an appropriate timeout scale factor), then I guess the relevant question is just how slow are they?

Given that the race detector dumps exact stack traces for any race it detects, and given that it's for a common platform and architecture (so not hard to find a local machine to run the tests on), I'd say that if the builder takes, say, twice as long as the longtest builder overall, that's probably fine. But if it's going to take, say, 10 hours per run, we probably ought to prune things back in some way.

The log you linked seems to have disappeared, but looking at the index/suffixarray tests it looks like they're running exhaustive checks over huge input spaces (2²¹ and 3¹⁴ cases, respectively). Given that those spaces have already been checked exhaustively, I wonder if those tests would be more cost-effective as fuzz tests sampling fewer elements, perhaps from an even larger space. 🤔

[mknyszek]

It's definitely not 10 hours but I don't have complete data. It does actually seem to be mostly keeping up with tip, so maybe it's fine.

Maybe let's just leave it up and we can work on improving run times more incrementally.

[bcmills]

It does look like x/tools/go/pointer and x/benchmarks/sweet/cmd/sweet might need either some test skips or a longer timeout scale. I see consistent timeouts in those.

[mknyszek]

x/benchmarks/sweet/cmd/sweet is just one test and yeah, it should probably just be disabled for this builder.

[gopherbot]

Change https://go.dev/cl/449518 mentions this issue: go/pointer: break TestInput into subtests and update skips