testing: document cap the number of concurrent goroutines with -race

[ericlagergren]

What version of Go are you using (go version)?

1.9.2
GOOS=linux
GOARCH=amd64

What did you do?

package main

import (
	"math/rand"
	"strconv"
	"testing"
	"time"
)

func TestGoRoutines(t *testing.T) {
	for i := 0; i < 1<<15; i++ {
		t.Run(strconv.Itoa(i), func(t *testing.T) {
			t.Parallel()
			time.Sleep(time.Duration(rand.Intn(500000000)))
		})
	}
}

saved as x_test.go and run with go test -race

What did you expect to see?

The tests to pass. Instead,

race: limit on 8192 simultaneously alive goroutines is exceeded, dying

As I understand it, this is likely a technical limitation on Go's race detector. However, it would be nice for the testing package to be aware of this and, if raceenabled, cap the number of parallel tests at something below the 8192 limit (say, 7000?). Perhaps a better way would be a warning, but I know Go doesn't really like warnings.

[ianlancetaylor]

I'm curious as to whether and how this arose in reality. Do you have tests that intentionally use t.Parallel to run 8192 concurrent goroutines?

[ericlagergren]

Yeah, so originally I had N tests each with M subtests, not too dissimilar from this example in the testing docs:

func TestGroupedParallel(t *testing.T) {
    for _, tc := range tests {
        tc := tc // capture range variable
        t.Run(tc.Name, func(t *testing.T) {
            t.Parallel()
            ...
        })
    }
}

Running the tests that way allowed them to complete in half the time (something like 1 instead of 2 minutes).

After drastically increasing the number of (auto-generated) M, I started running into mysterious TravisCI errors which I then discovered was the 8192 goroutine limit.

[ianlancetaylor]

I don't think is possible. There is already a limit on the number of tests that are run in parallel, controlled by the -test.parallel option, whose default is runtime.GOMAXPROCS(0). It doesn't help, though, because by the time we know that the test is going to run in parallel we have already started it. We can't not start a test, because then we would never finish running all the non-parallel tests.

I'm going to close this as impossible but I'd be willing to hear a proposal for how to implement it.

[ericlagergren]

@ianlancetaylor fair enough. Could this be a doc fix, though? It was confusing the first time I stumbled upon it. I mean, the testing docs themselves give an example (which I quoted above) that could easily be used to hit the 8,192 limit. I'd be down to submit a doc fix.

[ianlancetaylor]

I'm OK with a doc fix but I'm not sure where it should go. Maybe in https://blog.golang.org/race-detector ?

[ericlagergren]

Throwing a couple ideas out there:

• A comment under theTestGroupedParallel example that warns against using too many parallel tests if race is enabled. Perhaps something like, "Note: because it can be easy to run many parallel tests, care should be taken when the -race flag is toggled, as Go's race detector will kill the program if more than 8,192 goroutines are run simultaneously."

• A // BUG(...) comment which warns against it in testing. (But, it's not really a bug per se...)

• An addendum to the blog post is a good idea, but I've a feeling most people won't see it. (Could be mitigated if the text race: limit on 8192 simultaneously alive goroutines is exceeded, dying is added verbatim, since people will likely copy and paste that into Google.)

[ianlancetaylor]

I'm fine with a comment in TestGroupedParallel, want to send a pull request?

[ericlagergren]

@ianlancetaylor yessir. I'll get to it tonight. Do you want it inside TestGroupedParallel or just above/below it?

[ianlancetaylor]

I would guess below it, actually.

[gopherbot]

Change https://golang.org/cl/103160 mentions this issue: testing: document that -race can cause affect how many parallel tests

[ericlagergren]

@ianlancetaylor great that's where I added it. And darn, I should've tried making it with the new GitHub deal instead of git-codereview :)