FreeTCPAddr returns duplicate ports, results in bind: address already in use errors

[prettymuchbryce]

Summary of Bug

We're seeing quite a few bind: address already in use issues in our cli tests, and I believe it's related to the way network.New finds ports using FreeTCPAddr.

On Mac OS, calling net.Listen on localhost:0 generally produces monotonically increasing port numbers, so the bind: address already in use error does not occur.

On some distributions of Linux, however; the underlying TCP/IP stack is different, and the ports returned are more stochastic. As an experiment I created 10 goroutines which each called FreeTcpAddr 100 times on an Alpine Linux distro. It was common to see that on the 50th or 60th call, that I would get a duplicate port.

It's also important to note that when go tests are invoked with go test ./..., each package is run as a separate instance. Therefore, package level locks like this one don't work. Each package's tests will have a reference to a different lock.

This means each package's cli tests are invoking network.New in parallel, and calling FreeTCPAddr in parallel. Also note that FreeTCPAddr actually listens at localhost:0, and then immediately closes the listener, and there is some time between this and actually starting the underlying gRPC server. Within this time the port is actually free again, and it's possible that other tests running FreeTCPAddr will claim the same port.

I wonder if we can find a different way to find open ports for the gRPC servers rather than using FreeTCPAddr. For example, could we start the gRPC servers on port 0 directly, rather than first calling FreeTCPAddr, stopping the listener, and then starting the gRPC service on the returned port?

Also linking this open issue which seems to reference this bug.

Version

0.47-alpha2

[alexanderbez]

Yeah we see this issue pop up in our CI tests all the time. I don't know of another way to generate a free port. If you have ideas, preferably in an os-independent manner, we're open to PRs!

[yihuang]

We use fixed ports in our integration tests, so at least in CI it'll be stable ;D
Maybe find a range of free ports in advance before running test cases, so at least no conflict between the test cases

[tac0turtle]

we havent seen this in a while and with system tests we should not see this any longer