net/http: don't reuse connections that are experiencing errors [1.20 backport]

[btasker]

Requesting that #59690 be considered for backport into the next 1.20 minor release

Relevant change in Gerrit is https://go-review.googlesource.com/c/net/+/486156

Rationale:

When encountered, this is a serious issue: requests are blocked until the kernel gives up on the socket (with default settings on a Linux host, that's around 17 minutes). If the operator restarts the application as a result, data loss may occur.

Although the fix is in x/net v0.10.0 it will not be available to most developers who likely import net/http (and so rely on the h2 bundle rolled into Go's releases)

The only viable workaround is to disable the H2 client entirely via environment variable.

[mdempsky]

@neild What do you think about this?

Also, is this applicable to 1.19?

[mknyszek]

Friendly ping @neild. :)

[ianlancetaylor]

Opened #60662 for a 1.19 backport.

[Yuhjiang]

hello, btasker, your description inspired me. I've also encountered the problem with go version 1.18.6, and the problem would persist for 15-17min at every time. So I'm curious about the description of "with default settings on a Linux host, that's around 17 minutes". How can I get some related information about it?

[btasker]

Hi @Yuhjiang

Sure. I don't know your level, so apologies if any of this is teaching you to suck eggs.

So, the bit you know:

1. Your application establishes a TCP connection to $server and negotiates to use HTTP/2
2. At some point, packets start getting lost and the connection breaks
3. The application keeps trying to send new requests over that connection, which fail
4. About 15 minutes later, a new connection is established and the application recovers

That 15 minute delay in recovery is a product of the way that the Linux kernel handles TCP retransmissions between 2 and 4 above.

When things are fully broken (i.e. the other end doesn't seem to be responding at all), there's an exponential back-off between each retry, for example

Send
wait 200ms
retry 1
wait 400ms
retry 2
wait 800ms
retry 3
wait 1.6s

etc

The minimum and maximum length of delay between the retries is defined by TCP_RTO_MIN (default 200ms) and TCP_RTO_MAX (default 120s). These hardcoded into the kernel (although you can technically change them with BPF) here.

The number of retries, however, is governed by a sysctl - net.ipv4.tcp_retries2. The default value is 15 (so with each pause doubling in length, you get up to that 15 minute mark).

Eventually, the limit in net.ipv4.tcp_retries2 is hit and the kernel decides the connection is dead and closes it (causing it to drop out of the application's connection pool, with a new connection getting opened instead).

Mitigation

So, if you're looking to mitigate this through system config changes, you could do so via net.ipv4.tcp_retries2. If you drop it to 8 the maximum connection hang time will be around 1 minute (which still isn't great, but is in line with timeout thresholds that often get set at application level).

If you wanted to go a step further and adjust TCP_RTO_MIN and TCP_RTO_MAX using BPF there's (what looks like) a good write up at https://arthurchiao.art/blog/customize-tcp-initial-rto-with-bpf/.

Does that help? Happy to go into more (or less) depth if needed

[Yuhjiang]

@btasker Thank you very much for you detailed reply, I learn a lot from it. I've figured out the problem and the reason. Finally, you've been a big help. I will try x/[email protected] soon to verify if it could solve the problem.

[dmitshur]

Issue #60818 may also be relevant here.

[dmitshur]

Given the report in #60818 of this fix causing a problem, moving back to CherryPickCandidate state so we can take another look.

[alkmc]

@dmitshur seems that #60818 is closed now.
Perhaps it is the right time to look again into this issue?

[dmitshur]

@alkmc Note that #60818 was fixed by reverting the very change that's being requested for backport here, and its issue has been reopened. There's not much we can do, other than reject this cherry-pick candidate, until #59690 becomes resolved again.

[btasker]

@alkmc the way we've mitigated is to instead enable H2 healthchecks.

If you're importing net/http and relying on the H2 bundle, you can't directly access ReadIdleTimeout on the H2 object.

But, net/http2 includes ConfigureTransports which accepts a http.Transport and returns the h2 transport, so you can do

http2Trans, err := http2.ConfigureTransports(transport)
if err == nil {
   http2Trans.ReadIdleTimeout = time.Duration(ReadIdleTimeout)
}

The count-down to ReadIdleTimeout starts when the last frame is received (so is reset everytime the server responds). If nothing is received for that period, a H2 Ping is sent - if that's not answered (there's a PingTimeout attribute if you want to change the threshold for that) the connection is torn down.

One thing I did find though, there's something about setting ReadIdleTimeout which fixes/prevents the race that stopped closeIdleConnections() from working so you may find out your normal timeout setting starts working again (I haven't dug into why)

[dmitshur]

The upstream issue isn't resolved yet, and Go 1.20 has exited its support window per https://go.dev/doc/devel/release#policy.