net/http: support concurrent Request.Body reads & ResponseWriter.Write calls in HTTP/1.x server

[exel5]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?
   Go 1.5.2
2. What operating system and processor architecture are you using (go env)?
   windows_amd64
3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.
   A link on play.golang.org is best.

https://play.golang.org/p/DaWZXCQNfV
this example ends up with error "http: invalid Read on closed Body"

https://play.golang.org/p/WYCsIQzx_F
but changing 100 to 10 in strings.Repeat goes without any errors

also this example https://play.golang.org/p/YxjKnmgfGP
$ curl -d 'qweqweq weq weqwe qwe qew qwe' http://localhost:6060/
<qweqweq weq weq><we qwe qew qwe >

shows that reading and writing goes simultaneously and without errors

1. What did you expect to see?
   i expect to see the same error in both cases
2. What did you see instead?
   i see different behaviour

[bradfitz]

I've sent https://golang.org/cl/23011 to doucment the status quo.

We can keep this bug open to track making the HTTP/1.x server support concurrent reads & writes like HTTP/2.

[gopherbot]

CL https://golang.org/cl/23011 mentions this issue.

[bradfitz]

@dpiddy points out the closest thing in RFC 2616 about this topic:

        If an origin server receives a request that does not include an
        Expect request-header field with the "100-continue" expectation,
        the request includes a request body, and the server responds
        with a final status code before reading the entire request body
        from the transport connection, then the server SHOULD NOT close
        the transport connection until it has read the entire request,
        or until the client closes the connection. Otherwise, the client
        might not reliably receive the response message. However, this
        requirement is not be construed as preventing a server from
        defending itself against denial-of-service attacks, or from
        badly broken client implementations.

[herrberk]

I believe this is a critical issue affecting many users out there and should be addressed as quickly as possible. I was able to reproduce this issue using Transfer-Encoding: chunked as I explained in this related issue:

This sample server-client pair reproduces this issue:
https://play.golang.org/p/G8F0X4DQCDn

Please let me know if any assistance is needed to fix this issue for the next release. Thanks!

[Stebalien]

Relevant HTTP WG email thread: https://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0041.html

TL;DR: Yes, you can respond before reading the entire request.

---

        If an origin server receives a request that does not include an
        Expect request-header field with the "100-continue" expectation,
        the request includes a request body, and the server responds
        with a final status code before reading the entire request body
        from the transport connection, then the server SHOULD NOT close
        the transport connection until it has read the entire request,
        or until the client closes the connection. Otherwise, the client
        might not reliably receive the response message. However, this
        requirement is not be construed as preventing a server from
        defending itself against denial-of-service attacks, or from
        badly broken client implementations.

This section should be interpreted exactly as written. The server can't close the connection before reading the entire body of the request for the reason stated in that section. That doesn't mean the server can't respond concurrently.

[bradfitz]

@Stebalien, thanks for finding that.

FWIW, Go's HTTP/1.x server originally permitted this but we encountered enough confused implementations in the wild that would end up deadlocking if they got a response before they were expecting it. (e.g. somebody sent us a POST + body but the Go server sent an Unauthorized response before reading the body). The peer would then deadlock, never reading the response because it wasn't finished writing its body, and we weren't reading their body because our Handler was done running. We've changed behavior a few times over the years (read to EOF, read some, close immediately) and I can't even remember what we do now.

Still worth revisiting, but I suspect we might need to make this behavior opt-in for HTTP/1.x on a per-Handler basis somehow.

[herrberk]

@Stebalien @bradfitz I understand, permitting this can cause issues like you mentioned but it's already there for HTTP/2, so IMHO it should be added to HTTP/1.x at least as an opt-in feature.

[guonaihong]

I also encountered this problem

• server

func myecho(w http.ResponseWriter, r *http.Request) {
        n, err := io.Copy(w, r.Body)
        if err != nil {
                fmt.Printf("err = %v\n", err)
                //panic(err)
        }

        fmt.Printf("n = %d\n", n)
}

func netHTTPBug() {
        http.HandleFunc("/post", myecho)
        http.ListenAndServe(":8080", nil)
}

• client

seq 600 &>need.data
wc -c need.data 
2292 need.data

curl -X POST -d @./test.data 127.0.0.1:8080/post &>got.data

wc -c got.data 
829 got.data

I looked at the net/http code yesterday, and it seemed that it was enough to modify a bool variable. Is there any troublesome reason not to modify it?

[silverbp]

@guonaihong which bool variable are you referring to? I am running into the issue and looking for a solution to resolve it.

[guonaihong]

Here you can see the code in my screenshot. I remember setting discard to true to run my demo above.

@silverbp

[geofffranks]

This bug has been open for over six years, and is causing a great deal of pain for our customers. Are there any golang maintainers we can work with on trying to get this problem solved finally? Any chance you might be the right person for this @neild, or at least know someone who can help?

[gopherbot]

Change https://go.dev/cl/472636 mentions this issue: net/http: support full-duplex HTTP/1 responses

[gopherbot]

Change https://go.dev/cl/501300 mentions this issue: go1.21: document net/http.ResponseController.EnableFullDuplex

[oliverpool]

This has been implemented, documented and released in go1.21, so I guess this issue can be closed.