net/http: inconsistent results using os/exec in handler

[gopherbot]

by cgmurray:

What steps will reproduce the problem?
Run the attached program. See comment in main.go for instructions.

What is the expected output?
Consistent md5-checksum result

What do you see instead?
Random md5-checksum result

Which operating system are you using?
OSX 10.8.3

Which version are you using?  (run 'go version')
go version go1.1 darwin/amd64

Please provide any additional information below.
I've found two workarounds:

I) Change "Content-Type" to "_ontent-Type"
II) Set "Connection: close" header

Attachments:

1. main.go (770 bytes)

[rsc]

Comment 1:

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

[bradfitz]

Comment 2:

I can reproduce on a Mac.
If I either buffer the input or buffer the output, it passes.  (See attached x.go)   It
only produces inconsistent results if it streams in and streams out, which the net/http
package (nor HTTP, really) permits.
But in this case, the md5 binary shouldn't produce any output until its input's EOF, so
I think this should work.
I suspected maybe that the exec package's io.Copy to the rw from its internal os.Pipe
was causing a 0-byte write or something which forced the HTTP package to shut down its
input, but that doesn't appear to be the case.

Labels changed: added suggested.

Owner changed to @bradfitz.

Attachments:

1. x.go (1225 bytes)

[rsc]

Comment 3:

Labels changed: added go1.2.

[bradfitz]

Comment 4:

The problem is os/exec's io.Copy(ResponseWriter, os.Pipe) and finding that the
ResponseWriter implements io.ReaderFrom.
The net/http (*response).ReadFrom immediately then calls WriteHeader and flushes any
buffered writes, in prep to do a sendfile. 
func (w *response) ReadFrom(src io.Reader) (n int64, err error) {
        if !w.wroteHeader {
                w.WriteHeader(StatusOK)
        }
  ...
        w.w.Flush()  // get rid of any previous writes                                                                                             
        w.cw.flush() // make sure Header is written; flush data to rwc                                                                             
Writing the Header & sending it to the client causes the request.Body to be closed
(which md5 is still reading from).

[bradfitz]

Comment 5:

Frowned-upon patch at https://golang.org/cl/12632043/

[bradfitz]

Comment 6:

This issue was closed by revision de04bf2.

Status changed to Fixed.