404 responses for chunk-encoded requests may not be received

[rasjidw]

404 responses (or any response sent by the server prior to the entire request being sent) for chunk-encoded requests may not be received by the caller.

This appears to happen if the server closes the connection (after sending the 404 response) prior to the completion of the chunked post / upload.

Expected Result

Posting a chunked encoded request to an invalid URL should return the 404 response.

Actual Result

A requests.exceptions.ConnectionError: [Errno 32] Broken pipe error is raised instead.

Reproduction Steps

A minimal server using flask is given below:

#!/usr/bin/env python3

import sys

from flask import Flask, request
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

@app.route('/post_endpoint', methods=['POST'])
def post_normal_response():
    for line in request.stream:
        str_line = line.decode('utf-8')
        sys.stdout.write(str_line)
        sys.stdout.flush()
    return 'OK'

if __name__ == '__main__':
    app.run()

Client code illustrating the problem is:

#!/usr/bin/env python3

from time import sleep

import requests


VALID_URL = 'http://127.0.0.1:5000/post_endpoint'
INVALID_URL = 'http://127.0.0.1:5000/does_not_exist'


LINES = ['Line {}\n'.format(x) for x in range(5)]
DATA = ''.join(LINES).encode('utf-8') 

def chunked():
    for line in LINES:
       sleep(1)
       yield line.encode('utf-8') 

def test_url(url):
    r = requests.post(url, data=DATA)
    print(r)


    r = requests.post(url, data=chunked())
    print(r)

if __name__ == '__main__':
   test_url(VALID_URL)
   print('-----')
   test_url(INVALID_URL)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.4.0-104-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

Patch to address the issue

requests-patch.zip

[sethmlarson]

Thanks for this! Instead of submitting the patch as a zip, consider submitting a Pull Request and linking it to this issue. :)

[rasjidw]

Pull request done. If the approach is okay I can look at adding tests etc.

[sethmlarson]

Chunked requests are now handled by urllib3.