Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamed uploads fail against local httpbin #33

Open
jkbrzt opened this issue Mar 7, 2016 · 7 comments
Open

Streamed uploads fail against local httpbin #33

jkbrzt opened this issue Mar 7, 2016 · 7 comments

Comments

@jkbrzt
Copy link

jkbrzt commented Mar 7, 2016

Against the local httpbin It fails every time and the failuire has two flavours (see bellow).

Relevant: https://github.com/kennethreitz/requests/issues/2422

test.py

import sys
import requests


def stream():
    yield b'ham'
    yield b'spam'


def test_stream():
    r = requests.post('http://httpbin.org/post', data=stream())
    assert r.json()['data'] == 'hamspam'


def test_stream_local_httpbin(httpbin):
    r = requests.post(httpbin.url + '/post', data=stream())
    assert r.json()['data'] == 'hamspam'

body missing:

$ py.test test.py
===================================================== test session starts ======================================================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1, httpbin-0.2.3
collected 2 items

test.py .F

=========================================================== FAILURES ===========================================================
__________________________________________________ test_stream_local_httpbin ___________________________________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_stream_local_httpbin(httpbin):
        r = requests.post(httpbin.url + '/post', data=stream())
>       assert r.json()['data'] == 'hamspam'
E       assert '' == 'hamspam'
E         + hamspam

test.py:17: AssertionError
----------------------------------------------------- Captured stderr call -----------------------------------------------------
127.0.0.1 - - [07/Mar/2016 13:39:07] "POST /post HTTP/1.1" 200 389
============================================== 1 failed, 1 passed in 1.94 seconds ==============================================

requests.exceptions.ConnectionError: [Errno 32] Broken pipe:

$ py.test test.py
============================= test session starts =============================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1, httpbin-0.2.3
collected 2 items

test.py .F

================================== FAILURES ===================================
__________________________ test_stream_local_httpbin __________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_stream_local_httpbin(httpbin):
>       r = requests.post(httpbin.url + '/post', data=stream())

test.py:16:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/api.py:107: in post
    return request('post', url, data=data, json=json, **kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/api.py:53: in request
    return session.request(method=method, url=url, **kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/sessions.py:468: in request
    resp = self.send(prep, **send_kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/sessions.py:576: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests.adapters.HTTPAdapter object at 0x103501eb8>
request = <PreparedRequest [POST]>, stream = False
timeout = <requests.packages.urllib3.util.timeout.Timeout object at 0x103521860>
verify = True, cert = None, proxies = OrderedDict()

    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        """Sends PreparedRequest object. Returns Response object.

            :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
            :param stream: (optional) Whether to stream the request content.
            :param timeout: (optional) How long to wait for the server to send
                data before giving up, as a float, or a :ref:`(connect timeout,
                read timeout) <timeouts>` tuple.
            :type timeout: float or tuple
            :param verify: (optional) Whether to verify SSL certificates.
            :param cert: (optional) Any user-provided SSL certificate to be trusted.
            :param proxies: (optional) The proxies dictionary to apply to the request.
            """

        conn = self.get_connection(request.url, proxies)

        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(request)

        chunked = not (request.body is None or 'Content-Length' in request.headers)

        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError as e:
                # this may raise a string formatting error.
                err = ("Invalid timeout {0}. Pass a (connect, read) "
                       "timeout tuple, or a single float to set "
                       "both timeouts to the same value".format(timeout))
                raise ValueError(err)
        else:
            timeout = TimeoutSauce(connect=timeout, read=timeout)

        try:
            if not chunked:
                resp = conn.urlopen(
                    method=request.method,
                    url=url,
                    body=request.body,
                    headers=request.headers,
                    redirect=False,
                    assert_same_host=False,
                    preload_content=False,
                    decode_content=False,
                    retries=self.max_retries,
                    timeout=timeout
                )

            # Send the request.
            else:
                if hasattr(conn, 'proxy_pool'):
                    conn = conn.proxy_pool

                low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)

                try:
                    low_conn.putrequest(request.method,
                                        url,
                                        skip_accept_encoding=True)

                    for header, value in request.headers.items():
                        low_conn.putheader(header, value)

                    low_conn.endheaders()

                    for i in request.body:
                        low_conn.send(hex(len(i))[2:].encode('utf-8'))
                        low_conn.send(b'\r\n')
                        low_conn.send(i)
                        low_conn.send(b'\r\n')
                    low_conn.send(b'0\r\n\r\n')

                    # Receive the response from the server
                    try:
                        # For Python 2.7+ versions, use buffering of HTTP
                        # responses
                        r = low_conn.getresponse(buffering=True)
                    except TypeError:
                        # For compatibility with Python 2.6 versions and back
                        r = low_conn.getresponse()

                    resp = HTTPResponse.from_httplib(
                        r,
                        pool=conn,
                        connection=low_conn,
                        preload_content=False,
                        decode_content=False
                    )
                except Exception as e:
                    # If we hit any problems here, clean up the connection.
                    # Then, reraise so that we can handle the actual exception.
                    low_conn.close()
                    raise

        except (ProtocolError, socket.error) as err:
>           raise ConnectionError(err, request=request)
E           requests.exceptions.ConnectionError: [Errno 32] Broken pipe

/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/adapters.py:426: ConnectionError
---------------------------- Captured stderr call -----------------------------
127.0.0.1 - - [07/Mar/2016 13:36:28] "POST /post HTTP/1.1" 200 389
===================== 1 failed, 1 passed in 2.10 seconds ======================
@sigmavirus24
Copy link

This probably belongs as a report against httpbin.

@kevin1024
Copy link
Owner

@jkbrzt does this work against httpbin.org?

@jkbrzt
Copy link
Author

jkbrzt commented Mar 8, 2016

@kevin1024 yes it does work against httpbin.org. The test.py has two test cases:

  1. test_stream runs against httpbin.org and always passes
  2. test_stream_local_httpbin runs agans the local httpbin and always fails

@jkbrzt
Copy link
Author

jkbrzt commented Mar 8, 2016

I'm wondering if switching to Gunicorn (#28) would solve it. Will try to play with it a bit.

@kennethreitz
Copy link

Gunicorn solves everything :)

@jkbrzt
Copy link
Author

jkbrzt commented Mar 8, 2016

@kennethreitz I wish! 🦄

After some more debugging it doesn't seem to have anything to do with neither gunicorn nor pytest.

test.py:

from __future__ import print_function
import requests

def stream():
    yield b'spam'

def test_httpbin_org():
    r = requests.post('http://httpbin.org/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_local_httpbin():
    r = requests.post('http://127.0.0.1:5000/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_local_httpbin_gunicorn():
    r = requests.post('http://127.0.0.1:8000/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_pytest_httpbin(httpbin):
    r = requests.post(httpbin.url + '/post', data=stream())
    assert r.json()['data'] == 'spam'

if __name__ == '__main__':
    for test in [test_httpbin_org, test_local_httpbin, test_local_httpbin_gunicorn]:
        print(test.__name__, end=': ')
        try:
            test()
        except AssertionError as e:
            print('fail')
        else:
            print('ok')
python -m httpbin.core &  # naked httpbin
gunicorn httpbin:app &    # httpbin behind gunicorn
$ python test.py   # outside pytest
test_httpbin_org: ok
test_local_httpbin: fail
test_local_httpbin_gunicorn: fail
$ py.test test.py  # through pytest
=============================== test session starts ===============================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /Users/jakub/Code/OS/pytest-httpbin, inifile:
plugins: httpbin-0.2.2, cov-2.2.1
collected 4 items

test.py .FFF

==================================== FAILURES =====================================
_______________________________ test_local_httpbin ________________________________

    def test_local_httpbin():
        r = requests.post('http://127.0.0.1:5000/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:13: AssertionError
___________________________ test_local_httpbin_gunicorn ___________________________

    def test_local_httpbin_gunicorn():
        r = requests.post('http://127.0.0.1:8000/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:17: AssertionError
_______________________________ test_pytest_httpbin _______________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_pytest_httpbin(httpbin):
        r = requests.post(httpbin.url + '/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:21: AssertionError
------------------------------ Captured stderr call -------------------------------
127.0.0.1 - - [08/Mar/2016 18:19:27] "POST /post HTTP/1.1" 200 389
======================= 3 failed, 1 passed in 1.73 seconds ========================

@guyskk
Copy link

guyskk commented Nov 8, 2017

The test does not work against httpbin.org now, it will response 411 Length Required:

>>> import requests
>>> requests.__version__
'2.18.4'
>>> 
>>> def stream():
...     yield b'ham'
...     yield b'spam'
... 
>>> 
>>> r = requests.post('http://httpbin.org/post', data=stream())
>>> r
<Response [411]>
>>> r.reason
'Length Required'
>>> 

If test against a local gunicorn server, r.json()['data'] will be an empty string, the reason is werkzeug ignore chunked request body(werkzeug.wsgi.py:get_input_stream):

    # If the request doesn't specify a content length and there is no max
    # length set, returning the stream is potentially dangerous because it
    # could be infinite, maliciously or not. If safe_fallback is true, return
    # an empty stream for safety instead.
    if content_length is None and max_content_length is None:
        return safe_fallback and _empty_stream or stream

Sometimes it will cause requests.exceptions.ConnectionError: [Errno 32] Broken pipe, the reason is requests still write date after the server close connection, the net traffic is:

time>>>---------------------------------------------------------------------------------------------------------------->>>
requests:  send request headers                                                           send chunked body ...
httpbin:                                          send response    close connection

If add time.sleep(1) before the yield of stream function, it will always cause Broken pipe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants