Support chunked transfer encoding #1198

jan-auer · 2017-11-27T18:59:05Z

This PR introduces support for requests with chunked transfer encoding to WSGIRequestHandler by wrapping the input in a DechunkedInput. Chunks are unrolled in the following way

Read a line to get the chunk's size in bytes
Read the chunk data, but only to the length of the buffer
Reduce the chunk size by the number of bytes read
At the end of the chunk, skip the chunk's final newline

Once a chunk of size zero is encountered, a final newline is read, and then the input is marked as done. All further requests to readinto will exit early.

There is only a test for the success case. It needs to issue a manual HTTP to achieve a proper chunked multipart payload.

Note: For some reason, werkzeug crashes when parsing the response body if a Content-Length header is present. If you would like this fixed in this PR too, let me know. RFC 2616, Section 4.4 states that the Content-Length header must be ignored with any Transfer-Encoding other than identity.

Fixes #1149

davidism · 2017-11-27T19:53:37Z

Thanks for working on this!

davidism · 2017-11-27T19:54:08Z

There were some issues with our tests on Travis, you'll need to merge in master to fix the unrelated failing tests.

CONTRIBUTING.rst

@@ -54,7 +54,7 @@ Install Werkzeug in editable mode::

 Install the minimal test requirements::

-    pip install pytest pytest-xprocess requests
+    pip install pytest pytest-xprocess requests six


mitsuhiko · 2017-11-27T20:51:15Z

Generally looks good. Needs some fixes for the test failures and I will look into what uses the content length currently.

davidism · 2017-11-27T20:56:41Z

Hold off merging until I roll back the changes mentioned in #1149 (#1126), since they're related.

- Try to read multiple chunks until the buffer is filled - Add comments to make control flow more clear - Add doc comment for DechunkedInput

Required by RFC 2616, Section 4.4 See: https://www.greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4

jan-auer · 2017-11-28T12:03:57Z

Pushed some updates:

The DechunkedInput read only one chunk at a time and then immediately returned the buffer. Now it tries to fill the buffer and put as many chunks (or parts of it) as possible
Also added some comments and a docstring to make the control flow more clear. It's still a bit weird on first sight, but someone familiar with the spec should understand it easily
Fixed the bug where the Content-Length header crashed the form parser. The problem is that the content length header includes chunk headers and newlines, but the payload returned by DechunkedInput does not. The form parser, however, wraps the input in a LimitedStream if a Content-Length header is present, which asserts they are the same. By removing the content length, the form parser does not apply LimitedStream anymore and the problem goes away. Also, this is more spec compliant.
Replaced six by an explicit import based on the python version. Could have put it in _compat like suggested, but since httplib/http.client is only used in the test, it's easer this way.
Finally, the test had some unused code which is removed now.

werkzeug/formparser.py

@@ -38,7 +38,7 @@
 def default_stream_factory(total_content_length, filename, content_type,
                           content_length=None):
    """The stream factory that is used per default."""
-    if total_content_length > 1024 * 500:
+    if total_content_length is None or total_content_length > 1024 * 500:


davidism · 2017-11-28T15:52:22Z

Reverted #1126 in #1200, please merge master. Still working on unrelated test failures.

* master: Revert "Allow chunk request" fix flake8 errors codecov needs argparse on 2.6 Fix redis tests

jan-auer · 2017-11-28T16:04:00Z

Alright, here you go. FWIW, I also ran some manual tests with uwsgi 2.0.15 and master in both python 2.7 and 3.6 and it seemed to work just fine.

Just one thing I'm still wondering about is whether we need to take care of infinite streams in DechunkedInput, or is something else already dealing with that?

davidism · 2017-11-28T16:14:27Z

This shouldn't affect other WSGI server support of chunked encoding, but thanks for testing it.

#1126 was supposed to handle infinite streams by setting a hard limit on the content read in, but that was based on some incorrect assumptions from me. I can't recall any special handling in Gunicorn from when I was looking a while ago, but might be good to check.

body in send instead of endheaders

davidism · 2017-11-28T17:08:13Z

I merged master again and fixed some 2.6 and style issues. Travis should pass now.

jan-auer · 2017-11-28T17:46:37Z

Whoop, it's all green! 🎉

davidism · 2017-11-28T17:48:13Z

Would you mind looking at how this behaves with infinite streams (not chunked, but no content length) versus how Gunicorn behaves?

jan-auer · 2017-11-28T17:53:23Z

Sure 👍 Please let me know if you or @mitsuhiko have a perspective on this, though.

mitsuhiko · 2017-11-28T23:40:59Z

I will merge this in the current state now. I think we can look at the edge cases once we see how others are doing it. This is basically just the local dev setup anyways.

jan-auer added 2 commits November 27, 2017 19:41

Support chunked transfer encoding

f7bc8d3

Remove comment

beb2168

davidism reviewed Nov 27, 2017

View reviewed changes

davidism requested a review from mitsuhiko November 27, 2017 19:55

jan-auer added 4 commits November 28, 2017 10:24

Import HTTPConnection from correct module

1f91721

Remove unused test code

18f53e7

Improve DechunkedInput implementation and add comments

fcf6a06

- Try to read multiple chunks until the buffer is filled - Add comments to make control flow more clear - Add doc comment for DechunkedInput

Ignore Content-Length with chunked transfer encoding

16923bb

Required by RFC 2616, Section 4.4 See: https://www.greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4

jan-auer added 2 commits November 28, 2017 15:13

Default to streaming into a temporary file if content length is missing

9806c51

Fix a test assertion in python3

ce9b978

jan-auer commented Nov 28, 2017

View reviewed changes

jan-auer added 3 commits November 28, 2017 16:55

Merge branch 'master' into transfer-encoding-chunked

766ec31

* master: Revert "Allow chunk request" fix flake8 errors codecov needs argparse on 2.6 Fix redis tests

Use spooled temporary files to stream files

bb01f57

Add a test with both chunked encoding and content length

bea195e

davidism mentioned this pull request Nov 28, 2017

Regression with chunked transfer encoding? postmanlabs/httpbin#340

Closed

davidism added 2 commits November 28, 2017 08:47

Merge branch 'master' into transfer-encoding-chunked

7ef201e

fix 2.6 test, flake8

7302f1c

body in send instead of endheaders

This was referenced Jan 15, 2018

Scheduled weekly dependency update for week 02 massgo/league#183

Closed

Update werkzeug to 0.14.1 bhrutledge/jahhills.com#119

Closed

pyup-bot mentioned this pull request Jan 22, 2018

Scheduled weekly dependency update for week 03 massgo/league#184

Closed

This was referenced Jan 29, 2018

Scheduled weekly dependency update for week 04 massgo/league#185

Closed

Scheduled monthly dependency update for February watchdogpolska/petycja-norweskie#15

Closed

This was referenced Feb 5, 2018

Scheduled weekly dependency update for week 05 massgo/league#186

Closed

Update werkzeug to 0.14.1 eneldoserrata/odoo#31

Closed

Update werkzeug to 0.14.1 urfonline/api#36

Closed

Scheduled weekly dependency update for week 06 massgo/league#187

Closed

This was referenced Feb 13, 2018

Update werkzeug to 0.14.1 mozilla-services/pulseguardian#58

Closed

Update werkzeug to 0.14.1 gozer/irc-iplimit#7

Closed

Scheduled weekly dependency update for week 07 massgo/league#188

Closed

This was referenced Feb 26, 2018

Scheduled weekly dependency update for week 08 massgo/league#189

Closed

Scheduled monthly dependency update for March watchdogpolska/petycja-norweskie#16

Closed

pyup-bot mentioned this pull request Mar 5, 2018

Scheduled weekly dependency update for week 09 massgo/league#190

Closed

abathur mentioned this pull request Aug 18, 2018

SpooledTemporaryFile exceptions on file upload #1344

Closed

ghost mentioned this pull request Oct 19, 2018

Flask update needed fission/fission#896

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support chunked transfer encoding #1198

Support chunked transfer encoding #1198

jan-auer commented Nov 27, 2017 •

edited

Loading

davidism commented Nov 27, 2017

davidism commented Nov 27, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

mitsuhiko commented Nov 27, 2017

davidism commented Nov 27, 2017 •

edited

Loading

jan-auer commented Nov 28, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

davidism commented Nov 28, 2017

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

mitsuhiko commented Nov 28, 2017

Support chunked transfer encoding #1198

Support chunked transfer encoding #1198

Conversation

jan-auer commented Nov 27, 2017 • edited Loading

davidism commented Nov 27, 2017

davidism commented Nov 27, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

mitsuhiko commented Nov 27, 2017

davidism commented Nov 27, 2017 • edited Loading

jan-auer commented Nov 28, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

davidism commented Nov 28, 2017

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

davidism commented Nov 28, 2017

jan-auer commented Nov 28, 2017

mitsuhiko commented Nov 28, 2017

jan-auer commented Nov 27, 2017 •

edited

Loading

davidism commented Nov 27, 2017 •

edited

Loading