Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix deflate compression (#4506) #4511

Merged
merged 1 commit into from
Jan 27, 2020
Merged

Fix deflate compression (#4506) #4511

merged 1 commit into from
Jan 27, 2020

Conversation

socketpair
Copy link
Contributor

@socketpair socketpair commented Jan 17, 2020 •
edited
Loading

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
    • The format is <Name> <Surname>.
    • Please keep alphabetical order, the file is sorted by names.
  • Add a new news fragment into the CHANGES folder
    • name it <issue_id>.<type> for example (588.bugfix)
    • if you don't have an issue_id change it to the pr id after creating the pr
    • ensure type is one of the following:
      • .feature: Signifying a new feature.
      • .bugfix: Signifying a bug fix.
      • .doc: Signifying a documentation improvement.
      • .removal: Signifying a deprecation or removal of public API.
      • .misc: A ticket has been closed, but it is not of interest to users.
    • Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

@socketpair socketpair requested a review from asvetlov as a code owner January 17, 2020 14:27
@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Jan 17, 2020
@socketpair
Copy link
Contributor Author

socketpair commented Jan 17, 2020 •
edited
Loading

I'm not sure I've done it right.

Websockets must use deflate without headers according to RFC.

I did not find anything about multipart....so I did not fix it.

@socketpair socketpair force-pushed the wbits branch 4 times, most recently from 118a5a3 to 5ccb7a4 Compare January 17, 2020 16:14
@codecov-io
Copy link

codecov-io commented Jan 17, 2020 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@4d03dbb). Click here to learn what that means.
The diff coverage is 71.42%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #4511   +/-   ##
=========================================
  Coverage          ?   97.54%           
=========================================
  Files             ?       43           
  Lines             ?     8906           
  Branches          ?     1404           
=========================================
  Hits              ?     8687           
  Misses            ?       98           
  Partials          ?      121
Impacted Files Coverage Δ
aiohttp/http_writer.py 99.05% <ø> (ø)
aiohttp/web_response.py 97.71% <100%> (ø)
aiohttp/http_parser.py 96.42% <66.66%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d03dbb...ae7395a. Read the comment docs.

@socketpair socketpair force-pushed the wbits branch 4 times, most recently from f99c91b to 06a94d1 Compare January 17, 2020 16:40
@socketpair
Copy link
Contributor Author

tests failed on Cython on MAC:

image

@asvetlov
Copy link
Member

asvetlov commented Jan 18, 2020 •
edited
Loading

Please update against the master.
I've disabled the failed check, the buggy test was very weak anyway.

@socketpair socketpair force-pushed the wbits branch 3 times, most recently from f5f6895 to f7b7746 Compare January 24, 2020 18:20
socketpair
socketpair commented Jan 24, 2020
View reviewed changes
aiohttp/http_parser.py Outdated
# RFC1950
# bits 0..3 = CM = 0b1000 = 8 = "deflate"
# bits 4..7 = CINFO = 1..7 = windows size.
if not self._started_decoding and self.encoding == 'deflate' and chunk[0] & 0xf != 8:
Copy link
Contributor Author

@socketpair socketpair Jan 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on very small chunks it's not enough to trigger error in the first call to decompressor. I wrote the test on it.

So, I changed the code to check the first byte of the first chunk.

socketpair
socketpair commented Jan 24, 2020
View reviewed changes
aiohttp/http_parser.py

if chunk:
self._started_decoding = True
Copy link
Contributor Author

@socketpair socketpair Jan 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a bug. this value must be set as soon as we passed something to a decoder. Small data passed as a compressed chunk might not yeld decompressed chunk.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

@socketpair socketpair force-pushed the wbits branch 6 times, most recently from a558dd9 to d80be17 Compare January 24, 2020 19:24
@socketpair
Copy link
Contributor Author

aiohttp/http_parser.py:746:13: E125 continuation line with same indent as next logical line

I'm unable to fix it. It seems I tried everything.

@asvetlov please review and/or add reviewers

@asvetlov
Copy link
Member

Got you, I'll pick up the PR

socketpair
socketpair commented Jan 27, 2020
View reviewed changes
aiohttp/http_parser.py
# RFC1950
# bits 0..3 = CM = 0b1000 = 8 = "deflate"
# bits 4..7 = CINFO = 1..7 = windows size.
if not self._started_decoding and self.encoding == 'deflate' \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asvetlov should we decode non-RFC compliant data? I would remove such support. But unfortunately, this will break communication with deflate-enabled aiohttp server.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep it for a while. I have a feeling that we need to support this mode for three years at least.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

urllib3 used by requests has a similar workaround: https://github.com/urllib3/urllib3/blob/9971e27e83a891ba7b832fa9e5d2f04bbcb1e65f/src/urllib3/response.py#L55-L61

asvetlov
asvetlov approved these changes Jan 27, 2020
View reviewed changes
Copy link
Member

@asvetlov asvetlov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.
Let's merge and see if people start complaining :)
Please let me do merging myself, I want to backport it to 3.7 branch.

@asvetlov asvetlov merged commit 9c10806 into master Jan 27, 2020
@asvetlov asvetlov deleted the wbits branch January 27, 2020 09:34
asvetlov pushed a commit that referenced this pull request Jan 27, 2020
@socketpair @asvetlov
[3.7] 🗜 Fix deflate compression (#4506) (#4511)
0a4d64a 
(cherry picked from commit 9c10806)

Co-authored-by: Коренберг Марк <[email protected]>
asvetlov added a commit that referenced this pull request Jan 27, 2020
@asvetlov @socketpair
[3.7] 🗜 Fix deflate compression (#4506) (#4511) (#4534)
6de4222 
(cherry picked from commit 9c10806)

Co-authored-by: Коренберг Марк <[email protected]>

Co-authored-by: Коренберг Марк <[email protected]>
asvetlov pushed a commit that referenced this pull request Oct 16, 2020
@socketpair @asvetlov
🗜 Fix deflate compression (#4506) (#4511)
7931b62
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asvetlov asvetlov asvetlov approved these changes

Assignees
No one assigned
Labels
bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@socketpair @codecov-io @asvetlov