Force plain (not gzipped) aws response on aiohttp request

[rymir]

When using aiobotocore with dynamodb, requests fail on crc32 checksum computation as soon as the response data reaches ~5KB.

It seems that this is due to aws sending data back in gzip compressed format, aiohttp automatically uncompressing it and crc32 checksum being computed on the uncompressed data bytes.
While crc32 has to be computed on compressed data in order to pass.

The same requests work as expected when using botocore, this commit forces aws not to use gzip compression.

I created this fix to show the problem and to temporarily patch the issue, but the right way would be configuring aiohttp not to uncompress the response and leave that up to botocore, so we can take advantage of the gzip compression when data >= ~5KB.

[jettify]

Thanks for report, I will take a look once I got home, next week.

On Thu, 11 Aug 2016 at 05:13 Alexej Tessaro [email protected]
wrote:

When using aiobotocore with dynamodb, requests fail on crc32 checksum
computation as soon as the response data reaches ~5KB.

It seems that this is due to aws sending data back in gzip compressed
format, aiohttp automatically uncompressing it and crc32 checksum being
computed on the uncompressed data bytes.
While crc32 has to be computed on compressed data in order to pass.

The same requests work as expected when using botocore, this commit forces
aws not to use gzip compression.

I created this fix to show the problem and to temporarily patch the issue,
but the right way would be configuring aiohttp not to uncompress the
response and leave that up to botocore, so we can take advantage of the

gzip compression when data >= ~5KB.

You can view, comment on, or merge this pull request online at:

#57
Commit Summary

• Force plain (not gzipped) aws response on aiohttp request

File Changes

• M aiobotocore/endpoint.py
  https://github.com/aio-libs/aiobotocore/pull/57/files#diff-0 (1)

Patch Links:

• https://github.com/aio-libs/aiobotocore/pull/57.patch
• https://github.com/aio-libs/aiobotocore/pull/57.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#57, or mute the thread
https://github.com/notifications/unsubscribe-auth/AANoZ-zcwB6L2dbdNUo8wNuW38Awqndqks5qeue5gaJpZM4Jh7iN
.

[rymir]

@jettify any update on this one?

[jettify]

sorry for delay I will execute my test suite for s3 today, to make sure everything is ok. Also I need help with testing, could you port at least one test from botocore that hits dynamo?

[rymir]

I'm struggling finding the tests we could port, which one of https://github.com/boto/botocore/tree/master/tests you think we should port?

[jettify]

oh I see, they do not have any. Then very simple test create/delete dummy table will work fine.

[rymir]

@jettify ok, good chance to learn some pytest!

[jettify]

@rymir I executed s3 test suite with your branch, tests are green. Lets also add comment why that header is important, just to avoid confusion later.

[rymir]

@jettify added a test and a comment

[jettify]

thanks!