Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix UnicodeDecodeError in buffer_s3response() #2123

Merged
merged 1 commit into from
Apr 1, 2021

Conversation

sir-sigurd
Copy link
Member

@sir-sigurd sir-sigurd commented Mar 24, 2021

Description

s3_client.put_object(
    Bucket='quilt-dev-null',
    Key='test-s3-select',
    Body=b''.join(['ы'.encode() * 65 * 2 ** 9, b'\n'] * 2)
)
buffer_s3response(
    s3_client.select_object_content(
        Bucket='quilt-dev-null',
        Key='test-s3-select',
        ExpressionType='SQL',
        Expression='SELECT * FROM s3object s LIMIT 5',
        InputSerialization={'CSV': {}}, OutputSerialization={'CSV': {}}
    )
)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-22-f63d210a5ddd> in <module>
     10         ExpressionType='SQL',
     11         Expression='SELECT * FROM s3object s LIMIT 5',
---> 12         InputSerialization={'CSV': {}}, OutputSerialization={'CSV': {}}
     13     )
     14 )

<ipython-input-12-f3176c1b158b> in buffer_s3response(s3response)
     10     for event in s3response['Payload']:
     11         if 'Records' in event:
---> 12             records = event['Records']['Payload'].decode()
     13             response.write(records)
     14             found_records = True

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 64999: unexpected end of data

TODO

@codecov
Copy link

codecov bot commented Mar 24, 2021

Codecov Report

Merging #2123 (706e582) into master (5d49a4c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 706e582 differs from pull request most recent head 2bd0d66. Consider uploading reports for the commit 2bd0d66 to get more accurate results
Impacted file tree graph

@@           Coverage Diff           @@
##           master    #2123   +/-   ##
=======================================
  Coverage   46.35%   46.35%           
=======================================
  Files         429      429           
  Lines       20532    20534    +2     
  Branches     2412     2412           
=======================================
+ Hits         9517     9519    +2     
  Misses      10056    10056           
  Partials      959      959           
Flag Coverage Δ
api-python 89.71% <ø> (ø)
catalog 16.30% <ø> (ø)
lambda 93.03% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
lambdas/shared/t4_lambda_shared/utils.py 98.64% <100.00%> (ø)
lambdas/shared/tests/test_utils.py 97.29% <100.00%> (+0.07%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d49a4c...2bd0d66. Read the comment docs.

@sir-sigurd sir-sigurd force-pushed the fix-buffer-s3-select branch 3 times, most recently from d4b29fe to f8c5760 Compare March 30, 2021 16:09
@sir-sigurd sir-sigurd marked this pull request as ready for review March 30, 2021 16:09
@sir-sigurd sir-sigurd force-pushed the fix-buffer-s3-select branch from f8c5760 to d38be48 Compare March 30, 2021 16:12
@sir-sigurd sir-sigurd force-pushed the fix-buffer-s3-select branch 2 times, most recently from 706e582 to 2bd0d66 Compare April 1, 2021 13:05
@sir-sigurd sir-sigurd force-pushed the fix-buffer-s3-select branch from 2bd0d66 to a8d7a85 Compare April 1, 2021 13:06
@sir-sigurd sir-sigurd merged commit e840403 into master Apr 1, 2021
@sir-sigurd sir-sigurd deleted the fix-buffer-s3-select branch April 1, 2021 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants