Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDBRecord throws decimal.Rounded for large numbers (over 38 digits) #4845

Closed
miikka opened this issue Jul 30, 2024 · 5 comments · Fixed by #4863
Closed

DynamoDBRecord throws decimal.Rounded for large numbers (over 38 digits) #4845

miikka opened this issue Jul 30, 2024 · 5 comments · Fixed by #4863
Assignees
Labels
bug Something isn't working event_sources Event Source Data Class utility

Comments

@miikka
Copy link

miikka commented Jul 30, 2024

Expected Behaviour

When encountering a DynamoDB Streams event with a large number in the new/old image such as {"N": "110111111111111110000000000000000000000"}, Powertools' DynamoDB Streams data classes should be able to deserialize it like any other event. That is currently not the case if the number string is longer than 38 characters.

According to DynamoDB docs, numbers may have up to 38 digits of precision. However, the trailing zeros do not count towards the precision and creating a document with the number above works fine.

(Note: if you try to create such document using AWS Console, the editor there blocks it unless you use the JSON view. The form editor seems to use a simpler check for the number validity.)

Current Behaviour

A decimal.Rounded exception gets thrown with a stacktrace like this when you try to access record.dynamodb.new_image when using DynamoDBRecord/DynamoDBStreamEvent from DynamoDB Streams data classes to process a DynamoDB Streams events:

Rounded: [<class 'decimal.Rounded'>]
  File "aws_lambda_powertools/middleware_factory/factory.py", line 135, in wrapper
    response = middleware()
  File "aws_lambda_powertools/utilities/data_classes/event_source.py", line 39, in event_source
    return handler(data_class(event), context)
  File "handler.py", line 61, in handler
    xxx
  File "utils.py", line 88, in xxx
    item = record.dynamodb.new_image
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 155, in new_image
    return self._deserialize_dynamodb_dict("NewImage")
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in _deserialize_dynamodb_dict
    return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in <dictcomp>
    return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in _deserialize_l
    return [self.deserialize(v) for v in value]
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in <listcomp>
    return [self.deserialize(v) for v in value]
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 76, in _deserialize_n
    return DYNAMODB_CONTEXT.create_decimal(value)

Code snippet

from aws_lambda_powertools.utilities.data_classes.dynamo_db_stream_event import DynamoDBRecord
data = {
  "awsRegion": "eu-central-1",
  "dynamodb": {
    "ApproximateCreationDateTime": 1722318027.0,
    "Keys": {"id": {"S": "test"}},
    "NewImage": {
      "id": {"S": "test"},
      "number": {"N": "110111111111111110000000000000000000000"},
    },
    "OldImage": {
      "id": {"S": "test"},
    }
  }
}
DynamoDBRecord(data).dynamodb.new_image

Possible Solution

No response

Steps to Reproduce

  1. Create a lambda that processes DynamoDB Streams events with the Powertools data classes or the batch processing support and call record.dynamodb.new_image for a record in the event.
  2. Create a new item in the DynamoDB table with an attribute that has type "number" and value 110111111111111110000000000000000000000

Powertools for AWS Lambda (Python) version

2.40.1

AWS Lambda function runtime

3.11

Packaging format used

Lambda Layers

Debugging logs

No response

@miikka miikka added bug Something isn't working triage Pending triage from maintainers labels Jul 30, 2024
Copy link

boring-cyborg bot commented Jul 30, 2024

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@sthulb
Copy link
Contributor

sthulb commented Jul 30, 2024

Hi @miikka I’ll look into this tomorrow. Thanks for reporting it.

@sthulb sthulb removed the triage Pending triage from maintainers label Jul 31, 2024
@sthulb
Copy link
Contributor

sthulb commented Jul 31, 2024

Hi @miikka I've prepped a PR #4863 to handle this, it'll strip off tailing 0s after the 38th char (as per the docs).

Let me know if this would be sufficient.

@leandrodamascena leandrodamascena moved this from Triage to Working on it in Powertools for AWS Lambda (Python) Aug 1, 2024
@leandrodamascena leandrodamascena added the event_sources Event Source Data Class utility label Aug 1, 2024
@github-project-automation github-project-automation bot moved this from Working on it to Coming soon in Powertools for AWS Lambda (Python) Aug 2, 2024
Copy link
Contributor

github-actions bot commented Aug 2, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

@github-actions github-actions bot added the pending-release Fix or implementation already in dev waiting to be released label Aug 2, 2024
Copy link
Contributor

github-actions bot commented Aug 8, 2024

This is now released under 2.43.0 version!

@github-actions github-actions bot removed the pending-release Fix or implementation already in dev waiting to be released label Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working event_sources Event Source Data Class utility
Projects
Status: Coming soon
3 participants