Dynamodb batch writer periodically failing

[JamieCressey]

I'm using the following logic to pull content from a list and insert it into DynamoDB

dynamodb = boto3.resource("dynamodb")
keys_table = dynamodb.Table("my-dynamodb-table")

with keys_table.batch_writer() as batch:
        for key in objects[tmp_id]:
            batch.put_item(Item={
                    "cluster": cluster,
                    "tmp_id": tmp_id,
                    "manifest": manifest_key,
                    "key": key,
                    "timestamp": timestamp
            })

It appears to periodically append more than the 25 item limit to the batch and thus fails with the following error:

  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 121, in __exit__
    self._flush()
  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 102, in _flush
    RequestItems={self._table_name: self._items_buffer})
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 383, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 425, in _convert_to_request_dict
    api_params, operation_model)
  File "/Library/Python/2.7/site-packages/botocore/validate.py", line 273, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter RequestItems.my-dynamodb-table, value: 26, valid range: 1-25

Package version:

Python 2.7.10
boto3==1.2.2
botocore==1.3.12

[JamieCressey]

I've upgraded to the latest version:

boto3==1.2.4
botocore==1.3.28

And I'm still seeing the following errors:

Traceback (most recent call last):
  File "recreateManifests.py", line 119, in <module>
    main()
  File "recreateManifests.py", line 103, in main
    generateManifest(cluster, gameId)
  File "recreateManifests.py", line 47, in generateManifest
    pass
  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 121, in __exit__
    self._flush()
  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 102, in _flush
    RequestItems={self._table_name: self._items_buffer})
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 301, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 398, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: 1 validation error detected: Value '{my-dynamodb-table=[com.amazonaws.dynamodb.v20120810.WriteRequest@caad9cb8, com.amazonaws.dynamodb.v20120810.WriteRequest@61e6406e, com.amazonaws.dynamodb.v20120810.WriteRequest@956712cf, com.amazonaws.dynamodb.v20120810.WriteRequest@6a4e6b2c, com.amazonaws.dynamodb.v20120810.WriteRequest@38281ca, com.amazonaws.dynamodb.v20120810.WriteRequest@e54b13b8, com.amazonaws.dynamodb.v20120810.WriteRequest@3ab3e4fa, com.amazonaws.dynamodb.v20120810.WriteRequest@5613be8c, com.amazonaws.dynamodb.v20120810.WriteRequest@88ba82f0, com.amazonaws.dynamodb.v20120810.WriteRequest@6027e6c6, com.amazonaws.dynamodb.v20120810.WriteRequest@92c736b0, com.amazonaws.dynamodb.v20120810.WriteRequest@409cc0a8, com.amazonaws.dynamodb.v20120810.WriteRequest@11446ab, com.amazonaws.dynamodb.v20120810.WriteRequest@5ea5c50a, com.amazonaws.dynamodb.v20120810.WriteRequest@f7f40f90, com.amazonaws.dynamodb.v20120810.WriteRequest@d7f05656, com.amazonaws.dynamodb.v20120810.WriteRequest@2e97b025, com.amazonaws.dynamodb.v20120810.WriteRequest@4a6c5caa, com.amazonaws.dynamodb.v20120810.WriteRequest@7cf4ac5d, com.amazonaws.dynamodb.v20120810.WriteRequest@c3c33e61, com.amazonaws.dynamodb.v20120810.WriteRequest@191f8f13, com.amazonaws.dynamodb.v20120810.WriteRequest@84b3ad6, com.amazonaws.dynamodb.v20120810.WriteRequest@f98fa632, com.amazonaws.dynamodb.v20120810.WriteRequest@a3266b3a, com.amazonaws.dynamodb.v20120810.WriteRequest@3bda38b, com.amazonaws.dynamodb.v20120810.WriteRequest@bb0327eb, com.amazonaws.dynamodb.v20120810.WriteRequest@e2367e92, com.amazonaws.dynamodb.v20120810.WriteRequest@997fb396, com.amazonaws.dynamodb.v20120810.WriteRequest@2ffe803b, com.amazonaws.dynamodb.v20120810.WriteRequest@b1d3791, com.amazonaws.dynamodb.v20120810.WriteRequest@6ba46ad3, com.amazonaws.dynamodb.v20120810.WriteRequest@d2004cd3, com.amazonaws.dynamodb.v20120810.WriteRequest@a48308aa, com.amazonaws.dynamodb.v20120810.WriteRequest@5f38c976, com.amazonaws.dynamodb.v20120810.WriteRequest@b4c4adb5, com.amazonaws.dynamodb.v20120810.WriteRequest@f7b627cf, com.amazonaws.dynamodb.v20120810.WriteRequest@8fb1f4e6, com.amazonaws.dynamodb.v20120810.WriteRequest@e66b8dd6, com.amazonaws.dynamodb.v20120810.WriteRequest@51cb5b06, com.amazonaws.dynamodb.v20120810.WriteRequest@a7271c9e, com.amazonaws.dynamodb.v20120810.WriteRequest@36ea63f1, com.amazonaws.dynamodb.v20120810.WriteRequest@c53592cc, com.amazonaws.dynamodb.v20120810.WriteRequest@dfb3fef8, com.amazonaws.dynamodb.v20120810.WriteRequest@1cc789ac, com.amazonaws.dynamodb.v20120810.WriteRequest@5176a361, com.amazonaws.dynamodb.v20120810.WriteRequest@e1f3e634, com.amazonaws.dynamodb.v20120810.WriteRequest@eecdfac2, com.amazonaws.dynamodb.v20120810.WriteRequest@3e0c3b05, com.amazonaws.dynamodb.v20120810.WriteRequest@bda16038, com.amazonaws.dynamodb.v20120810.WriteRequest@75957bb0, com.amazonaws.dynamodb.v20120810.WriteRequest@b0a01bde, com.amazonaws.dynamodb.v20120810.WriteRequest@4ce07193, com.amazonaws.dynamodb.v20120810.WriteRequest@545aef1e, com.amazonaws.dynamodb.v20120810.WriteRequest@dd80befd, com.amazonaws.dynamodb.v20120810.WriteRequest@329b99c1, com.amazonaws.dynamodb.v20120810.WriteRequest@780a9950, com.amazonaws.dynamodb.v20120810.WriteRequest@de1817ae, com.amazonaws.dynamodb.v20120810.WriteRequest@a28fc592, com.amazonaws.dynamodb.v20120810.WriteRequest@baede6, com.amazonaws.dynamodb.v20120810.WriteRequest@46e27b32, com.amazonaws.dynamodb.v20120810.WriteRequest@80241ac1, com.amazonaws.dynamodb.v20120810.WriteRequest@2bf904cb, com.amazonaws.dynamodb.v20120810.WriteRequest@b6c5bf38, com.amazonaws.dynamodb.v20120810.WriteRequest@f1499ce6, com.amazonaws.dynamodb.v20120810.WriteRequest@5753b553, com.amazonaws.dynamodb.v20120810.WriteRequest@512a9368, com.amazonaws.dynamodb.v20120810.WriteRequest@4115d89e, com.amazonaws.dynamodb.v20120810.WriteRequest@f3e13134, com.amazonaws.dynamodb.v20120810.WriteRequest@50776ec8, com.amazonaws.dynamodb.v20120810.WriteRequest@41d202e, com.amazonaws.dynamodb.v20120810.WriteRequest@583d9132, com.amazonaws.dynamodb.v20120810.WriteRequest@7a2eb5a1, com.amazonaws.dynamodb.v20120810.WriteRequest@137da8bb, com.amazonaws.dynamodb.v20120810.WriteRequest@e7040a26, com.amazonaws.dynamodb.v20120810.WriteRequest@4bd068ef, com.amazonaws.dynamodb.v20120810.WriteRequest@bf96f14f, com.amazonaws.dynamodb.v20120810.WriteRequest@57e51a3d, com.amazonaws.dynamodb.v20120810.WriteRequest@3ee19d15, com.amazonaws.dynamodb.v20120810.WriteRequest@9ea416bd, com.amazonaws.dynamodb.v20120810.WriteRequest@46ad556c, com.amazonaws.dynamodb.v20120810.WriteRequest@a6735644, com.amazonaws.dynamodb.v20120810.WriteRequest@c9707595, com.amazonaws.dynamodb.v20120810.WriteRequest@fbf398db, com.amazonaws.dynamodb.v20120810.WriteRequest@3faa6017, com.amazonaws.dynamodb.v20120810.WriteRequest@a6634f68, com.amazonaws.dynamodb.v20120810.WriteRequest@79624e50, com.amazonaws.dynamodb.v20120810.WriteRequest@84821619, com.amazonaws.dynamodb.v20120810.WriteRequest@2d2e84d1, com.amazonaws.dynamodb.v20120810.WriteRequest@9350a7d7, com.amazonaws.dynamodb.v20120810.WriteRequest@3f6321dd, com.amazonaws.dynamodb.v20120810.WriteRequest@9ea8a76c, com.amazonaws.dynamodb.v20120810.WriteRequest@958d00fc, com.amazonaws.dynamodb.v20120810.WriteRequest@826ec192, com.amazonaws.dynamodb.v20120810.WriteRequest@3eadbdea, com.amazonaws.dynamodb.v20120810.WriteRequest@93b81725, com.amazonaws.dynamodb.v20120810.WriteRequest@1c5da0f0, com.amazonaws.dynamodb.v20120810.WriteRequest@72462bc0, com.amazonaws.dynamodb.v20120810.WriteRequest@bd234df5, com.amazonaws.dynamodb.v20120810.WriteRequest@e39af82a, com.amazonaws.dynamodb.v20120810.WriteRequest@ca83fa6c]}' at 'requestItems' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 25, Member must have length greater than or equal to 1]

[JordonPhillips]

Thanks for the report! We'll take a look.

[jamesls]

I believe I see the issue. Working on a fix.

[ggtools]

I don't know if that might be included in the fix but I faced this issue when working a heavily loaded environment. In addition sending more than 25 operations in the batch I also noticed that at some point DynamoDB was refusing the batch with ProvisionedThroughputExceededException.

While this might be the responsibility of the caller during put_item it might be tricky for the call to deal with the errors during the final flush which is performed in __exit__

Does it makes sense to catch the ClientError with the error code ProvisionedThroughputExceededException and consider it as no element has been flushed?

[jamesls]

The ProvisionedThroughputExceededException is being retried (https://github.com/boto/botocore/blob/develop/botocore/data/_retry.json#L78-L94). What might be happening is that all the 10 retry attempts are being exhausted at which point we'll propagate the ProvisionedThroughputExceededException exception.

I think we may need to allow the user to override the max number of retry attempts so users can set it to a higher number as needed.