Accommodate separate set of credentials to access DynamoDB

[MusKaya]

Description

When using an S3 compatible storage other than AWS S3, one has a set of access and secret key for the storage and another set for the DynamoDB. In this case, we need an option to provide separate credentials for DynamoDB access.

Use Case
Concurrent writes to an S3 compatible storage that does not share the credentials as DynamoDB.

Related Issue(s)

[tustvold]

Which S3 compatible store are you using, most of them shouldn't need DynamoDb as they support conditional writes natively

[MusKaya]

Which S3 compatible store are you using, most of them shouldn't need DynamoDb as they support conditional writes natively

Hi @tustvold, I'm using Linode Object Storage (which has strong consistency).

[tustvold]

So strong consistency is a necessary but insufficient condition here, to not need locking you would need support for conditional writes. I have not been able to ascertain if it supports this.

[MusKaya]

I experimented with AWS_S3_ALLOW_UNSAFE_RENAME='true' and observed that concurrent writes overwrite the json files and the later one wins the race condition.

We need to provide the DynamoDB credentials to enable concurrent writes and the object storage & DynamoDB require separate pairs of secret & access keys.

Can we consider adding optional parameters (under DeltaTable storage_options) to take DynamoDB credentials in case they differ from S3 credentials?

[Dobatymo]

I need this also. I am using AWS S3 and DynamoDB. Both are are accessed using temporary tokens which I get by assuming a role. However S3 and dynamodb are in different regions and the tokens are scoped to the specific region. So I need to pass a separate set of credentials to both of them. I supply the S3 region with aws_region and the DynamoDB region indirectly with aws_endpoint_url_dynamodb.

Would there be another way to solve this?

[thomas-chauvet]

I've opened a PR that should solve this :)