AWSWrangler & Transfer Acceleration on S3 buckets

[gassyelmo]

Hello community!

I'm encountering some unexpected behavior when dealing with awswrangler pandas when trying to use the list (wr.s3.list_objects) and/or copy data (wr.s3.copy_objects) methods on s3 buckets. I'm currently benchmarking different solutions to see which one would be the fastest for copying data from one s3 bucket to another ... and awswrangler seems promising. The code I'm using below does work .... yet only with S3 transfer acceleration enabled. Can anyone tell me what I'm missing? Does Awswrangler simply not work without TA enabled on the buckets? Is it due to the fact that I'm using this on my local machine and will it work differently when moved towards an EC2?

Any insights are much appreciated! And to confirm - the code is working perfectly when running this with Transfer acceleration enabled on both S3 buckets.

error :

Notice that the preceding step (s3_client.list_objects_v2(Bucket="devs3copybucketfoldertestssrc")) using the client IS working !?
I have even enabled the disabling of the TA in the profile file (fefault), to no avail.

code :

from botocore.config import Config

import awswrangler as wr
import boto3.session
import boto3 

custom_config = Config(
    s3={"use_accelerate_endpoint": False}  # Disable Transfer Acceleration
)

session = boto3.Session(
    aws_access_key_id="xxx",
    aws_secret_access_key="yyyy",
    region_name="eu-central-1"
)

srcbucketname = "devs3copybucketfoldertestssrc"
dstbucketname = "devs3copybucketfoldertestsdst"

s3_client = session.client("s3", config=custom_config)

srcfiledetails = s3_client.list_objects_v2(Bucket="devs3copybucketfoldertestssrc")
srcfilenames = ["s3://" + srcbucketname + "/" + fname["Key"] for fname in srcfiledetails['Contents']]

#This works!

objects = wr.s3.list_objects("s3://devs3copybucketfoldertestssrc/",boto3_session=session)

wr.s3.copy_objects(
    paths=srcfilenames,
    source_path="s3://devs3copybucketfoldertestssrc/tmpfiles/",
    target_path="s3://devs3copybucketfoldertestsdst/transferred/"
)

#both of the above methods do not work without Transfer Accelaration enabled!?

objects = s3_client.list_objects_v2(Bucket="devs3copybucketfoldertestsdst", Prefix="transferred")
print (objects['Contents'])

[gassyelmo]

after trying to run this on a different instance - it worked. Not sure why this is ... but it seems related to my aws profile settings or the boto client session I was using. This is strange as I configured the TA enable option multiple times on my local machine and the used profiles.