Cross-Region S3 Copy with SourceClient Fails in Boto3 and AWS CLI

[DaanVanHauwermeiren]

Describe the bug

When performing a cross-region S3 copy operation using Boto3 (or the AWS CLI), the SourceClient parameter in Boto3 and the --endpoint-url parameter in the AWS CLI are not applied consistently. This results in errors when attempting to copy objects from a source bucket in one region to a destination bucket in another region.
Note that this was tested on Scaleway, not on AWS.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Using Boto3 client's copy command to copy object from one bucket (in one region) to another bucket (in another region).

Current Behavior

Expected Behavior: The object should copy successfully from the source bucket to the destination bucket across regions, using the SourceClient to correctly resolve the source bucket's region.

Actual Behaviour: an error is raised.

botocore.exceptions.ClientError: An error occurred (NoSuchBucket) when calling the CopyObject operation: The specified bucket does not exist

The copy command does not use information from the SourceClient input, and only uses the info (credentials, location, etc.) from the client on which the copy method was called.

I also tried this with the aws cli, but got the same results:

aws s3 sync s3://source-bucket s3://dest-bucket \
    --source-region fr-par \
    --region nl-ams \
    --endpoint-url https://s3.fr-par.scw.cloud \
    --profile mys3profile

The aws cli seems to fall back on an amazonaws endpoint:

fatal error: Could not connect to the endpoint URL: "https://source-bucket.s3.fr-par.amazonaws.com/?list-type=2&prefix=&encoding-type=url"

Reproduction Steps

import boto3
from dotenv import dotenv_values
config = dotenv_values(".env")

# Initialize source and destination clients
s3_session = boto3.Session(
    aws_access_key_id=config.get("SCW_ACCESS_KEY"),
    aws_secret_access_key=config.get("SCW_SECRET_KEY"),
    region_name="fr-par",
)
src_s3 = s3_session.client(
    service_name="s3",
    region_name="fr-par",
    endpoint_url="https://s3.fr-par.scw.cloud",
)
s3_session = boto3.Session(
    aws_access_key_id=config.get("SCW_ACCESS_KEY"),
    aws_secret_access_key=config.get("SCW_SECRET_KEY"),
    region_name="nl-ams",
)
dest_s3 = s3_session.client(
    service_name="s3",
    region_name="nl-ams",
    endpoint_url="https://s3.nl-ams.scw.cloud",
)

# Set up source and destination parameters
copy_source = {
    "Bucket": "source_bucket_name",
    "Key": "source_object_name",
}

# Attempt to copy with SourceClient
dest_s3.copy(
    copy_source,
    "destination_bucket_name",
    source_object_name,
    SourceClient=src_s3
)

Possible Solution

I could not get it to work using boto3, but I managed to get a solution that was acceptable to me using rclone.

Example config to be placed in ~.conf/rclone/rclone.conf:

[scw_s3_fr]
type = s3
provider = Scaleway
access_key_id = ...
secret_access_key = ...
region = fr-par
endpoint = s3.fr-par.scw.cloud
acl = private

[scw_s3_nl]
type = s3
provider = Scaleway
access_key_id = ...
secret_access_key = ...
region = nl-ams
endpoint = s3.nl-ams.scw.cloud
acl = private

sync the source to the destination one-way:

rclone sync scw_s3_fr:source-bucket scw_s3_nl:destination-bucket -P --metadata --checksum --check-first

Additional Information/Context

No response

SDK version used

boto3='1.35.66'

Environment details (OS name and version, etc.)

Python 3.11.2 (main, Mar 7 2023, 16:53:12) [GCC 12.2.1 20230201] on linux