AWS Data Transfer Hub | S3 transfer from US to China Bucket failed with error "InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records"

[rajatstg]

Hello Developer(s),
The following issue encountered in Data Transfer Hub for which I request your assistance to provide resolution:

Action Performed:

1. To create DTH service, used "(Option 1) Launch the Cloudformation stack in AWS Regions" as given in URL: https://docs.aws.amazon.com/solutions/latest/data-transfer-hub/step-1.-option-1-launch-the-stack-in-aws-regions.html.

2. Using DTH, intiated copy task for 1.6 TB of data from S3 bucket of us-west-2 (Oregon) region to S3 bucket of eu-central-1 (Frankfurt) region. Used the secret created in AWS Secret Manger (Oregon Region). The copy started successfully as shown below and stopped it manually as it is a test.

Note: W.r.t the secret used, the permissions for the IAM user (with AKSK) are inplace and also verified with AWS Support.

Timestamp Message
2025-01-27 16:45:28 net.ipv4.tcp_congestion_control = bbr
2025-01-27 16:45:28 2025/01/27 11:15:28 Start running Finder job
2025-01-27 16:45:28 2025/01/27 11:15:28 Get secret Value of XXXX-secret from Secrets Manager
2025-01-27 16:45:28 2025/01/27 11:15:28 next tick is:2025-01-28 02:00:00 +0000 UTC
2025-01-27 16:45:28 2025/01/27 11:15:28 Queue DTH-S3EC2-0012f-S3TransferQueue-W0vzgpO1ZMIq has 0 not visible message(s) and 0 visable message(s)
2025-01-27 16:45:28 2025/01/27 11:15:28 Giant object merging Step Function arn:aws:states:us-west-2:XXXXXXXXXX:stateMachine:DTH-S3EC2-0012f-MultiPart-ControllerSM has 0 not competed task(s)
2025-01-27 16:45:28 2025/01/27 11:15:28 Prefix List File:
2025-01-27 16:45:28 2025/01/27 11:15:28 List common prefixes from / with depths 0
2025-01-27 16:45:28 2025/01/27 11:15:28 prefix:
2025-01-27 16:45:28 2025/01/27 11:15:28 Comparing within prefix /
2025-01-27 16:45:29 2025/01/27 11:15:28 Scanning in destination prefix /
2025-01-27 16:45:29 2025/01/27 11:15:29 Totally 2 objects in destination prefix /
2025-01-27 16:45:29 2025/01/27 11:15:29 Start comparing and sending...
2025-01-27 16:45:29 2025/01/27 11:15:29 Completed in prefix /, found 1 batches (3 objects) in total
2025-01-27 16:45:29 2025/01/27 11:15:29 Finder Job Completed in 532.363661ms, found 3 objects in total
2025-01-27 16:45:33 Exit Finder proccess, trying to set auto scaling group desiredCapacity to 0 to terminate instance after 60 seconds...

3. Used DTH to copy same 1.6 TB of data from S3 bucket of us-west-2 (Oregon) region to S3 bucket of cn-north-1 (Beijing - China). Used the same secret as used in previous action. The copy failed with the following error message.

Note: W.r.t the secret used, the permissions for IAM user with AKSK are inplace and also verified with AWS Support just in case.

Timestamp Message
2025-01-27 16:30:10 net.ipv4.tcp_congestion_control = bbr
2025-01-27 16:30:10 2025/01/27 11:00:10 Start running Finder job
2025-01-27 16:30:10 2025/01/27 11:00:10 Get secret Value of XXXXXXX-secret from Secrets Manager
2025-01-27 16:30:10 2025/01/27 11:00:10 next tick is:2025-01-28 02:00:00 +0000 UTC
2025-01-27 16:30:10 2025/01/27 11:00:10 Queue DTH-S3EC2-52de2-S3TransferQueue-FKipHAxXurf3 has 0 not visible message(s) and 0 visable message(s)
2025-01-27 16:30:10 2025/01/27 11:00:10 Giant object merging Step Function arn:aws:states:us-west-2:XXXXXXXXXXXX:stateMachine:DTH-S3EC2-52de2-MultiPart-ControllerSM has 0 not competed task(s)
2025-01-27 16:30:10 2025/01/27 11:00:10 Prefix List File:
2025-01-27 16:30:10 2025/01/27 11:00:10 List common prefixes from / with depths 0
2025-01-27 16:30:10 2025/01/27 11:00:10 prefix:
2025-01-27 16:30:10 2025/01/27 11:00:10 Comparing within prefix /
2025-01-27 16:30:11 2025/01/27 11:00:10 Scanning in destination prefix /
2025-01-27 16:30:11 2025/01/27 11:00:11 Unable to list objects in / - operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 2FJ2PA9FK47J5FQ9, HostID: mOJ5VG1p8GlCpjqz86lofaHIq/1ZrplohROyUm6DFQ6AfBRrBFoz8fixTjYQvDis01TxwEVC6dc=, api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
2025-01-27 16:30:11 2025/01/27 11:00:11 S3> Unable to list object in / - operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 2FJ2PA9FK47J5FQ9, HostID: mOJ5VG1p8GlCpjqz86lofaHIq/1ZrplohROyUm6DFQ6AfBRrBFoz8fixTjYQvDis01TxwEVC6dc=, api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
2025-01-27 16:30:11 2025/01/27 11:00:11 Error listing objects in destination bucket - operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 2FJ2PA9FK47J5FQ9, HostID: mOJ5VG1p8GlCpjqz86lofaHIq/1ZrplohROyUm6DFQ6AfBRrBFoz8fixTjYQvDis01TxwEVC6dc=, api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
2025-01-27 16:30:15 Exit Finder proccess, trying to set auto scaling group desiredCapacity to 0 to terminate instance after 60 seconds...

Troubleshooting steps performed:

1. No bucket policy that blocks the access.

2. Relevant IAM permissions are in place.

3. Verified Secret and its format. Created secret as key/value pairs as per the below format given in the documentation.

{
"access_key_id": "",
"secret_access_key": ""
}

4. Ran traceroute from a test EC2 instance created in the VPC created by Cloudformation for DTH service.

traceroute -T -p 443 s3.eu-central-1.amazonaws.com
traceroute -T -p 443 s3.cn-north-1.amazonaws.com.cn

Output of above traceroutes are given below:

OREGON TO FRANKFURT:
[root@ip-10-0-0-xx ec2-user]# traceroute -T -p 443 s3.eu-central-1.amazonaws.com
traceroute to s3.eu-central-1.amazonaws.com (52.219.72.183), 30 hops max, 60 byte packets
1 * * *
2 240.4.228.4 (240.4.228.4) 18.147 ms 240.0.64.7 (240.0.64.7) 18.115 ms 240.0.64.6 (240.0.64.6) 18.101 ms
3 240.0.88.7 (240.0.88.7) 148.769 ms 240.0.88.5 (240.0.88.5) 150.517 ms 242.13.183.193 (242.13.183.193) 18.032 ms
4 240.0.88.18 (240.0.88.18) 148.386 ms 240.0.88.7 (240.0.88.7) 149.207 ms 240.0.88.31 (240.0.88.31) 146.910 ms
5 240.0.88.17 (240.0.88.17) 148.083 ms 240.0.88.27 (240.0.88.27) 145.504 ms 240.0.88.22 (240.0.88.22) 150.364 ms
6 240.0.88.23 (240.0.88.23) 150.396 ms 241.0.3.79 (241.0.3.79) 148.933 ms 240.0.88.31 (240.0.88.31) 148.894 ms
7 241.0.3.74 (241.0.3.74) 151.317 ms * *
8 * * *
9 * * *
10 s3.eu-central-1.amazonaws.com (52.219.72.183) 149.816 ms * 146.586 ms

OREGON TO BEIJING:
[root@ip-10-0-0-xx ec2-user]# traceroute -T -p 443 s3.cn-north-1.amazonaws.com.cn
traceroute to s3.cn-north-1.amazonaws.com.cn (54.222.52.121), 30 hops max, 60 byte packets
1 100.91.5.139 (100.91.5.139) 28.976 ms 100.91.5.135 (100.91.5.135) 29.288 ms 100.91.5.181 (100.91.5.181) 28.368 ms
2 100.100.4.65 (100.100.4.65) 29.636 ms 100.100.4.7 (100.100.4.7) 29.322 ms 100.100.4.117 (100.100.4.117) 28.295 ms
3 100.100.68.132 (100.100.68.132) 28.012 ms 100.100.92.4 (100.100.92.4) 29.472 ms 100.100.80.68 (100.100.80.68) 29.831 ms
4 100.100.64.137 (100.100.64.137) 29.549 ms 100.100.72.9 (100.100.72.9) 30.474 ms 100.100.76.73 (100.100.76.73) 28.725 ms
5 100.100.8.78 (100.100.8.78) 27.505 ms 100.100.8.68 (100.100.8.68) 30.410 ms 100.100.8.64 (100.100.8.64) 29.165 ms
6 219.158.43.5 (219.158.43.5) 29.734 ms 29.462 ms 29.113 ms
7 219.158.96.25 (219.158.96.25) 227.029 ms 227.004 ms 226.464 ms
8 219.158.10.45 (219.158.10.45) 220.282 ms 220.617 ms 220.629 ms
9 * * *
10 * * *
11 124.65.59.130 (124.65.59.130) 240.200 ms 238.251 ms 124.65.57.238 (124.65.57.238) 223.337 ms
12 bt-204-238.bta.net.cn (202.106.204.238) 237.278 ms 226.482 ms 226.363 ms
13 54.222.0.169 (54.222.0.169) 213.463 ms 54.222.0.167 (54.222.0.167) 202.532 ms 54.222.0.175 (54.222.0.175) 206.671 ms
14 * * 54.222.0.216 (54.222.0.216) 183.765 ms
15 * 54.222.0.129 (54.222.0.129) 184.433 ms *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 s3.cn-north-1.amazonaws.com.cn (54.222.52.121) 235.953 ms 220.199 ms 221.184 ms

It was observed the above traceroutes didnt use the path of Direct Connect however still able to reach both the destination.
The VPC created for DTH doesnt have interface and so Direct Connect is not in picture.
The Direct Connect is a requirement as mentioned in the URL https://docs.aws.amazon.com/solutions/latest/data-transfer-hub/solution-overview.html for Data Transfer Hub to communicate with destination.
However without Direct Connect it worked for Frankfurt region but not for China region.

Hence, need your assistance to fix this issue or let us know if we missed anything. Thank you.

[bassemwanis]

Thank you for bringing this issue to our attention. While we investigate the issue, could you please confirm that you have reviewed the details on how to Transfer S3 object via Direct Connect? Additionally, could you please refer to the troubleshooting section of the implementation guide for potential solutions?

[rajatstg]

In the URL https://docs.aws.amazon.com/solutions/latest/data-transfer-hub/transfer-s3-object-via-direct-connect.html, from the before 4 options, 2 & 3 are not feasible for us. Already tried option 1 without success as the DTH is deployed in Source bucket region which is us-west-2. However, not tried option 4 as is says "In this scenario, DTH is deployed in the destination side (China) and within a VPC with public access" which is not the case, DTH is deployed in US Oregon region.

1. Create an Amazon S3 transfer task
2. Create an Amazon ECR transfer task
3. Transfer S3 object from Alibaba Cloud OSS
4. Transfer S3 object via Direct Connect

Troubleshooting sections already referred and followed without success. Worked with AWS Support viz, S3, IAM, DX, & Solutions team. With all their confirmation I have raise an issue.

It would be great if you can connect with me via Chime or Zoom, so that I can show the DTH page and initiate a transfer for your understanding and to know if I missed anything. If yes, please email me the meeting url for a remote session during your morning hours between 9:00am to 11:30am EST.

[bassemwanis]

@rajatstg can you follow these steps and change DTH deployment to be in the destination side.