Region setting not working

[kemege]

I'm currently using s5cmd to backup files to CTClouds (It provides S3-compitable storage, running by China Telecom).
The s5cmd binary I'm using is compiled locally using go get.

I'm copying files to a bucket on region "sh2", and this information is specified in ~/.aws/config. However, when uploading, s5cmd always tries to upload to us-east-1, thus causing a AuthorizationHeaderMalformed error. Setting env AWS_REGION or AWS_DEFAULT_REGION to "sh2" won't change anything.

I noticed that since 1.2.0, s5cmd includes a feature "automatically determines region information of destination buckets". I'm wondering if it has anything to do with this feature, or is there any way I can confirm that, since adding --log debug won't give me any additional information.

[igungor]

Hi @kemege

Have you tried with v1.1.0? Has this worked before?

[kemege]

Yes, s5cmd worked well before. I tried v1.1.0 on the same machine and files are uploaded successfully.
On another machine I am using a binary compiled with go get at Sep. 26, 2020 (between the release of v1.1.0 and v1.2.0), and everything looks fine.

I guess the reason why s5 cp fails might be in the implementation of CTClouds (since other people don't have similar problems), but I wonder if there's any way (stderr, log files, etc.) I can confirm it.
Besides, setting AWS_REGION or AWS_DEFAULT_REGION should override the region inferred by s5cmd, but this does not seem to happen.

[Contextualist]

I'm experiencing the same issue working with Scaleway's endpoint. v1.2.1 does not respect the AWS_REGION or AWS_DEFAULT_REGION settings, but v1.1.0 does.

[kemege]

I'm not familiar with the code of s5cmd, but after some searching, I think this issue might be caused by the removal of the region setting here:
2dad7e1#diff-22a91901acea81cf09b58954babe267e1a5b503dbce7c37ab0df11ebfc86b0b1L649

But I'm not sure whether adding this back will solve this issue or break the functionality of cross-region transfer support.

[kemege]

It turns out that the problem is caused by the behavior of S3 service provider.
Some S3-compatible storages do not set X-Amz-Bucket-Region header in HeadBucketRequest, so s3manager.GetBucketRegionWithClient gets an empty string. Then it's further normalized by s3.NormalizeBucketLocation and become us-east-1.

The normalization happened within s3manager.GetBucketRegionWithClient, so it might be easy to make some changes for personal use, but a general solution might not be easy to implement.