Region-specific endpoints are not used when AWS_DEFAULT_REGION is set

[philvarner]

What is the bug?

Setting AWS_DEFAULT_REGION does not send requests to the appropriate region-specific S3 endpoint, even though the documentation seems to indicate otherwise. This still works for all "commercial" regions, but fails for "opt-in" regions.

Running:

AWS_DEFAULT_REGION=me-south-1 gdalinfo /vsis3/my-bucket-in-me-south-1/myfile.txt

Sends the request to my-bucket-in-me-south-1.s3.amazonaws.com instead of my-bucket-in-me-south-1.s3.me-south-1.amazonaws.com, resulting in an error ERROR 17: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to. gdalinfo failed - unable to open '/vsis3/my-bucket-in-me-south-1/myfile.txt'. because me-south-1 is an opt-in region for which s3.amazonaws.com does not forward requests.

Ideally, this behavior would be aligned with the behavior of aws-cli and use the S3 region-specific endpoints, but I think this at least needs better documentation in the GDAL Virtual File Systems docs.

Steps to reproduce the issue

Running this command results in the following error:

$ AWS_DEFAULT_REGION=me-south-1 gdalinfo /vsis3/my-bucket-in-me-south-1/myfile.txt 
ERROR 17: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
gdalinfo failed - unable to open '/vsis3/my-bucket-in-me-south-1/myfile.txt'.

This error is because the request is sent to my-bucket-in-me-south-1.s3.amazonaws.com instead of my-bucket-in-me-south-1.s3.me-south-1.amazonaws.com, and is not forwarded because me-south-1 is an opt-in region. It seems that s3.amazonaws.com is always used, even for commercial regions.

Doing a similar command with the aws CLI without the AWS_DEFAULT_REGION begin set results in a similar error:

$ aws s3 ls s3://my-bucket-in-me-south-1/myfile.txt 

An error occurred (IllegalLocationConstraintException) when calling the ListObjectsV2 operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

Making this same call (correctly) with either AWS_DEFAULT_REGION=me-south-1 set or --region me-south-1 works, as documented and expected.

I would expect both gdalinfo and aws-cli to behave the same here, especially given that gdal VFS docs state:

AWS_DEFAULT_REGION=value: Set the AWS region to which requests should be sent.

which is not actually happening.

This does work if I explicitly set the AWS_S3_ENDPOINT to the me-south-1 endpoint like:

$ AWS_S3_ENDPOINT=s3.me-south-1.amazonaws.com gdalinfo /vsis3/my-bucket-in-me-south-1/myfile.txt

However, there's no indication in the VFS doc for that parameter that this must to be done for opt-in regions, and should be done for commercial regions if you actually want the requests to go straight there instead of being forwarded from s3.amazonaws.com:

AWS_S3_ENDPOINT=value: Defaults to s3.amazonaws.com. Allows the use of /vsis3/ with non-AWS remote object stores that use the AWS S3 protocol.

Versions and provenance

$ gdalinfo --version
GDAL 3.8.3, released 2024/01/04

MacOS, Homebrew.

Additional context

AWS documents the "commercial" vs. "opt-in" regions here: https://docs.aws.amazon.com/controltower/latest/userguide/opt-in-region-considerations.html