[C++] Use custom url for s3 using AWS_ENDPOINT_URL

[Kotwic4]

Describe the enhancement requested

AWS_ENDPOINT_URL is now supported by the AWS for custom url (for example localhost).
More info about it docs and original github issue. It was merged into botocore in this pr.

What I can do with boto:

import os
import boto3

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

session = boto3.session.Session()
s3_client = session.client(
    service_name="s3",
)
print(s3_client.list_buckets()["Buckets"])

What I have to do in pyarrow:

import os
from pyarrow import fs

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

s3 = fs.S3FileSystem(endpoint_override=os.environ["AWS_ENDPOINT_URL"])
print(s3.get_file_info(fs.FileSelector("", recursive=False)))

What I would like to do in pyarrow:

import os
from pyarrow import fs

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

s3 = fs.S3FileSystem()
print(s3.get_file_info(fs.FileSelector("", recursive=False)))

This will allow me to use s3:// instead of creating file system

#current way
s3 = fs.S3FileSystem(endpoint_override=endpoint_url)
file = pq.ParquetFile('mybucket/my_file.parquet', filesystem=s3)

#possible future
file = pq.ParquetFile('s3://mybucket/my_file.parquet')

Component(s)

Python

[westonpace]

Arrow uses AWS' C++ SDK. From this table it appears that the C++ SDK does not yet support this feature:

Perhaps the best place to advocate for this feature would be on the AWS C++ SDK repo. If it is added there then we would pick up support for it automatically once we upgraded to the latest SDK version.

That being said, it should be possible for us to provide support for this, even if the SDK does not, in case someone wanted to create a PR.

[kou]

That being said, it should be possible for us to provide support for this, even if the SDK does not, in case someone wanted to create a PR.

This may work:

diff --git a/cpp/src/arrow/filesystem/s3fs.cc b/cpp/src/arrow/filesystem/s3fs.cc
index c57fc7f29..b0c2d973e 100644
--- a/cpp/src/arrow/filesystem/s3fs.cc
+++ b/cpp/src/arrow/filesystem/s3fs.cc
@@ -339,6 +339,7 @@ Result<S3Options> S3Options::FromUri(const Uri& uri, std::string* out_path) {
   }
 
   bool region_set = false;
+  bool endpoint_override_set = false;
   for (const auto& kv : options_map) {
     if (kv.first == "region") {
       options.region = kv.second;
@@ -347,6 +348,7 @@ Result<S3Options> S3Options::FromUri(const Uri& uri, std::string* out_path) {
       options.scheme = kv.second;
     } else if (kv.first == "endpoint_override") {
       options.endpoint_override = kv.second;
+      endpoint_override_set = true;
     } else if (kv.first == "allow_bucket_creation") {
       ARROW_ASSIGN_OR_RAISE(options.allow_bucket_creation,
                             ::arrow::internal::ParseBoolean(kv.second));
@@ -357,6 +359,12 @@ Result<S3Options> S3Options::FromUri(const Uri& uri, std::string* out_path) {
       return Status::Invalid("Unexpected query parameter in S3 URI: '", kv.first, "'");
     }
   }
+  if (!endpoint_override_set) {
+    auto endpoint = std::getenv("AWS_ENDPOINT_URL");
+    if (endpoint) {
+      options.endpoint_override = endpoint;
+    }
+  }
 
   if (!region_set && !bucket.empty() && options.endpoint_override.empty()) {
     // XXX Should we use a dedicated resolver with the given credentials?

BTW, the following will work with the current implementation:

file = pq.ParquetFile(f's3://mybucket/my_file.parquet?endpoint_override={os.environ["AWS_ENDPOINT_URL"]}')

[Kotwic4]

I think it should just be in cpp sdk then. Raised an issue there

[kou]

@adbmal Could you add take only comment here to assign this issue to you?

[adbmal]

take

[amoeba]

I think it would make sense to document this in https://arrow.apache.org/docs/cpp/env_vars.html. Happy to file an issue and submit a patch for that.

[kou]

It's a good idea! Please do it!

[adbmal]

I think it would make sense to document this in https://arrow.apache.org/docs/cpp/env_vars.html. Happy to file an issue and submit a patch for that.

@kou I think it is a minor change, no need to file an issue, here is the PR, please review.
MINOR: [Docs] update document for AWS_ENDPOINT_URL environment variable #36826