Provide environment variables to override default endpoint URL

[arthurzenika]

In a world of environment variables used as configuration to switch between environments, it would be really useful if boto3 would support an variable to switch between endpoints.

In our use case we want to use boto3 with minio https://min.io in the dev and continuous integration environments and switching with a variable such as AWS_S3_ENDPOINT_URL would be super useful.

Right now, we have to hack our way though some custom python code, or have some startup writing of an aws configuration file, which is not ideal.

See https://docs.min.io/docs/aws-cli-with-minio for how the "alternative" entrypoint is used.

[swetashre]

@arthurlogilab - Thank you for your post. I would mark this as a feature request. But for now as a workaround you can use endpoint_url during client creation.

Here is the documentation for how to use endpoint_url during client creation:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

[oleksandr]

It would be nice to have as in some cases boto3 is used as dependency and there's no way to access it's constructor - environment variables is the only way to change configuration.

[rwillmer]

I've submitted a PR to provide this endpoint_url env var.
If it doesn't get accepted or needs more work, you might find my boto3 fork gives you what you need in the meantime. https://github.com/rwillmer/boto3

[rwillmer]

As I mentioned back in Jan, I've submitted a PR back in Jan to provide a fix for this which people would like to see in core. What's the process for getting someone to review and merge?

[bradenkinard]

@kdaily This is a pretty highly requested feature with simple implementation that has been ready to merge for nearly a year #2746. Is there anything else that needs to be done to get this merged?

[serbannistor]

+1 for this feature request

[ThyrixYang]

I'm trying to use tensorboard with a private s3 service, but the constructor of boto3.client is using the default endpoint_url...

[kellertk]

Thank you for posting your feedback here, and our apologies that we’ve been thinking this over for a long time without much forward motion. There are similar requests to implement this feature in a few of the AWS SDKs and the AWS CLI, so in order to coordinate those teams - and hopefully make the discussions a little easier to follow - we’ve created a new issue in aws/aws-sdk here: aws/aws-sdk#229

[tim-finnigan]

Hi all,

We recently added a pull request (aws/aws-sdk#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file.

You can read the proposal here.

For more information on how to give feedback, please see this comment on the aws/aws-sdk repository:

aws/aws-sdk#229 (comment)

Thanks!

[kdaily]

I'm happy to announce that the ability to configure the endpoint URL via the shared configuration file and environment variables is now available in the AWS Python SDK! You can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

To start using this feature, install boto3>=1.28.0.

To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Look forward to a blog post demonstrating the use of this feature on the AWS Developer Tools Blog!