Provide environment variables to override default endpoint URL

[arthurzenika]

In a world of environment variables used as configuration to switch between environments, it would be really useful if aws-cli would support a variable to switch between endpoints.

In our use case we want to use aws-cli with minio https://min.io in the dev and continuous integration environments and switching with a variable such as AWS_S3_ENDPOINT_URL would be super useful.

Right now, we have to hack our way though having some custom command line, or have some startup writing of an aws configuration file, which is not ideal.

See https://docs.min.io/docs/aws-cli-with-minio for how the "alternative" entrypoint is used.

[swetashre]

@arthurlogilab - Thank you for your post. You can use --endpoint-url as a workaround.

It seems like there is already a related issue opened in Boto3 repo. boto/boto3#2099

[arthurzenika]

@swetashre I'm sorry I did not find the issue in question (I did search for it!), could you point me the issue number ?

[arthurzenika]

I'm not sure I understand, does this mean aws-cli uses boto3 so if this is implemented in boto3 then aws-cli automatically has it ?

[swetashre]

@arthurlogilab - I meant currently you can use --endpoint-url as a workaround for cli command. I would mark this as a feature request.

[arthurzenika]

@swetashre yes it is a feature request, thanks for re-opening this issue.

[dramaticlly]

second to this feature request, I think it's important and helpful to have this additional configurable value on environment variable instead of command line parse arguments. Currently this is neither part of profile or environment variable, which means user need to append this argument every time if wanted to hit custom endpoint

Service aware custom endpoint-url override would be superb

[willfish]

Bump

[kraduk]

you could do something along the lines of

aws s3 $args cp $src $dest

whe you want a custom endpoint define args as '--endpoint-url $endpoint' otherwise leave it as null. Just as easy really

However the feature would be useful for when you are using s3 in code rather than via the cli in shell

[johansmolinski]

This should be implemented on the same level as you can provide credentials and region. Either as environment variables or in a profile. I am struggling setting up a docker based environment where local offline development goes towards localstack and production goes towards AWS. Containers running the same image, just configured via environment variables.

[rchenzheng]

I got around this using an alias in the interim in case anyone was interested

alias aws='aws --endpoint-url http://website'

https://stackoverflow.com/a/55490960/12155219

[kellertk]

Thank you for posting your feedback here, and our apologies that we’ve been thinking this over for a long time without much forward motion. There are similar requests to implement this feature in a few of the AWS SDKs and the AWS CLI, so in order to coordinate those teams - and hopefully make the discussions a little easier to follow - we’ve created a new issue in aws/aws-sdk here: aws/aws-sdk#229

[kdaily]

Hi all,

We recently added a pull request (aws/aws-sdk#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file.

You can read the proposal here.

For more information on how to give feedback, please see this comment on the aws/aws-sdk repository:

aws/aws-sdk#229 (comment)

Thanks!

[kdaily]

I'm happy to announce that the ability to configure the endpoint URL via the shared configuration file and environment variables is now available in the AWS CLI v1 and v2! You can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

To start using this feature, install the AWS CLI >=1.29.0 or >=2.13.0.

To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Look forward to a blog post demonstrating the use of this feature with the AWS CLI on the AWS Developer Tools Blog!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[lorengordon]

@kdaily Just curious, do you know if this was implemented for all SDKs? I couldn't tell from the linked SDK issue... Thanks!

[kdaily]

@lorengordon, this feature is currently available through the Python SDK, the .NET SDK, and Powershell. Other SDKs will add support in the future.

[lorengordon]

@lorengordon, this feature is currently available through the Python SDK, the .NET SDK, and Powershell. Other SDKs will add support in the future.

Appreciate it, thanks! Hoping to see it in the Go SDK, in particular!

[MProx]

@kdaily Thanks for this, it's very helpful. Just a quick question: will this environment variable work with Lambda container images? I'm trying to locally test a lambda container built with the AWS base image for python (specifically, public.ecr.aws/lambda/python:3.10). The function uses other AWS services, like Secrets Manager, and I'm also using moto docker image to mock these locally. I've tried and I can't seem to get the local lambda emulator to use the local endpoint and port instead of the real AWS one.

EDIT: I know I could just specify the endpoint when I create the boto3 client, but I don't want to do that because I want to change the URL using environment variables only. I don't want to inject extra lines into my production code in the Lambda just to make it work with local testing.

[kdaily]

I'm excited to announce that we have published an AWS Developer Blog post about this feature. Let us know in the feedback links on the post what you think!

https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/

[kdaily]

@MProx:

will this environment variable work with Lambda container images?

I added a backlog item to check on that. I would expect that it is when Lambda runtime has a version of the Python SDK that supports the feature.