Override service endpoint URL through configuration files/environment variables #229
Comments
"I'm not dead yet." - Monty Python |
|
I would be happy to see this implemented as environment variables. We currently do all of our AWS configuration with environment variables in development. Being able to set the endpoint there would allow us to call Lambdas in local SAM and S3 buckets with local MinIO, while reverting to the official AWS endpoints in prod with no code changes. Of course, we can use a wrapper around the specific SDKs, but that isn't as nice as being able to directly import the SDK without thinking about it. |
|
Hi all, Thanks for your patience. I just linked a pull request (#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file. You can skip directly to read the proposal here. 🗒️ Providing feedbackIf you're already here and have high level feedback, we'll read and review and comments directly on this issue. If you have feedback on specific parts of the proposal, we'd appreciate if you provide feedback via the GitHub pull request by commenting on the lines where you have questions or suggestions. After reading the proposal and comments, you can give feedback on the pull request in the following ways:
Thank you! |
|
I was really surprised to see there was still no corresponding option in the aws config file equivalent to the The proposal in the previous comment (Shared configuration file) looks great! |
|
The global endpoint for all services would be a really useful addition for us. Having the ability to set a local url as the default massively reduces the risk of accidentally modifying real AWS resources when developing locally. |
|
I should be able to just set |
|
It doesn't have to be non-AWS endpoints. It is a problem of VPC endpoints that are provided by AWS for a variety of services. |
We also need to disable ssl verification as the certificate doesnt match the endpoint URL |
|
Is there any progress on that? |
|
My use case involves using private VPC endpoints for AWS S3, which will be different depending on environment. Per AWS documentation, we are required to override S3 service endpoint URL to accomplish this. |
|
This feels like a really simple problem for it to be taking >8 years to accomplish... For now I just use a bash script that I alias as #!/bin/bash
if [[ -z "${AWS_ENDPOINT_URL}" ]]; then
/opt/homebrew/bin/aws $@
else
/opt/homebrew/bin/aws --endpoint-url ${AWS_ENDPOINT_URL} $@
fi@e-compton @dingobar Maybe this at least smooths things over for you two. |
|
If I can have different profiles why not have a default endpoint for that profile? If one is given at the command line, then it could override the profile. It nags me to put it always in. |
|
What a joke. |
|
Wanted to share @kdaily's updates that this has been implemented in the AWS CLI and Python SDK Boto3:
|
|
I'm excited to announce that we have published an AWS Developer Blog post about this feature. Let us know in the feedback links on the post what you think! |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
|
What about the Java SDK? |
Asking the real question here |
Summary
It would be nice to be able to override a service endpoint URL through configuration files, environment variables, or some other runtime mechanism instead of only in code.
Description
This issue de-duplicates several issues and pull requests from other AWS SDKs and various tools, specifically:
This Stack Overflow post also describes the feature request: https://stackoverflow.com/questions/32618216/override-s3-endpoint-using-boto3-configuration-file.
First of all, we’d like to thank everyone for their feedback and patience while we examine these requests and determine how to move forward. Up front, there is an assumption in some of the threads linked above that AWS doesn’t want to implement this because it encourages reuse of the SDK for non-AWS products:
This is not the case, and we want to be clear that it is not a conflict of interest preventing the merging of that specific pull request. As @CodePint mentioned, “The functionality is already already exposed via the
endpoint_urlparameter which can be passed to the boto3 client function, the ability to configure it using an env var is purely an extension of that.” In the linked Stack Overflow post, a few commenters posted the same thing. When instantiating a service client in a recent-ish version of the SDK, you can specify the endpoint URL for that service. This feature request doesn’t change that functionality, but instead extends it through the use of a configuration file or environment variable.So again, today everyone should be able to specify an endpoint URL for a service, but it needs to be done in the service client instantiation. This feature request is to explore methods that might make this more convenient or more ergonomic.
Implementation Challenges
A handful of issues have been opened across AWS SDKs requesting the ability to customize an endpoint URL either via an environment variable or profile configuration. There are many trade-offs to consider here that make the implementation of this change more challenging than it might seem. For example, a maintainer described in this comment how:
This implementation detail is one item we need to consider (and configuration file management in general), but there are others. Before moving forward, we must consider how this feature will work across SDKs to ensure a consistent experience and maintain backwards compatibility. One important point is that we don’t want to implement this in only one of the SDKs, or only in the AWS CLI. In general, all of the AWS tools follow the same configuration file/environment variable syntax and we want to keep it that way - as @kapilt suggested, this change would need to be coordinated across the SDKs for consistency.
Next Steps
We acknowledge the demand for this feature and are actively discussing the options internally. Expect an update on this thread within a week with our progress on our decisions here. I can promise that we won’t have this done in a week - but I can also promise that by that time we’ll have had our internal discussion and be able to present some options for further discussion with the community. Anything that I can share, I will here.
We use 👍 to track community engagement on issues, so if this issue is relevant to you, please react with a 👍 emoji. Some of the linked threads above have a substantial number of reactions, which does help us gauge this feature’s important to our customers. If you’d like, you can also post more specific feedback relating to how you’d use a configuration file/environment managed endpoints and why the current method of specifying this info in the service client instantiation doesn’t work for you. We’ll take that into consideration as we design the feature.
The text was updated successfully, but these errors were encountered: