Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS credentials through metadata service or boto auto-detection #105

Merged
merged 4 commits into from
Sep 7, 2024

Conversation

StephenChan
Copy link
Member

@StephenChan StephenChan commented Sep 6, 2024

  • Metadata service (STS): This is like PR Allow using AWS metadata service when available. #104 from @michaelconnor00 with the addition of catching botocore.exceptions.NoCredentialsError, which gets raised when credentials can't be obtained this way.

  • Boto auto-detection: After having another overall look at get_s3_conn(), I noticed that the comment of

    If [spacer-config] credentials are None, it will default to using credentials in ~/.aws/credentials

    didn't actually hold, because the spacer code would raise a ConfigError if spacer-config credentials were None. I decided to just remove that check that would lead to the ConfigError. The check additionally seemed a bit misleading anyway, because it's no guarantee of having valid credentials; the boto3.resource() call doesn't actually use the credentials right away. You'll only know if they're valid credentials when the resource (or 'connection', as we currently call it) is actually used to access an S3 bucket or object for the first time.

Will take another look at this later to make sure I didn't miss something else.

…Allow boto auto-detection (using a `credentials` file, etc.) with spacer config values of `None`.

The previous code claimed to allow auto-detection with `None`, but it actually raised a ConfigError in that case instead of allowing auto-detection to happen.
@michaelconnor00
Copy link

Looks good to me.

Closes #104

This is what the modern boto API calls it, and 'connection' is a little semantically misleading when it doesn't actually 'connect' (i.e. use credentials to access anything) right away.
@StephenChan
Copy link
Member Author

Thanks. I just renamed get_s3_conn() to get_s3_resource() and added unit tests, which should be enough for this PR.

I'll address the CI failing check in another PR; looks to be related to numpy 2.x being allowed by the package requirements.

After that I'll release a new pyspacer version, and let me know if any other issues arise. For example, I don't know what it looks like when an STS-retrieved resource expires (I assume that is something that happens eventually).

@StephenChan StephenChan merged commit 2b5ff35 into main Sep 7, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants