HADOOP-19384. S3A: Add support for ProfileCredentialsProvider #7284
Conversation
This commit adds a wrapper for the AWS ProfileCredentialsProvider.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
changed the title
There was a problem hiding this comment.
This needs to be optional.
Adding something new to the chain has already caused problems and failures in the unit tests.
Because a lot of those developers have the AWS SDK installed, our test runs can accidentally pick this up when we do not intend to -hiding regression which then only surface in production. We have a hit exactly this problem in the past -and it is exactly the reason that the Yetus test runs have so many failing unit tests while you do not.
Instead then, please add it as a new class, apply the suggestions, and then document how to use it in the S3A documentation file hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/authentication.md
And as requested provide ways through the configuration to actually provide the path to the the profile file and which profile to use.
You can just ask for the software.amazonaws provider today; the S3A connector can authenticate with any in the SDK itself, unless it needs a special configuration. Picking up paths and profiles from the Configuration object is exactly the kind of configuration which justifies the effort.
On the topic of configuration
- nI would suggest the following two names
fs.s3a.auth.profile.file
fs.s3a.auth.profile.name
- If these are set then they MUST override the env vars of AWS_SHARED_CREDENTIALS_FILE, and AWS_PROFILE respectively.
Test wise: you should be able to write a unit test to attempt to load a dummy file the test setup writes to a temp dir, with the default profile and another one, returning the credentials you expect in both cases.
Finally, regarding the Itest failures, one of them looks like an intermittent test timing one. Proxy one is new. Does your test setup actually include a proxy? It would be good if you could debug this.
| @@ -1430,7 +1430,8 @@ | |||
| org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider, | |||
| org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, | |||
| software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider, | |||
| org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider | |||
| org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider, | |||
| org.apache.hadoop.fs.s3a.ProfileAWSCredentialsProvider | |||
There was a problem hiding this comment.
make this optional but document it
| import java.nio.file.Path; | ||
|
|
||
| @InterfaceAudience.Public | ||
| @InterfaceStability.Stable |
| @@ -0,0 +1,46 @@ | |||
| package org.apache.hadoop.fs.s3a; | |||
|
|
|||
| import org.apache.commons.lang3.SystemUtils; | |||
There was a problem hiding this comment.
nit: use same import ordering as other classes
|
|
||
| public ProfileAWSCredentialsProvider(Configuration conf) { | ||
| ProfileCredentialsProvider.Builder builder = ProfileCredentialsProvider.builder(); | ||
| builder.profileName("default").profileFile(ProfileFile.builder().content(getCredentialsPath()).build()); |
There was a problem hiding this comment.
profile name should be configurable
| public static final String NAME | ||
| = "org.apache.hadoop.fs.s3a.ProfileAWSCredentialsProvider"; | ||
|
|
||
| private ProfileCredentialsProvider pcp; |
| private static Path getCredentialsPath() { | ||
| String credentialsFile = SystemUtils.getEnvironmentVariable("AWS_SHARED_CREDENTIALS_FILE", null); | ||
| Path path = (credentialsFile == null) ? | ||
| FileSystems.getDefault().getPath(SystemUtils.getUserHome().getPath(),".aws","credentials") |
There was a problem hiding this comment.
this path should be configurable
| @@ -84,7 +85,8 @@ public final class CredentialProviderListFactory { | |||
| EnvironmentVariableCredentialsProvider.class, | |||
| IAMInstanceCredentialsProvider.class, | |||
| SimpleAWSCredentialsProvider.class, | |||
| TemporaryAWSCredentialsProvider.class)); | |||
| TemporaryAWSCredentialsProvider.class, | |||
| ProfileAWSCredentialsProvider.class)); | |||
There was a problem hiding this comment.
not a standard one...will need to be explicitly configured
|
|
||
| @InterfaceAudience.Public | ||
| @InterfaceStability.Stable | ||
| public class ProfileAWSCredentialsProvider implements AwsCredentialsProvider { |
There was a problem hiding this comment.
extends AbstractAWSCredentialProvider
and move into package auth
This commit adds a wrapper for the AWS
ProfileCredentialsProvider.
How was this patch tested?
The patch was tested by running the hadoop-aws integration tests with fs.s3a.aws.credentials.provider and fs.s3a.assumed.role.credentials.provider configured to only include org.apache.hadoop.fs.s3a.ProfileAWSCredentialsProvider. Buckets/endpoints used were in the us-east-1 region. 2 test failures that seem unrelated to this change have details in the JIRA.