Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to use AWS classes as a custom credentials provider #22007

Closed
wants to merge 1 commit into from
Closed

Make it easier to use AWS classes as a custom credentials provider #22007

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions docs/src/main/sphinx/object-storage/legacy-s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,11 @@ rotate credentials on a regular basis without any additional work on your part.

## Custom S3 credentials provider

A custom credentials provider can be used to provide temporary credentials from
STS (using `STSSessionCredentialsProvider`), IAM role-based credentials (using
`STSAssumeRoleSessionCredentialsProvider`), or credentials for a specific use
case (e.g., bucket/user specific credentials).

You can configure a custom S3 credentials provider by setting the configuration
property `trino.s3.credentials-provider` to the fully qualified class name of
a custom AWS credentials provider implementation. The property must be set in
Expand All @@ -140,12 +145,10 @@ connector property.

The class must implement the
[AWSCredentialsProvider](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSCredentialsProvider.html)
interface and provide a two-argument constructor that takes a
`java.net.URI` and a Hadoop `org.apache.hadoop.conf.Configuration`
as arguments. A custom credentials provider can be used to provide
temporary credentials from STS (using `STSSessionCredentialsProvider`),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trino-hdfs is legacy & to be removed rather sooner than later. if this was broken and didn't work, maybe no need to fix it? Just remove incorrect stuff from the docs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It worked, just was hard to use. This feature is missing in new native fs library, so I'd also like to discuss if we want to add it there. Being able to choose a single credentials provider instead of relying on the (default) chain solves some issues.

How soon are we going to remove this, weeks or months?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd hope it's sooner than later (so weeks rather than months).
I am not against having this feature (done correctly) in the new native fs library, if it solves problems. Would it be possible to have an issue describing those problems?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the issue linked in the PR description

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The linked issue is sufficient to me to justify adding the configuration to the new s3 filesystem. @nineinchnick do you plan on working on this?

IAM role-based credentials (using `STSAssumeRoleSessionCredentialsProvider`),
or credentials for a specific use case (e.g., bucket/user specific credentials).
interface and provide one of the following:
* A two-argument constructor that takes a `java.net.URI` and a Hadoop
`org.apache.hadoop.conf.Configuration` as arguments.
* A constructor without any arguments.

(hive-s3-security-mapping)=

Expand Down
23 changes: 16 additions & 7 deletions lib/trino-hdfs/src/main/java/io/trino/hdfs/s3/TrinoS3FileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1197,17 +1197,26 @@ private AWSCredentialsProvider createAwsCredentialsProvider(URI uri, Configurati
return provider;
}

private static AWSCredentialsProvider getCustomAWSCredentialsProvider(URI uri, Configuration conf, String providerClass)
private static AWSCredentialsProvider getCustomAWSCredentialsProvider(URI uri, Configuration conf, String providerClassName)
{
try {
log.debug("Using AWS credential provider %s for URI %s", providerClass, uri);
return conf.getClassByName(providerClass)
.asSubclass(AWSCredentialsProvider.class)
.getConstructor(URI.class, Configuration.class)
.newInstance(uri, conf);
log.debug("Using AWS credential provider %s for URI %s", providerClassName, uri);

Class<? extends AWSCredentialsProvider> providerClass = conf.getClassByName(providerClassName)
.asSubclass(AWSCredentialsProvider.class);
try {
return providerClass
.getConstructor(URI.class, Configuration.class)
.newInstance(uri, conf);
}
catch (NoSuchMethodException e) {
return providerClass
.getConstructor()
.newInstance();
}
}
catch (ReflectiveOperationException e) {
throw new RuntimeException(format("Error creating an instance of %s for URI %s", providerClass, uri), e);
throw new RuntimeException(format("Error creating an instance of %s for URI %s", providerClassName, uri), e);
}
}

Expand Down
9 changes: 9 additions & 0 deletions lib/trino-hdfs/src/test/java/io/trino/hdfs/s3/TestTrinoS3FileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3EncryptionClient;
import com.amazonaws.services.s3.S3ClientOptions;
Expand Down Expand Up @@ -573,11 +574,19 @@ public void testCustomCredentialsProvider()
throws Exception
{
Configuration config = new Configuration(false);
// use a class which constructor takes a URI and a Configuration object
config.set(S3_CREDENTIALS_PROVIDER, TestCredentialsProvider.class.getName());
try (TrinoS3FileSystem fs = new TrinoS3FileSystem()) {
fs.initialize(new URI("s3n://test-bucket/"), config);
assertInstanceOf(getAwsCredentialsProvider(fs), TestCredentialsProvider.class);
}

// use a class which constructor doesn't take any arguments
config.set(S3_CREDENTIALS_PROVIDER, WebIdentityTokenCredentialsProvider.class.getName());
try (TrinoS3FileSystem fs = new TrinoS3FileSystem()) {
fs.initialize(new URI("s3n://test-bucket/"), config);
assertInstanceOf(getAwsCredentialsProvider(fs), WebIdentityTokenCredentialsProvider.class);
}
}

@Test
Expand Down