Load AWS S3 files from the SDK

[jrobinso]

From htslib mailing list

"Samtools/htslib will read the usual environment variables (AWS_ACCESS_KEY_ID, etc) and configuration files (~/.aws/credentials, ~/.s3cfg, ~/.awssecret) to find your credentials to use in authenticating to AWS S3."

[bbreton]

Supporting this would save me a large amount of pain. I'm currently using a proxy to access private s3, but it isn't an ideal setup. As an implementation note being able to use the same format as the s3 cli uses i.e. s3://bucket/folder/file.bam like htslib does would be easiest for me.

For anyone trying to do this absent this feature being added https://github.com/nkrumm/s3proxy works and is really simple though there are better/more feature complete s3 proxies out there they are harder to setup.

[jrobinso]

Could you elaborate on what a "credential" file is, or point me to documentation? Most people using AWS with IGV use signed URLs.

…

On Wed, Jan 17, 2018 at 11:35 AM, Ben Breton ***@***.***> wrote: Supporting this would save me a very large amount of pain. I'm currently using a proxy to access private s3, but it isn't an ideal setup. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#320 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA49HNdME5-THfKqXYwHLYS991Vq4Qkzks5tLkqjgaJpZM4KH4mC> .

[bbreton]

The aws doco is probably best for understanding it:
https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html

If you implement accessing s3 with the aws java sdk should allow you to use them as accessing them is an integral part of the library.

I'm not sure how realistic implementing this is for IGV's development environment. If you can't use the AWS provided SDK and even if you can it might be quite a bit of effort for something that has multiple other solutions (proxies and signed urls).

There might be other options as well to integrate it. I believe htslib added it because one of their underlying libraries for accessing remote files started supporting it so it required minimal effort from the developers to add.

I have contemplated doing signed urls, but it is somewhat of a pain to do and at least for my personal development use the s3 proxy was easier than implementing them.

AWS java s3 sdk and the s3 section of the doco.
https://aws.amazon.com/sdk-for-java/
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html

[jrobinso]

Placing this link here for now, I will return to it. Its not clear how much effort is involved, the most difficult thing will probably be creating some kind of browser interface for a user to select their file ("object"). Perhaps in the first version we can require the user know and enter their bucket name as a string.

https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html

[brainstorm]

Safe to close since #620 has been merged @jrobinso ?

@bbreton AWS is now supported on IGV desktop, check out the docs ;)

[jrobinso]

Yes, thanks to @brainstorm @reisingerf and others at UMCCR for adding support for this. Currently this is only available in the snapshot build but we will release it in the next few days.

[bbreton]

Thank you very much. I appreciate all the devs hard work!