Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Federated Amazon S3 support for IGV desktop, migration towards Java11, Gradle and better release engineering #620

Merged
merged 143 commits into from
Sep 10, 2019
Merged

Federated Amazon S3 support for IGV desktop, migration towards Java11, Gradle and better release engineering #620

Show file tree
Hide file tree
Changes from 140 commits
Commits
Show all changes
143 commits
Select commit Hold shift + click to select a range
73005fa
First successful oauth flow with AWS Cognito+Google social login
brainstorm Jan 13, 2019
9a3b67f
Working AWS Cognito+Google federated example. Now we should parse the…
brainstorm Jan 15, 2019
8e12acb
Add cognito Oauth flow logic and JWT helper functions
brainstorm Jan 16, 2019
b112c58
First AWS S3 listing through cognito authenticated user using STS
brainstorm Jan 16, 2019
953bbe0
Bump htsjdk version to make sure all S3 bugs are addressed. Now it ge…
brainstorm Jan 17, 2019
f5a948d
Polish gradle dependencies, we do not want to pull all of AWS jdk. Re…
brainstorm Jan 21, 2019
2489e26
Kill custom getAuthPage function and fix scope for AWS while keeping …
brainstorm Jan 21, 2019
8ebedae
Drafting the UI event mechanism via GenericLoadDialog inspired on the…
brainstorm Jan 22, 2019
6b5a758
First working version of S3 file browser with JTree. Fix Oauth state …
brainstorm Jan 30, 2019
9d5d8a5
Finally working for .bam files, thanks @reisingerf for all the UI hel…
brainstorm Jan 30, 2019
5b2122c
Merge branch 'master' into aws_support
brainstorm Jan 30, 2019
e815af1
Remove stray AWS test constant. Run gradle build from .travis.yml to …
brainstorm Jan 30, 2019
2d461c2
Java8 seems to build fine, apply the same dependencies/repository dir…
brainstorm Jan 30, 2019
a19ae27
Drop htsjdk from Gradle, @igvteam seems to prefer .jars under /lib in…
brainstorm Jan 30, 2019
d57e327
Factor out a bit Amazon/Google-specific bits from Oauth via getters/s…
brainstorm Jan 31, 2019
170dfcf
Refactor the actionListener for the dynamic S3 JTree loader Swing dia…
brainstorm Jan 31, 2019
f024b0c
Narrowed outstanding XXX (TODOs) to just ResourceLocator ones. Catch …
brainstorm Jan 31, 2019
e288214
Fix more Oauth2 profile defects. Add logged in message in statusbar. …
brainstorm Feb 1, 2019
069ff8b
Need to rethink/refactor the whole token refresh logic. Disabling cro…
brainstorm Feb 1, 2019
2b2e4cf
Fix remaining oustanding XXX from the resourcelocator and manually au…
brainstorm Feb 6, 2019
8bb2d77
Correct defects on oauth-config.json handling of optional vs mandator…
brainstorm Feb 6, 2019
17f2e3c
Clarify current OAuth config parsing logic, does not support multi-pr…
brainstorm Feb 7, 2019
ffd905e
Ignore IDEA .iml files [ci skip]
brainstorm Feb 7, 2019
7b9d2f3
GoogleUtils compiles fine locally, temporarily invalidating travisCI …
brainstorm Feb 7, 2019
73f3645
Invalidate TravisCI cache explicitly: https://docs.travis-ci.com/user…
brainstorm Feb 7, 2019
d5a6faf
Only build on script, gradle assemble first on install
brainstorm Feb 7, 2019
b7b504a
Line 326 on ResourceLocator is pointing to an old asset there seems t…
brainstorm Feb 7, 2019
d7b795a
TravisCI docs misleading about ./gradlew or gradlew
brainstorm Feb 7, 2019
d634ff7
travisci sudo should bypass the cache, according to docs: https://doc…
brainstorm Feb 7, 2019
49dd392
Actually validate that travisci cache is the issue. All those commits…
brainstorm Feb 7, 2019
d04a85b
Not a TravisCI artifact but IDEA editor perhaps?
brainstorm Feb 7, 2019
85443d1
Local build for umccr-specific releases
brainstorm Feb 7, 2019
993b7d3
No need to assemble/check, build does it all, including tests
brainstorm Feb 7, 2019
422b7a6
Setup TravisCI-triggered GitHub releases for IGV-UMCCR
brainstorm Feb 7, 2019
cf905e3
Build release artifacts and push them to releases on TravisCI pass
brainstorm Feb 7, 2019
693e8bc
Activate globbing for release objects
brainstorm Feb 7, 2019
a9e9ab3
Avoid duplicated releases
brainstorm Feb 7, 2019
82743a5
Reduce release tag verbosity [ci skip]
brainstorm Feb 7, 2019
27b9cca
Working AWS Cognito+Google federated example. Now we should parse the…
brainstorm Jan 15, 2019
363386f
First AWS S3 listing through cognito authenticated user using STS
brainstorm Jan 16, 2019
67faf8c
Adding event so that UI reacts with an AWS S3 dataset selection box a…
brainstorm Jan 21, 2019
3b50fff
Refactor the actionListener for the dynamic S3 JTree loader Swing dia…
brainstorm Jan 31, 2019
6495b45
Fix more Oauth2 profile defects. Add logged in message in statusbar. …
brainstorm Feb 1, 2019
3104bb2
Correct defects on oauth-config.json handling of optional vs mandator…
brainstorm Feb 6, 2019
2173ed6
Add working multi-jdk travisCI. Please @jrobinso @davideby consider t…
brainstorm Feb 7, 2019
2b9a439
Apply deploy to right umccr branch
brainstorm Feb 7, 2019
adad0ac
Push new IGV UMCCR release
brainstorm Feb 7, 2019
0cc9e9f
On deployment tags and this flag are mutually exclusive: https://docs…
brainstorm Feb 7, 2019
3eaf62d
Clarify current OAuth config parsing logic, does not support multi-pr…
brainstorm Feb 7, 2019
5137d7d
Merge branch 'master' into aws_support
brainstorm Feb 9, 2019
dcfffe7
Missing GoogleUtils import in ResourceLocator.
brainstorm Feb 9, 2019
156538e
Adjust .travis.yml for both jdk8 and 11
brainstorm Feb 9, 2019
0379a5d
Remove deploy objects since there is no interest in implementing an a…
brainstorm Feb 9, 2019
bad0c32
Re-triggering travisci build since there seems to be just a socket ti…
brainstorm Feb 10, 2019
f461531
Merge branch 'master' into aws_support
brainstorm Feb 22, 2019
acddacc
Migrate to Java11, therefore incorporating AWS SDK for Java V2, which…
brainstorm Mar 2, 2019
77cd0df
Fix last warning, everything compiles but when trying to run the MacA…
brainstorm Mar 2, 2019
0214aa1
Exclude groups that conflict with java boot layer testing. Thanks @re…
brainstorm Mar 4, 2019
dff774f
Override old slf4j-api implementation with newer version, just 1 fail…
brainstorm Mar 4, 2019
30e616c
Fish out example bam/bai/csi files from https://github.com/igvteam/ig…
brainstorm Mar 4, 2019
6765e6e
Update travisCI targets with new IGV Java11 namings
brainstorm Mar 4, 2019
5487bc8
No longer build for Java8, changes are backwards incompatible
brainstorm Mar 4, 2019
119246f
Bump up Gradle version, remove Java 10 limits (Java 11 bytecode is no…
brainstorm Mar 4, 2019
cac5aba
https://github.com/igvteam/igv/commit/8808982b4238614f49c8dca31d6bd4a…
brainstorm Mar 4, 2019
51e7f32
Print full trace when tests fail on CI
brainstorm Mar 4, 2019
84ff254
Merge branch 'master' into aws_support
brainstorm Mar 4, 2019
917f8b0
Add jar from upstream igv to debug/bisect this regression?: https://s…
brainstorm Mar 4, 2019
21cf7f1
Add correct JAR.
brainstorm Mar 4, 2019
022df72
Ignoring Goby testcase for now. Test-data needs to be regenerated for…
brainstorm Mar 7, 2019
84146ba
AWS presigned urls working for V2 AWS JAVA SDK with session-bound exp…
brainstorm Mar 12, 2019
9c1bdda
Exclude jsap and htsjdk without version since the latter is under /li…
brainstorm Mar 12, 2019
a7ec494
HTSJDK 2.19.0 released, solving the CSI read issues/tests. Deprecate …
brainstorm Mar 18, 2019
ad7e047
Attempt to cut down on dependencies required/pulled by AWS SDK, follo…
brainstorm Apr 1, 2019
cfa7477
org.apache.log4j deprecation in favour of org.apache.logging.log4j (n…
brainstorm Apr 1, 2019
1d28991
Ignore test files
brainstorm Apr 1, 2019
a9798f4
Auto stash before rebase of "refs/heads/umccr"
brainstorm Apr 1, 2019
9037510
Add back ONE_WEEK from GenomeManager, accidental discard during rebase
brainstorm Apr 1, 2019
f30bc36
Fix SVG export by using newest batik dependencies instead of legacy o…
brainstorm Apr 3, 2019
c9e9d7d
Add TODO/clarification, users need an 'empty' ~/.aws/credentials to h…
brainstorm May 7, 2019
20879ba
Fix batik-script RhinoInterpreterFactory error by excluding it via Ma…
brainstorm Jul 8, 2019
bac273f
Solve conflicts and merge with upstream changes and improvements, inc…
brainstorm Jul 8, 2019
862d60d
Merge branch 'master' into aws_support
brainstorm Jul 9, 2019
42c83a9
LogManager instead of old logger, next up: filtering of buckets to be…
brainstorm Jul 10, 2019
40c1f4f
Put back unused imports, rearrangements and optimizations in favour o…
brainstorm Jul 12, 2019
471d3cc
Only test with java11 for now
brainstorm Jul 12, 2019
b8b821e
Merge branch 'master' into aws_support
brainstorm Jul 12, 2019
6049ae1
Revert log4j related changes for more 'germane goodness'. Match log4j…
brainstorm Jul 15, 2019
3a5c65c
Merge w/ upstream after rolling back log4j changes
brainstorm Jul 15, 2019
dc4626c
Second pass of jim-germane-ification, adding/removing unused imports …
brainstorm Jul 15, 2019
2fb151b
IGV Version 2.4 check makes the testsuite fail, wrap it inside try/catch
brainstorm Jul 15, 2019
722111e
Minor (hopefully last) dead code, imports and newlines, germane-ifica…
brainstorm Jul 15, 2019
d1ac21d
Allow opening preferences panel when IGV version is not valid.
brainstorm Jul 15, 2019
e96cc65
Put back proguard
brainstorm Jul 18, 2019
06a5c4b
Fix session loading
reisingerf Jul 19, 2019
b05f887
Merge branch 'oauth_scopes_session' of github.com:umccr/igv into grad…
brainstorm Jul 22, 2019
cebad0e
Merge oauth_session_fix codereview and fixes by @reisingerf, thanks a…
brainstorm Jul 22, 2019
c96d119
Bypass need to have local .aws/credentials file
brainstorm Jul 22, 2019
a5d893b
Use anonymouscredprovider instead
brainstorm Jul 22, 2019
3de13b5
Fix the negative presign URL issue on newer AWS-JAVA-SDK-V2 that is >…
brainstorm Jul 22, 2019
0649637
Flo random updates/tests
reisingerf Jul 23, 2019
b74ddf5
Remove S3Presigner debug post negative expiration bug
brainstorm Jul 23, 2019
9df1eaa
Fix session/token refresh issues by passing s3:// urls to the Resourc…
brainstorm Jul 23, 2019
abf5d6b
Rollback about.properties traceback.
brainstorm Jul 23, 2019
8a51f3e
Ongoing proguard/gradle refactoring.
brainstorm Jul 23, 2019
80c90b4
General cleanup. State clearly that we should never persist refresh t…
brainstorm Jul 23, 2019
92d9195
The new access keys from the refresh token gives us more time to re-p…
brainstorm Jul 25, 2019
d98729b
Cache/process S3 presign requests with new access_tokens and refreshe…
brainstorm Jul 28, 2019
9ea35a1
Finally fixed the S3 re-presigning bug. Culprit was comparing X-Amz-D…
brainstorm Jul 29, 2019
c42caa1
Start work with VCF checkReader() method which renews pre-signed urls…
brainstorm Jul 29, 2019
27df0ba
List only buckets we can read from
reisingerf Jul 29, 2019
b03395c
Fix VCF reader caching issue
reisingerf Jul 29, 2019
0bea1e3
Upgrade to Gradle 5.5.1 since some 'too old' errors were blocking pro…
brainstorm Jul 29, 2019
66235d9
Merge branch 'gradle_cleanup' of github.com:umccr/igv into gradle_cle…
brainstorm Jul 29, 2019
bc9af56
Simplified gradle tasks by using more of Gradles convention over conf…
brainstorm Jul 30, 2019
7ae1ad3
Avoid easy warning when running tests: warning: non-varargs call of v…
brainstorm Jul 30, 2019
dbed1eb
All targets seem to be working and Gradle cleanup is mostly done, nex…
brainstorm Jul 30, 2019
4e38cf2
Avoid cache invalidation
brainstorm Jul 30, 2019
ba81fe4
Debug what is going on on travis, all tests pass locally
brainstorm Jul 30, 2019
960a350
Auto-accept buildScans on TravisCI
brainstorm Jul 30, 2019
02dfd1d
Gradle docs apparently do not apply, going the UNIX way with /bin/yes
brainstorm Jul 30, 2019
e5cebbf
Pass on proguard for Java11. META-INF versions clash with it
brainstorm Jul 30, 2019
214cd31
Switch to --debug instead of --scan
brainstorm Jul 30, 2019
88ece89
Do not instantiate AmazonUtils class during testing just to fetch the…
brainstorm Jul 30, 2019
4ca7562
Do not persist refresh token [ci skip]
brainstorm Jul 30, 2019
39b4167
Check buckets for read permission in parallel
reisingerf Jul 30, 2019
87fd78f
Revert arbitrarily short expiry times for debugging purposes
brainstorm Jul 30, 2019
89c8b1e
Merge branch 'gradle_cleanup' of github.com:umccr/igv into gradle_cle…
brainstorm Jul 30, 2019
edb15fd
Some log.info statements should be log.debug instead [ci skip]
brainstorm Jul 30, 2019
edf3d75
Fix duplication bug in S3 load dialog
reisingerf Jul 30, 2019
d97a934
Add load on double-click support
reisingerf Jul 30, 2019
fa866e1
Do not stop listening after the first command to the socket, otherwis…
brainstorm Jul 31, 2019
0b726f0
This is handled now on the AWSUtils and underlying reader logic
brainstorm Jul 31, 2019
59d1145
Disable verbose Gradle debug output on travis. Add pointer to AWS bac…
brainstorm Aug 2, 2019
9dc2cf9
Small changes that might make the code review more 'germane'-friendly
brainstorm Aug 2, 2019
6389a6c
Small clarification, remove XXX and small germane-ification
brainstorm Aug 2, 2019
80aee85
Merge branch 'master' into aws_support
jrobinso Aug 2, 2019
9fad72b
If indexExists, useIndex must be true? /cc @jrobinso @reisingerf
brainstorm Aug 2, 2019
3a7caf2
Merge branch 'master' into aws_support
jrobinso Aug 23, 2019
6c42dee
Apply changes suggested by @jrobinso first code review. Along with us…
brainstorm Aug 26, 2019
d5c3327
Bundle build corrections
davideby Aug 26, 2019
257fd3d
Only add the Amazon menu when a suitable oauth-config.json file is pr…
brainstorm Aug 28, 2019
a25bb4a
disable s3 load menu until logged in
jrobinso Sep 10, 2019
b38249d
use cognito:username if username is missing
jrobinso Sep 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
# Created by .gitignore support plugin (hsz.mobi)
.idea/workspace.xml
*.iml
test/data/out/
/build
/build_java11
/bin
igv.log
*.log
/.gradle
.project
.classpath
/.idea/
/.settings
/proguard_log
/proguard_log
.vscode
out/
.DS_Store
*.bedgraph
*.bam
*.csi.gz
*.sai
9 changes: 2 additions & 7 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,10 @@
language: java
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
- rm -fr $HOME/.gradle/caches/*/plugin-resolution/
cache:
directories:
- $HOME/.gradle/caches/
- $HOME/.gradle/wrapper/
jdk: openjdk11
install:
- ./gradlew -b build_java11.gradle assemble
- ./gradlew assemble
script:
- ./gradlew -b build_java11.gradle check
- jdk_switcher use openjdk8
- ./gradlew compileJava
- ./gradlew check
12 changes: 4 additions & 8 deletions IGV.iml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<module external.linked.project.id="igv" external.linked.project.path="$MODULE_DIR$" external.root.project.path="$MODULE_DIR$" external.system.id="GRADLE" external.system.module.group="" external.system.module.version="user" type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/out/test" isTestSource="true" />
<sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/test/src" isTestSource="true" />
<excludeFolder url="file://$MODULE_DIR$/.gradle" />
<excludeFolder url="file://$MODULE_DIR$/build" />
<excludeFolder url="file://$MODULE_DIR$/out" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="ojdbc6" level="project" />
<orderEntry type="library" name="lib" level="project" />
<orderEntry type="library" name="lib1" level="project" />
<orderEntry type="library" name="java" level="project" />
</component>
</module>
Loading