Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrades to refinement functions #175

Merged
merged 187 commits into from
Jul 29, 2021
Merged
Changes from 1 commit
Commits
Show all changes
187 commits
Select commit Hold shift + click to select a range
932075a
Change individual boundary refinement
nickjcroucher May 24, 2021
8acd320
Change GPU shared memory management
nickjcroucher May 24, 2021
5df8ee9
Import rmm package
nickjcroucher May 24, 2021
e7767cd
Update refinement model assignment
nickjcroucher May 24, 2021
2e2ea87
Fix network weights processing
nickjcroucher May 25, 2021
fb8dce9
Fix distmat subsetting
nickjcroucher May 25, 2021
d5ab4f0
Fix list class check
nickjcroucher May 25, 2021
746f236
Load GPU packages for visualisation
nickjcroucher May 25, 2021
609249c
Load cugraph
nickjcroucher May 25, 2021
f03b3ee
Convert cupy array to list
nickjcroucher May 25, 2021
3feecf6
Fix function name
nickjcroucher May 25, 2021
5f54941
Make sequence name processing consistent
nickjcroucher May 26, 2021
300a851
Replace dots with underscores in names
nickjcroucher May 26, 2021
7f7bb88
Replace dots with underscores in names correctly
nickjcroucher May 26, 2021
91fe22a
Enable visualisation of individuall-refined clusters
nickjcroucher Jun 7, 2021
7445d4c
Simplify indiv-refine reading loop
nickjcroucher Jun 7, 2021
a0a0dad
Save each refined graph to a different file
nickjcroucher Jun 7, 2021
e3d3b3c
Enable querying with indiv-refine
nickjcroucher Jun 7, 2021
c15b6ba
Enable querying of indiv-refine models
nickjcroucher Jun 8, 2021
a05b92c
Enable output of separate networks with indiv-refine models
nickjcroucher Jun 8, 2021
a7406f3
Change processing of reference networks
nickjcroucher Jun 8, 2021
d6477a3
Fix reference expansion on query
nickjcroucher Jun 8, 2021
ac2b17a
Sote indiv_refine status correctly
nickjcroucher Jun 8, 2021
219c788
Change reference selection check
nickjcroucher Jun 9, 2021
82a9a31
Remove redundant dborder variable
nickjcroucher Jun 9, 2021
d75842c
Use new network construction functions for assignment
nickjcroucher Jun 9, 2021
4e921c6
Improve ref-query graph construction
nickjcroucher Jun 10, 2021
adfe2aa
Fix ref-query edge additions
nickjcroucher Jun 10, 2021
af827ad
Fix query testing
nickjcroucher Jun 10, 2021
2f07f0b
Check for graph weights
nickjcroucher Jun 10, 2021
e0add4c
Remove cupyx function
nickjcroucher Jun 10, 2021
9aedb5c
Change GPU flag
nickjcroucher Jun 10, 2021
e955ff0
Fix GPU network reference graph construction
nickjcroucher Jun 10, 2021
91ff53c
Edit messages when fetching network
nickjcroucher Jun 11, 2021
a5757fd
Fix selection of references with GPUs
nickjcroucher Jun 11, 2021
cde0c91
Alter GPU reference graph construction
nickjcroucher Jun 11, 2021
ce0ca1d
Update GPU graph loading
nickjcroucher Jun 11, 2021
0f07027
Fix boolean argument
nickjcroucher Jun 11, 2021
d7b6752
Update tests
nickjcroucher Jun 11, 2021
1d5f12d
Change qq_dist update
nickjcroucher Jun 11, 2021
212dec5
Fix reference file writing
nickjcroucher Jun 11, 2021
71e1b01
Enable deletion of random matches in HDF
nickjcroucher Jun 11, 2021
c8bfb6e
Fix edge weight calculation format
nickjcroucher Jun 11, 2021
c0a27a9
Attempt to resolve GPU bus error
nickjcroucher Jun 11, 2021
8492787
Remove attempt to resolve GPU bus error
nickjcroucher Jun 11, 2021
4a57971
Change processing of previous networks with cugraph
nickjcroucher Jun 13, 2021
afdde8c
Make edge tuples consistent
nickjcroucher Jun 13, 2021
d220f22
Fix GPU tests
nickjcroucher Jun 14, 2021
cc2591f
Fix GPU construction for single-edge graphs
nickjcroucher Jun 14, 2021
ed26be3
Change GPU package import
nickjcroucher Jun 15, 2021
83f3438
Update pruning of sequences on model fitting
nickjcroucher Jun 16, 2021
5647b24
Merge with updated master branch
nickjcroucher Jul 2, 2021
2261e36
Avoid system recursion limit
nickjcroucher Jul 2, 2021
69454e2
Change legend position
nickjcroucher Jul 2, 2021
1f0c78c
Update GPU tests
nickjcroucher Jul 4, 2021
0b26c31
Fix import of GPU libraries
nickjcroucher Jul 5, 2021
7fec151
Update GPU command lines
nickjcroucher Jul 5, 2021
41e7821
Update reference pick to use GPU
nickjcroucher Jul 5, 2021
53464cb
Fix function load
nickjcroucher Jul 5, 2021
e70b02a
Fix graph file name
nickjcroucher Jul 5, 2021
cf33f65
Save MST network
nickjcroucher Jul 6, 2021
52d71e0
Move MST network save
nickjcroucher Jul 6, 2021
70c14cc
Save Newick MST
nickjcroucher Jul 6, 2021
58ad996
Change file mode
nickjcroucher Jul 6, 2021
f25752d
Overwrite not append for weighted networks
nickjcroucher Jul 7, 2021
5ae2eab
Remove quotations from dendropy newick strings
nickjcroucher Jul 7, 2021
0e1f552
Remove quotations from dendropy newick strings
nickjcroucher Jul 7, 2021
abecf50
Fix weighted network error
nickjcroucher Jul 7, 2021
b88e05a
Change dense network construction
nickjcroucher Jul 7, 2021
6fb579d
Fix tuple generation
nickjcroucher Jul 7, 2021
2379cde
Fix tuple generation
nickjcroucher Jul 7, 2021
6bb281e
Fix tuple generation
nickjcroucher Jul 7, 2021
149110f
All dtype flexibility
nickjcroucher Jul 7, 2021
0124b66
Change distance processing
nickjcroucher Jul 7, 2021
0fcde0d
Fix distmat name
nickjcroucher Jul 7, 2021
1c853e6
Change network df construction
nickjcroucher Jul 7, 2021
32b24d4
Remove custom function as unnecessary
nickjcroucher Jul 7, 2021
dcf8afb
Restore function for modification
nickjcroucher Jul 7, 2021
80c763f
Remove unneeded function after benchmarking
nickjcroucher Jul 7, 2021
e8c978c
Remove function after further benchmarking
nickjcroucher Jul 7, 2021
edfe217
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 7, 2021
4c500d1
Remove function after further benchmarking
nickjcroucher Jul 7, 2021
2f9c44d
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 7, 2021
d819829
Update QC of query distances
nickjcroucher Jul 7, 2021
da348a9
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 7, 2021
3031454
Update QC routines
nickjcroucher Jul 7, 2021
c7cc993
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 7, 2021
8940556
Fix indexing of query values
nickjcroucher Jul 8, 2021
a2dd161
Make query functions consistent with arguments
nickjcroucher Jul 8, 2021
1ecaeff
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
121cdd3
Remove jaccard wrapped from queries
nickjcroucher Jul 8, 2021
8985f5b
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
c6b5cac
Change string to list
nickjcroucher Jul 8, 2021
99727bb
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
e70812d
Change to kmer list
nickjcroucher Jul 8, 2021
ede9349
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
46c1187
Switch off GPUs for Jaccard calculation
nickjcroucher Jul 8, 2021
326b183
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
401271f
Change sketchlib function name
nickjcroucher Jul 8, 2021
057a6eb
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 8, 2021
30f9853
Fix fit example plotting for queries
nickjcroucher Jul 8, 2021
346c98c
Remove jaccardDist function
nickjcroucher Jul 8, 2021
678ac8a
Use cupyx functions
nickjcroucher Jul 8, 2021
08c7a02
Update and document lineage assignment
nickjcroucher Jul 8, 2021
0dc0745
Visualisation using sparse distance matrix
nickjcroucher Jul 12, 2021
b761599
Update sparse MST arguments
nickjcroucher Jul 12, 2021
1b1b525
Fix visualisation rlist reading
nickjcroucher Jul 12, 2021
dd81208
Fix use of previous data
nickjcroucher Jul 12, 2021
b504f1f
Update command line arguments
nickjcroucher Jul 12, 2021
809d4ff
Fix function name in call
nickjcroucher Jul 12, 2021
09daef0
Fix variable names
nickjcroucher Jul 12, 2021
a75b641
Remove check of undefined variable
nickjcroucher Jul 12, 2021
69ee541
Edit undefined variables
nickjcroucher Jul 12, 2021
ab6e7fb
Change loading of previous MST
nickjcroucher Jul 12, 2021
dfb6639
Fix MST function call
nickjcroucher Jul 12, 2021
7702ba5
Change edges DF
nickjcroucher Jul 12, 2021
bb6012f
Merge branch 'refine_fix' of https://github.com/johnlees/PopPUNK into…
nickjcroucher Jul 12, 2021
0ec7fb6
Address possible bug in graph extraction
nickjcroucher Jul 12, 2021
adbcda4
Convert values to integers
nickjcroucher Jul 12, 2021
ad531ab
Fix variable name
nickjcroucher Jul 12, 2021
83305e2
Process distmat for t-sne
nickjcroucher Jul 12, 2021
fe8761a
Fix plotting for reference databases
nickjcroucher Jul 12, 2021
3241a0e
Fix plot fit isolate names
nickjcroucher Jul 12, 2021
ad8918d
Convert lists to arrays
nickjcroucher Jul 12, 2021
f00f77d
Generate networks from sparse matrices
nickjcroucher Jul 13, 2021
c017d4c
Ensure variable defined
nickjcroucher Jul 13, 2021
9f389d4
Test if rank_fit supplied
nickjcroucher Jul 13, 2021
be84f95
Fix network construction from sparse matrices
nickjcroucher Jul 13, 2021
f8dbdba
Import function
nickjcroucher Jul 13, 2021
badfba0
Ensure variable always defined
nickjcroucher Jul 13, 2021
8cce9a8
Fix network output function
nickjcroucher Jul 13, 2021
eac9eea
Only calculate MST once
nickjcroucher Jul 13, 2021
9436228
Add docstring for new function
nickjcroucher Jul 13, 2021
105db6c
Add script for comparing network properties and sequence properties
nickjcroucher Jul 13, 2021
b5644cd
Fix data frame properties
nickjcroucher Jul 13, 2021
52c4e35
Change sample ordering
nickjcroucher Jul 13, 2021
2796773
Change file name processing
nickjcroucher Jul 13, 2021
886e654
Enable sparse matrix processing
nickjcroucher Jul 13, 2021
adf7a62
Fix column ordering
nickjcroucher Jul 13, 2021
449f220
Fix column ordering
nickjcroucher Jul 13, 2021
3249f05
Change column order on loading
nickjcroucher Jul 13, 2021
a9b84a9
Change column renaming
nickjcroucher Jul 13, 2021
a4ddb97
Change taxon ordering
nickjcroucher Jul 13, 2021
6efdd84
Sort vertex order
nickjcroucher Jul 14, 2021
e988561
Remove unnecessary distance argument
nickjcroucher Jul 14, 2021
f4b2cd1
Define suffix variable
nickjcroucher Jul 15, 2021
2fd9df2
Re-enable joining of databases with lineages
nickjcroucher Jul 15, 2021
2be55dd
Remove unnecessary branch
Jul 27, 2021
5054109
Check model type before assigning different modes
Jul 27, 2021
3a5fb22
Fix file name processing
Jul 27, 2021
ec70882
Clarify conditional test
Jul 27, 2021
aeeb3dc
Fix section title
nickjcroucher Jul 27, 2021
d16b5da
Fix whitespace
nickjcroucher Jul 27, 2021
9e2ad62
Require necessary function argument
nickjcroucher Jul 27, 2021
722d9a1
Fix whitespace
nickjcroucher Jul 27, 2021
8098aab
Fix section title
nickjcroucher Jul 27, 2021
7d1bd04
Tidy up function calls
Jul 27, 2021
a8ffbd4
Remove parentheses from isolate names
Jul 27, 2021
2d70db8
Initialise weights list
Jul 27, 2021
efa5a7d
Clarify assert statement for references
Jul 27, 2021
cc0f023
Remove defaults
Jul 27, 2021
08276f5
Consolidate info scripts into new code
Jul 27, 2021
0613aa3
Update function docstring
Jul 27, 2021
3ee5683
Add info scripts
Jul 27, 2021
ffac5dc
Fix compilation issues
Jul 27, 2021
ed17e82
Fix syntax error
Jul 27, 2021
723736b
Remove default values
Jul 27, 2021
e410974
Restructure functions
Jul 27, 2021
a67de16
Bump version
Jul 27, 2021
1f0946e
Move default values
Jul 27, 2021
dc59e91
Fix model object reference
Jul 27, 2021
235f9b6
Update tuple generation function
Jul 27, 2021
90ae4df
Add missing brackets
Jul 28, 2021
90be56d
Change args name
Jul 28, 2021
dc69a47
Change network loading
Jul 28, 2021
2a524eb
Correct network file name processing
nickjcroucher Jul 28, 2021
26f2f04
Update tests for info and trees
nickjcroucher Jul 28, 2021
4af6078
Fix info test commands
nickjcroucher Jul 28, 2021
cbfcdd6
Change component analysis
nickjcroucher Jul 28, 2021
4d6c1a2
Update tuple generation functions
nickjcroucher Jul 28, 2021
0ec1bcc
Clarify model type variable names
nickjcroucher Jul 28, 2021
6f74e52
Correct tuple generation order
nickjcroucher Jul 28, 2021
6a90918
Tidy up section headings
nickjcroucher Jul 28, 2021
e1a3ffd
Update function arguments
nickjcroucher Jul 28, 2021
e81f6ff
Fix graph suffix error
nickjcroucher Jul 29, 2021
8dffc61
Correct network input file
nickjcroucher Jul 29, 2021
5facf69
Check for strand-specific databases
nickjcroucher Jul 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update GPU command lines
nickjcroucher committed Jul 5, 2021
commit 7fec15107d49ef00dcc287799f1ec6cac3a4ba94
6 changes: 3 additions & 3 deletions test/test-gpu.py
Original file line number Diff line number Diff line change
@@ -70,7 +70,7 @@
# viz
sys.stderr.write("Running visualisations (poppunk_visualise)\n")
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --microreact --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --cytoscape --network-file example_db/example_db_graph.gt --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --cytoscape --network-file example_db/example_db_graph.csv.gz --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --phandango --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --grapetree --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz_subset --microreact --include-files subset.txt --gpu-graph", shell=True, check=True)
@@ -85,11 +85,11 @@

# t-sne
sys.stderr.write("Running tsne viz\n")
subprocess.run(python_cmd + " ../poppunk_tsne-runner.py --distances example_db/example_db.dists --output example_tsne --perplexity 5 --verbosity 1 --gpu-graph", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_tsne-runner.py --distances example_db/example_db.dists --output example_tsne --perplexity 5 --verbosity 1 --use-gpu", shell=True, check=True)

# prune
sys.stderr.write("Running poppunk_prune\n")
subprocess.run(python_cmd + " ../poppunk_prune-runner.py --distances example_db/example_db.dists --ref-db example_db --remove subset.txt --output example_prune --gpu-dist", shell=True, check=True)
subprocess.run(python_cmd + " ../poppunk_prune-runner.py --distances example_db/example_db.dists --ref-db example_db --remove subset.txt --output example_prune", shell=True, check=True)

# references
sys.stderr.write("Running poppunk_references\n")