Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable parallelization #38

Merged
merged 23 commits into from
Apr 26, 2022
Merged

Enable parallelization #38

merged 23 commits into from
Apr 26, 2022

Conversation

sethaxen
Copy link
Member

This PR sets up internal usage of Transducers. The main benefits of this are to allow fine-grained control of parallel execution. With this PR both ELBO computation and multi-path Pathfinder are run with multithreading by default (when compatible with the user-provided RNG), but this can be customized by the user to be distributed, sequential, etc.\

This PR supersedes #11.

@codecov
Copy link

codecov bot commented Apr 24, 2022

Codecov Report

Merging #38 (518c3c4) into main (972e0ef) will increase coverage by 0.05%.
The diff coverage is 100.00%.

❗ Current head 518c3c4 differs from pull request most recent head 27e542a. Consider uploading reports for the commit 27e542a to get more accurate results

@@            Coverage Diff             @@
##             main      #38      +/-   ##
==========================================
+ Coverage   99.10%   99.15%   +0.05%     
==========================================
  Files          11       12       +1     
  Lines         335      357      +22     
==========================================
+ Hits          332      354      +22     
  Misses          3        3              
Impacted Files Coverage Δ
src/Pathfinder.jl 100.00% <ø> (ø)
src/elbo.jl 100.00% <100.00%> (ø)
src/multipath.jl 96.66% <100.00%> (+0.51%) ⬆️
src/mvnormal.jl 100.00% <100.00%> (ø)
src/optimize.jl 100.00% <100.00%> (ø)
src/singlepath.jl 100.00% <100.00%> (ø)
src/transducers.jl 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 972e0ef...27e542a. Read the comment docs.

@sethaxen
Copy link
Member Author

sethaxen commented Apr 26, 2022

Until JuliaFolds/Transducers.jl#521 and JuliaFolds/Transducers.jl#522 are resolved, we would need to accept type piracy and instability, so for the moment, we'll just use Transducers to parallelize multi-path Pathfinder (and leave ELBO estimation un-parallelized)

By introducing a mutating pattern in a few key places, type-stability is rescued.

@sethaxen sethaxen marked this pull request as ready for review April 26, 2022 20:22
@sethaxen sethaxen changed the title Use transducers Enable parallelization Apr 26, 2022
@sethaxen sethaxen merged commit f9f09da into main Apr 26, 2022
@sethaxen sethaxen deleted the transducers branch April 26, 2022 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant