Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel trim #656

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Parallel trim #656

wants to merge 2 commits into from

Conversation

gertjanvanzwieten
Copy link
Contributor

parallelize Topology.trim

@joostvanzwieten @gertjanvanzwieten
add parallel.Manager, subset of multiproc.Manager
0ce4f1d 
With `parallel.fork` data cannot be shared between processes using shared
memory, except for arrays. `multiprocessing.Manager` supports sharing data via
pipes or sockets using pickle as serialization. This patch adds a minimal subset of
`multiprocessing.Manager` to `parallel`, supporting shared lists and dicts. If
parallel mode is disabled, a fake manager is returned which creates regular
lists and dicts, eliminating overhead of serialization, transfering and
deserialization.
@joostvanzwieten @gertjanvanzwieten
parallelize Topology.trim
e20a066 
`Topology.trim` can be quite slow if the levelset is expensive and maxrefine is large.
Although the work is embarrassingly parallel, this was not exploited,
because we can't make a shared memory array of element.Reference objects. With
the newly added `parallel.Manager` we have an alternative way of sharing data
between processes: using pickle and pipes. This patch adds support for parallel
trimming using the `parallel.Manager.list`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gertjanvanzwieten @joostvanzwieten