multiprocessing + multithreading should cooperate on the number of concurrent threads #1588
Comments
ttung
pushed a commit
that referenced
this issue
Oct 10, 2019
We want to support multiprocessing + imagestack for two use cases: 1. algorithms that do not release the GIL do not run well under multithreading. 2. it allows us to parallelize across FOVs in an interesting way [1] This adds a test case that we can pass an ImageStack to a child process for processing. Note that this will not pass travis until spacetx/slicedimage#125 is landed and released. Test plan: this is the test plan Depends on #1586 [1] Ideally, we should make it such that the child workers cooperatively share the compute resources. This is not done today. See #1588 for what needs to be done next.
ttung
pushed a commit
that referenced
this issue
Oct 10, 2019
We want to support multiprocessing + imagestack for two use cases: 1. algorithms that do not release the GIL do not run well under multithreading. 2. it allows us to parallelize across FOVs in an interesting way [1] This adds a test case that we can pass an ImageStack to a child process for processing. Note that this will not pass travis until spacetx/slicedimage#125 is landed and released. Test plan: this is the test plan Depends on #1586 [1] Ideally, we should make it such that the child workers cooperatively share the compute resources. This is not done today. See #1588 for what needs to be done next.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Once we support passing ImageStacks from process -> process, we need a way to coordinate how much parallelism the individual processes kick off. Each process will see a system with N cpus and start up N threads, which won't work out well.
One quick and dirty hack is to suppress multithreading when we do multiprocessing + ImageStacks. However, if one process takes a lot longer to process than the other processes, then we waste compute at the end of the processing.
Ideally, the solution is to have a centralized semaphore that each thread needs to acquire to start. Each worker process can start up N threads, but the system as a whole can only run N threads concurrently.
The text was updated successfully, but these errors were encountered: