Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImageStack multiprocessing problematic on Windows #834

Closed
ttung opened this issue Dec 7, 2018 · 0 comments · Fixed by #1544
Closed

ImageStack multiprocessing problematic on Windows #834

ttung opened this issue Dec 7, 2018 · 0 comments · Fixed by #1544
Labels
research issues/questions opened for research purposes
Milestone
0.2.0

Comments

@ttung
Copy link
Collaborator

ttung commented Dec 7, 2018 •
edited
Loading

On Windows, multiprocessing is accomplished through CreateProcess, which is more like an exec. If the func method of ImageStack depends on any in-process state (i.e., state that is accessible through globals instead of values serialized across the process boundary), then it will not be present in child processes. See https://bugs.python.org/issue8713 for more details.

Possible solutions:

  1. Ban the use of global state. This is not possible to do programmatically. Instead, we will have to use tests to verify that we don't violate this rule.
  2. Require Windows users to use WSL. POSIX fork() is supported in this mode.
@ttung ttung mentioned this issue Dec 7, 2018
Closed
@ttung ttung mentioned this issue Dec 15, 2018
Merged
@neuromusic neuromusic added this to the 0.1.0 milestone Dec 19, 2018
@neuromusic neuromusic removed this from the 0.2.0 milestone May 3, 2019
@shanaxel42 shanaxel42 added this to the 0.2.0 milestone May 7, 2019
@shanaxel42 shanaxel42 added the research issues/questions opened for research purposes label May 7, 2019
@neuromusic neuromusic mentioned this issue Jul 13, 2019
Closed
This was referenced Sep 9, 2019
Merged
Merged
ttung pushed a commit that referenced this issue Sep 10, 2019
[easy] use the xarray property
0878b0b 
Instead of reaching into the private _data field, use the xarray property.

Test plan: `pytest -v -n8  --ff starfish/core/image/Filter/test`

Part of #834
@ttung ttung mentioned this issue Sep 10, 2019
Merged
ttung pushed a commit that referenced this issue Sep 13, 2019
[easy] use the xarray property (#1542)
20d526a 
Instead of reaching into the private _data field, use the xarray property.

Test plan: `pytest -v -n8  --ff starfish/core/image/Filter/test`

Part of #834
ttung pushed a commit that referenced this issue Sep 18, 2019
Get rid of shared state for LocalMaxPeakFinder (#1541)
85fc872 
Shared state is fundamentally incompatible with multithreading.

1. Each tile computes its own threshold.
2. Don't share the spot coords.

Part of #834
ttung pushed a commit that referenced this issue Sep 18, 2019
Switch to python multithreading
c87706c 
smFISH notebook is ~2% faster.  Didn't benchmark anything else though.

Test plan: passes unit tests, used the -alltest magic branch name introduced in #1543
Depends on: #1541, #1542, #1543
Fixes #834
ttung pushed a commit that referenced this issue Sep 18, 2019
Switch to python multithreading (#1544)
9f64c76 
smFISH notebook is ~2% faster.  Didn't benchmark anything else though.

Test plan: passes unit tests, used the -alltest magic branch name introduced in #1543
Depends on: #1541, #1542, #1543
Fixes #834
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
research issues/questions opened for research purposes
Projects
None yet
Milestone
0.2.0
Development

Successfully merging a pull request may close this issue.

Switch to python multithreading spacetx/starfish
3 participants
@ttung @neuromusic @shanaxel42