-
-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiprocessing support #2815
Multiprocessing support #2815
Conversation
the underlying issue here is that store operations don't work in child processes. |
I added some cleanup logic tied to |
@martindurant this seems like your wheelhouse -- would you mind checking this over? |
This is absolutely what you must do. |
(I should have added locks to the list of things to watch out for) |
I think this PR is ready for review. The test I added checks that array indexing works in a multiprocessing context, which is a fine test, but the actual substance is whether our asynchronous store methods work with multiprocessing, and that can be tested without arrays. But I'd rather treat that as a test refactor for the future. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm OK keeping these, effectively, as integration tests rather than store-specific. The top-level sync() we call internally is important here.
Ensure that global resources are reset after a fork. Without this function, | ||
forked processes will retain invalid references to the parent process's resources. | ||
""" | ||
global loop, iothread, _executor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
strictly speaking, loop and iothread don't need to be global, since they are mutated in-place.
pytest.param( | ||
"fork", | ||
marks=pytest.mark.skipif( | ||
sys.platform in ("win32", "darwin"), reason="fork not supported on Windows or OSX" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because windows should only perform spawn? This decorator is a bit verbose, it would be OK to put if ... : pytest.skip()
in the body of the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I prefer (verbose) parametrization over checking the OS in the test itself. The former gives better feedback in the test summary.
The "indirect changes" tab is picking up many lines in testing/stateful.py . I don't know why they show up here - maybe this file wasn't being tracked at all before? |
OK, it looks like not all of the coverage reports were in yet :) |
thanks for your help martin! |
I wanted to use this PR to add tests to ensure that basic array operations work in a multiprocessing context. But right now I just have 1 test, and it's failing with an infinite stall. Since this breaks the entire test suite, we should also fix the underlying bug in this PR. Stay tuned!
closes #2812