Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for the issue #3646. #3687

Merged
merged 1 commit into from
Jul 4, 2023

Conversation

dmitry-ganyushin
Copy link
Contributor

It looks like the default for macos and python 3.8+ "spawn" method does not work on macos, swith to "fork" for all situations. https://bugs.python.org/issue33725. Rebased on the current master.

… and python 3.8+ "spawn" method does not work on macos, swith to "fork" for all situations. https://bugs.python.org/issue33725. Rebased on the current master.
@eisenhauer
Copy link
Member

"Fix" is a strong word here. This change avoids a problem via an unknown mechanism, but I haven't seen anything that indicates that it repairs it. The docs for the python multiprocessing package say:
On macOS, the *spawn* start method is now the default. The *fork* start method should be considered unsafe as it can lead to crashes of the subprocess as macOS system libraries may start threads. See :issue:33725.

@dmitry-ganyushin
Copy link
Contributor Author

dmitry-ganyushin commented Jul 4, 2023

That test shows that the manual is only partly correct: the "spawn" method does not work universally as it is expected by Python principles. It yields the following traceback for the test:
Traceback (most recent call last):
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/work/testing/adios2/engine/dataman/TestDataMan1D.py", line 34, in test_run
s.start()
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

@eisenhauer
Copy link
Member

I'm familiar with the details, and if there was an open Python issue or a PR for changing it back to fork in a future version, I'd be more inclined to think that this is the right thing to do. I just haven't seen anything like that.

But, you know, it doesn't matter. This is a minor test, duplicating a problem a user had. Lets just merge.

@eisenhauer eisenhauer merged commit c8c5238 into ornladios:master Jul 4, 2023
@dmitry-ganyushin dmitry-ganyushin deleted the i3646-mult-macos branch July 4, 2023 20:21
@ax3l
Copy link
Contributor

ax3l commented Jul 26, 2023

cc @vicentebolea for release_29?

@vicentebolea vicentebolea added this to the v2.9.1 milestone Jul 26, 2023
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Aug 2, 2023
This was referenced Aug 2, 2023
vicentebolea added a commit that referenced this pull request Aug 3, 2023
* release_29:
  Bump version to v2.9.1
  Merge pull request #3676 from pnorbert/c_version2
  Merge pull request #3697 from ornladios/dependabot/pip/docs/cryptography-41.0.2
  Merge pull request #3698 from vicentebolea/add-gitattribute-file
  Merge pull request #3704 from ornladios/dependabot/pip/docs/pygments-2.15.0
  Merge pull request #3712 from ornladios/dependabot/pip/docs/certifi-2023.7.22
  Merge pull request #3687 from dmitry-ganyushin/i3646-mult-macos
  Merge pull request #3699 from vicentebolea/revert-normalize-eol
  Merge pull request #3689 from vicentebolea/fix-evpath-modules
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants