ModuleNotFoundError with multiprocessing and import-mode=importlib

[jaraco]

In jaraco/keyring#673, I captured an issue stemming from the introduction of --import-mode importlib to jaraco/skeleton projects. In jaraco/keyring@6ff02e0, I'm attempting to enable importlib mode for all projects in order to support projects that use namespaces and those that don't in a uniform way. However, enabling importlib mode causes failures in tests that involve multiprocessing. Consider this minimal example:

 @ cat > tox.ini
[testenv]
deps =
	pytest
commands =
	pytest {posargs:--import-mode=importlib}
 @ mkdir tests
 @ cat > tests/test_multiprocess.py
import multiprocessing


def multiprocess_do():
    pass


def test_multiprocess():
    proc1 = multiprocessing.Process(target=multiprocess_do)
    proc1.start()
    proc1.join()
    assert proc1.exitcode == 0

 @ tree
.
├── tests
│   └── test_multiprocess.py
└── tox.ini

2 directories, 2 files
 @ tox
py: commands[0]> pytest --import-mode=importlib
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/draft
collected 1 item

tests/test_multiprocess.py F                                             [100%]

=================================== FAILURES ===================================
______________________________ test_multiprocess _______________________________

    def test_multiprocess():
        proc1 = multiprocessing.Process(target=multiprocess_do)
        proc1.start()
        proc1.join()
>       assert proc1.exitcode == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <Process name='Process-1' pid=38572 parent=38570 stopped exitcode=1>.exitcode

tests/test_multiprocess.py:12: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tests'
=========================== short test summary info ============================
FAILED tests/test_multiprocess.py::test_multiprocess - AssertionError: assert...
============================== 1 failed in 0.09s ===============================
py: exit 1 (0.19 seconds) /Users/jaraco/draft> pytest --import-mode=importlib pid=38570
  py: FAIL code 1 (0.22=setup[0.02]+cmd[0.19] seconds)
  evaluation failed :( (0.25 seconds)

The same error occurs even if tests/__init__.py is present. The issue occurs also with pytest 8.0.2 and 7.4.4. My guess is that it never worked with this mode.

It seems that tests.test_multiprocess is in sys.modules and objects passed between the processes might reference that module, so the child process needs to have the same path handling that the parent had but doesn't.

Running tox -- --import-mode prepend or tox -- --import-mode append bypasses the failure.

Can multiprocessing be made to work with import-mode importlib?

[jaraco]

Also, the failure doesn't occur if the tests are invoked with .tox/py/bin/python -m pytest, presumably because that mode adds . to sys.path.