multiprocessing.Manager() + cd leads to EOFError

[sizmailov]

Describe the bug
Change of directory + multiprocessing.Manager() entrance results to EOFError with coverage 5.0

To Reproduce

1. What version of Python are you using?
   Python 3.8.0rc1
   It also fails on travis with Python 3.6.3

2. What version of coverage.py are you using?

coverage debug sys

-- sys -------------------------------------------------------
                   version: 5.0
                  coverage: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages/coverage/__init__.py
                    tracer: -none-
                   CTracer: available
      plugins.file_tracers: -none-
       plugins.configurers: -none-
 plugins.context_switchers: -none-
         configs_attempted: .coveragerc
              configs_read: .coveragerc
               config_file: .coveragerc
           config_contents: '[run]\nconcurrency = multiprocessing'
                 data_file: -none-
                    python: 3.8.0rc1 (default, Oct  1 2019, 21:48:24) [GCC 7.4.0]
                  platform: Linux-4.15.0-70-generic-x86_64-with-glibc2.27
            implementation: CPython
                executable: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin/python
              def_encoding: utf-8
               fs_encoding: utf-8
                       pid: 11108
                       cwd: /mnt/md0/Documents/PycharmProjects/remote-runner
                      path: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python38.zip
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/lib-dynload
                            /usr/lib/python3.8
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages/remote_runner-0.2.0-py3.8.egg
               environment: -none-
              command_line: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin/coverage debug sys
           sqlite3_version: 2.6.0
    sqlite3_sqlite_version: 3.22.0

3. --
4. What code are you running?

import multiprocessing
import os

if not os.path.exists("./tmp"): os.mkdir("./tmp")
os.chdir("./tmp")

with multiprocessing.Manager() as manager:
    pass

print("ok")

5. What commands did you run?

python -m coverage run repro.py

Traceback (most recent call last):
  File "repro.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

python repro.py

ok

Expected behavior
ok message

Additional context
Seems like bug was introduced with 5.0 release

[nedbat]

Hmm, something is different for me here on Mac OS. Under Python 3.8, I see the same behavior either with or without coverage, and with coverage==4.5.4 or coverage==5.0.1. I get a stack trace like this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 262, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 95, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/private/tmp/bug890/bug890.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/managers.py", line 579, in start
    self._process.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "bug890.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

If I change the code to this, it works under either:

import multiprocessing
import os

if not os.path.exists("./tmp"): os.mkdir("./tmp")
os.chdir("./tmp")

if __name__ == "__main__":
    with multiprocessing.Manager() as manager:
        pass

print("ok")

I'm not sure what the code is meant to do, and so which should go in the __main__ clause.

[sizmailov]

Thanks for spending time on it!

I'm sorry to not emphasize .coverargerc settings, it's hidden under coverage debug sys:

[run]
concurrency = multiprocessing

Without concurrency setting it doesn't throw.

The bug is 100% reproducible on all OS and python versions I've tried. I've set up a small repo to demonstrate, please see it's travis build failures.

For some reason windows builds hang forever after first failure command, therefore I cancelled windows jobs after first failure. (I'm not good at windows scripting, please tell how to fix if you know)

Note there are two scripts: bug890.py and bug890_with_main_guard.py. Bug manifests in both.

[sizmailov]

Here is build of coverage-4.5.4 branch with bug890_with_main_guard.py tests only:

https://travis-ci.com/sizmailov/reproduce-coverage-bug-890/builds/142106771

No failures.

[nedbat]

Thanks, the repo helps with the details.

[nedbat]

This is now fixed in df82a96. Thanks for the detailed report.

[nedbat]

This is now available in coverage 5.0.2