From 5e7ea95d9d5c3b80a67ffbeebd76ce4fc327dd8e Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Fri, 22 Sep 2023 22:04:20 -0700 Subject: [PATCH] gh-100228: Document the os.fork threads DeprecationWarning. (#109767) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the `os.fork` posix threads detected `DeprecationWarning` in 3.12 What's New, os, multiprocessing, and concurrent.futures docs. Many reviews and doc cleanup edits by Adam & Hugo. 🥳 Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade --- Doc/library/concurrent.futures.rst | 8 +++++++ Doc/library/multiprocessing.rst | 6 +++++ Doc/library/os.rst | 36 ++++++++++++++++++++++++++---- Doc/whatsnew/3.12.rst | 12 ++++++++++ 4 files changed, 58 insertions(+), 4 deletions(-) diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 09c9fc4e6e227a..6503d1fcf70a32 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -293,6 +293,14 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. The *max_tasks_per_child* argument was added to allow users to control the lifetime of workers in the pool. + .. versionchanged:: 3.12 + On POSIX systems, if your application has multiple threads and the + :mod:`multiprocessing` context uses the ``"fork"`` start method: + The :func:`os.fork` function called internally to spawn workers may raise a + :exc:`DeprecationWarning`. Pass a *mp_context* configured to use a + different start method. See the :func:`os.fork` documentation for + further explanation. + .. _processpoolexecutor-example: ProcessPoolExecutor Example diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 38d24a86072970..2f0f1f800fdc94 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -131,6 +131,12 @@ to start a process. These *start methods* are Code that requires *fork* should explicitly specify that via :func:`get_context` or :func:`set_start_method`. + .. versionchanged:: 3.12 + If Python is able to detect that your process has multiple threads, the + :func:`os.fork` function that this start method calls internally will + raise a :exc:`DeprecationWarning`. Use a different start method. + See the :func:`os.fork` documentation for further explanation. + *forkserver* When the program starts and selects the *forkserver* start method, a server process is spawned. From then on, whenever a new process diff --git a/Doc/library/os.rst b/Doc/library/os.rst index c67b966f777db8..74897a76b1d20a 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -4157,15 +4157,38 @@ written in Python, such as a mail server's external command delivery program. .. audit-event:: os.fork "" os.fork + .. warning:: + + If you use TLS sockets in an application calling ``fork()``, see + the warning in the :mod:`ssl` documentation. + .. versionchanged:: 3.8 Calling ``fork()`` in a subinterpreter is no longer supported (:exc:`RuntimeError` is raised). - .. warning:: - - See :mod:`ssl` for applications that use the SSL module with fork(). + .. versionchanged:: 3.12 + If Python is able to detect that your process has multiple + threads, :func:`os.fork` now raises a :exc:`DeprecationWarning`. + + We chose to surface this as a warning, when detectable, to better + inform developers of a design problem that the POSIX platform + specifically notes as not supported. Even in code that + *appears* to work, it has never been safe to mix threading with + :func:`os.fork` on POSIX platforms. The CPython runtime itself has + always made API calls that are not safe for use in the child + process when threads existed in the parent (such as ``malloc`` and + ``free``). + + Users of macOS or users of libc or malloc implementations other + than those typically found in glibc to date are among those + already more likely to experience deadlocks running such code. + + See `this discussion on fork being incompatible with threads + `_ + for technical details of why we're surfacing this longstanding + platform compatibility problem to developers. - .. availability:: Unix, not Emscripten, not WASI. + .. availability:: POSIX, not Emscripten, not WASI. .. function:: forkpty() @@ -4178,6 +4201,11 @@ written in Python, such as a mail server's external command delivery program. .. audit-event:: os.forkpty "" os.forkpty + .. versionchanged:: 3.12 + If Python is able to detect that your process has multiple + threads, this now raises a :exc:`DeprecationWarning`. See the + longer explanation on :func:`os.fork`. + .. versionchanged:: 3.8 Calling ``forkpty()`` in a subinterpreter is no longer supported (:exc:`RuntimeError` is raised). diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 4ee87974a98d16..22538a476d31b4 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -1065,6 +1065,18 @@ Deprecated contain the creation time, which is also available in the new ``st_birthtime`` field. (Contributed by Steve Dower in :gh:`99726`.) +* :mod:`os`: On POSIX platforms, :func:`os.fork` can now raise a + :exc:`DeprecationWarning` when it can detect being called from a + multithreaded process. There has always been a fundamental incompatibility + with the POSIX platform when doing so. Even if such code *appeared* to work. + We added the warning to to raise awareness as issues encounted by code doing + this are becoming more frequent. See the :func:`os.fork` documentation for + more details. + + When this warning appears due to usage of :mod:`multiprocessing` or + :mod:`concurrent.futures` the fix is to use a different + :mod:`multiprocessing` start method such as ``"spawn"`` or ``"forkserver"``. + * :mod:`shutil`: The *onerror* argument of :func:`shutil.rmtree` is deprecated as will be removed in Python 3.14. Use *onexc* instead. (Contributed by Irit Katriel in :gh:`102828`.)