gh-109047: Add PythonFinalizationError exception

[vstinner]

Add PythonFinalizationError. This exception derived from RuntimeError is raised when an operation is blocked during the Python finalization.

The following functions now raise PythonFinalizationError, instead of RuntimeError:

• _thread.start_new_thread()
• os.fork()
• os.fork1()
• os.forkpty()

Morever, _winapi.Overlapped finalizer now logs an unraisable PythonFinalizationError, instead of an unraisable RuntimeError.

• Issue: test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047

---

📚 Documentation preview 📚: https://cpython-previews--109809.org.readthedocs.build/

[vstinner]

Ubuntu failed (logs reformatted):

FAIL: test_interpreter_shutdown (test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError:

b'Exception in thread Thread-1:\n
Traceback (most recent call last):\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1059, in _bootstrap_inner\n
    self.run()\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/concurrent/futures/process.py", line 344, in run\n
    self.add_call_item_to_queue()\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/concurrent/futures/process.py", line 399, in add_call_item_to_queue\n
    self.call_queue.put(_CallItem(work_id,\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/queues.py", line 94, in put\n
    self._start_thread()\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/queues.py", line 177, in _start_thread\n
    self._thread.start()\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 978, in start\n
    _start_new_thread(self._bootstrap, ())\n
PythonFinalizationError: can\'t create new thread at interpreter shutdown\n
Traceback (most recent call last):\n
  File "<string>", line 1, in <module>\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/spawn.py", line 122, in spawn_main\n
    exitcode = _main(fd, parent_sentinel)\n
               ^^^^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/spawn.py", line 132, in _main\n
    self = reduction.pickle.load(from_parent)\n
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/synchronize.py", line 115, in __setstate__\n
    self._semlock = _multiprocessing.SemLock._rebuild(*state)\n
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n
FileNotFoundError: [Errno 2] No such file or directory\n'

is not false

[AA-Turner]

Perhaps drop the 'Python' prefix (or perhaps change to IntepreterFinalizationError etc) -- as a reader, I'd be confused as to what Python is trying to signify in the name.

A

[vstinner]

IntepreterFinalizationError

Python can have multiple interpreters. This exception is when the main interpreter is being finalized. I prefer to document it as "Python finalization".

If it's called FinalizationError, you may ask: which finalization? All Python objects have finalizers.

[AA-Turner]

PrimaryIntepreterFinalizationError? It's perhaps getting a bit silly, though. I still think that 'PythonFinalizationError' is unclear--for example as you noted I would naively think that it would apply to e.g. subinterpreter finalisation, as that's still part of Python.

Whilst FinalizationError is generic (as you say), that might not be a bad thing, in case custom object finalisers need to raise errors of a similar nature as here?

A

[vstinner]

I prefer the term "Python finalization error".

PrimaryIntepreterFinalizationError?

What is being finalized is not a sub-interpreter or the main interpeter, but in fact, it's the world "Python world": anything related to Python. It's wider than the main interpreter. There are many "global variables" cross-interprereters. Like path configuration (_PyPathConfig_ClearGlobal()), runtime (_PyRuntime_Finalize()), etc.

For example, before, Py_Initialize(), it's not possible to create a Python object. Py_Finalize() destroys ("finalizes") all Python objects. When Py_Finalize() completes, it's no longer possible to create new Python objects.

That's why https://docs.python.org/dev/c-api/init.html and https://docs.python.org/dev/c-api/init_config.html#init-config are about "Python Initialization". Sadly, https://docs.python.org/dev/c-api/init.html#c.Py_FinalizeEx is not well documented.

https://docs.python.org/dev/library/sys.html#sys.is_finalizing mentions "the main Python interpreter".

I'm not happy about https://docs.python.org/dev/glossary.html#term-interpreter-shutdown glossary entry. It isn't clear if we are talking about the main interpreter, a sub-interpreter, or "Python".

Moreover, there is a competition between "shutting down" and "finalization" terms in the documentation. It should be unified.

"Finalization" term is part of the public API: sys.is_finalizing() and Py_Finalize().

"Shutdown" is used in one private API: threading._shutdown().

My notes of Python finalization: https://pythondev.readthedocs.io/finalization.html

[vstinner]

My fix PR gh-109810 does not strictly need a new exception. So close this PR for now, until someone comes with a more concrete use case which requires adding a new exception.

[vstinner]

I created a new PR for this exception: #115352