Skip to content

Commit

Permalink
Troubleshooting guide: add dynamic patcher problem handling
Browse files Browse the repository at this point in the history
- add basic description of the module cleanup hooks
  • Loading branch information
mrbean-bremen committed Mar 7, 2024
1 parent 2551f7b commit 22958d3
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 6 deletions.
53 changes: 53 additions & 0 deletions docs/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,59 @@ There are some situations where that may happen, probably without you noticing:
* The same is true, if you use ``setUpPyfakefs`` or ``setUpClassPyfakefs`` in a unittest context, or if you use
the ``patchfs`` decorator. ``Patcher`` instances created in the tests will be ignored likewise.

.. _failing_dyn_patcher:

Tests failing after a test using pyfakefs
-----------------------------------------
If previously passing tests fail after a test using ``pyfakefs``, something may be wrong with reverting the
patches. The most likely cause is a problem with the dynamic patcher, which is invoked if modules are loaded
dynamically during the tests. These modules are removed after the test, and reloaded the next time they are
imported, to avoid any remaining patched functions or variables. Sometimes, there is a problem with that reload.

If you want to know if your problem is indeed with the dynamic patcher, you can switch it off by setting
:ref:`use_dynamic_patch` to `False` (here an example with pytest):

.. code:: python
@pytest.fixture
def fs_no_dyn_patch():
with Patcher(use_dynamic_patch=False):
yield
def test_something(fs_no_dyn_patch):
... # do the testing
If in this case the following tests pass as expected, the dynamic patcher is indeed the problem.
If your ``pyfakefs`` test also works with that setting, you may just use this. Otherwise,
the dynamic patcher is needed, and the concrete problem has to be fixed. There is the possibility
to add a hook for the cleanup of a specific module, which allows to change the process of unloading
the module. This is currently used in ``pyfakefs`` for two cases: to reload ``django`` views instead of
just unloading them (needed due to some django internals), and for the reload of a specific module
in ``pandas``, which does not work out of the box.

A cleanup handler takes the module name as an argument, and returns a Boolean that indicates if the
cleanup was handled (by returning `True`), or if the module should still be unloaded. This handler has to
be added to the patcher:

.. code:: python
def handler_no_cleanup(_name):
# This is the simplest case: no cleanup is done at all.
# This makes only sense if you are sure that no file system functions are called.
return True
@pytest.fixture
def my_fs():
with Patcher():
patcher["modulename"] = handler_no_cleanup
yield
As this may not be trivial, we recommend to write an issue in ``pyfakefs`` with a reproducible example.
We will analyze the problem, and if we find a solution we will either get this fixed in ``pyfakefs``
(if it is related to a commonly used module), or help you to resolve it.


.. _`multiprocessing`: https://docs.python.org/3/library/multiprocessing.html
.. _`subprocess`: https://docs.python.org/3/library/subprocess.html
Expand Down
9 changes: 3 additions & 6 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -671,19 +671,16 @@ If you want to clear the cache just for a specific test instead, you can call
fs.clear_cache()
...
.. _use_dynamic_patch:

use_dynamic_patch
~~~~~~~~~~~~~~~~~
If ``True`` (the default), dynamic patching after setup is used (for example
for modules loaded locally inside of functions).
Can be switched off if it causes unwanted side effects, though that would mean that
dynamically loaded modules are no longer patched, if they use file system functions.
See also :ref:`failing_dyn_patcher` in the troubleshooting guide for more information.

A possible problem with dynamically loaded modules is their unloading, for example if
reloading the module fails after unloading due to a problem in the module.
This kind of problem may be solved more specifically by registering a handler
for specific modules (using `Patcher.register_cleanup_handler`),
that will be called during the cleanup process. This is used internally
to handle known problems with the `django` and `pandas` packages.

.. _convenience_methods:

Expand Down

0 comments on commit 22958d3

Please sign in to comment.