modulefinder chokes on numpy - dereferencing None in spec.loader #84530

GregWhiteley · 2020-04-21T07:33:38Z

BPO	40350
Nosy	@warsaw, @ericvsmith, @ericsnowcurrently, @FFY00, @caje731
PRs	gh-84530: Introduce a new type for namespace packages #19917 gh-84530: fix namespace package support in modulefinder #29196
Files	fulllog.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-21.07:33:37.811>
labels = ['3.8', 'type-bug', 'library', '3.9']
title = 'modulefinder chokes on numpy - dereferencing None in spec.loader'
updated_at = <Date 2021-10-23.21:13:12.303>
user = 'https://bugs.python.org/GregWhiteley'

bugs.python.org fields:

activity = <Date 2021-10-23.21:13:12.303>
actor = 'FFY00'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-21.07:33:37.811>
creator = 'Greg Whiteley'
dependencies = []
files = ['49079']
hgrepos = []
issue_num = 40350
keywords = ['patch']
message_count = 6.0
messages = ['366912', '367856', '367943', '368112', '380087', '404911']
nosy_count = 7.0
nosy_names = ['barry', 'eric.smith', 'dkasak', 'eric.snow', 'FFY00', 'cajetan.rodrigues', 'Greg Whiteley']
pr_nums = ['19917', '29196']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40350'
versions = ['Python 3.8', 'Python 3.9']

GregWhiteley · 2020-04-21T07:33:37Z

Issue:

Running ModuleFinder.run_script() on numpy versions 1.16.1 to 1.18.3 (maybe more) fails with backtrace. See steps to reproduce below.

I do not see this problem on earlier versions of python than 3.8 (tested 3.4, 3.5, 3.6 on ubuntu LTSs), but the code has changed around 3.8.

The failure comes to this line of modulefinder.py

https://github.com/python/cpython/blame/master/Lib/modulefinder.py#L79

    if spec.loader.is_package(name):
        return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)

I can work around it by changing that to check for None

    if spec.loader is not None and spec.loader.is_package(name):
        return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)

Environment:

Ubuntu 20.04 with default python3, python3-pip

Steps to reproduce:

# note any nump version I've tried 1.16.1 and greater fails - I included 1.18.3 to be precise for reproduciton
$ pip3 install "numpy==1.18.3"
$ cat test.py
import numpy

$ python3
>>> from modulefinder import ModuleFinder
>>> finder = ModuleFinder()
>>> finder.run_script("test.py")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/modulefinder.py", line 165, in run_script
    self.load_module('__main__', fp, pathname, stuff)
...

300 lines of stack elided - see attached fulllog.txt

...
File "/usr/lib/python3.8/modulefinder.py", line 433, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
File "/usr/lib/python3.8/modulefinder.py", line 378, in _safe_import_hook
self.import_hook(name, caller, level=level)
File "/usr/lib/python3.8/modulefinder.py", line 177, in import_hook
q, tail = self.find_head_package(parent, name)
File "/usr/lib/python3.8/modulefinder.py", line 233, in find_head_package
q = self.import_module(head, qname, parent)
File "/usr/lib/python3.8/modulefinder.py", line 320, in import_module
fp, pathname, stuff = self.find_module(partname,
File "/usr/lib/python3.8/modulefinder.py", line 511, in find_module
return _find_module(name, path)
File "/usr/lib/python3.8/modulefinder.py", line 78, in _find_module
if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'

>>

Obviously I can't tell if numpy or modulefinder is the real culprit.

Let me know if I can give any more information.

ericsnowcurrently · 2020-05-01T15:40:47Z

Ah, namespace packages. :) Yeah, the code is not taking the "spec.loader is None" case into account. I expect the fix would be to add handling of that case a few lines up in the code, right after handling BuiltinImporter and FrozenImporter. Offhand I'm not sure if the "type" should be _PKG_DIRECTORY or some new one just for namespace packages. How does imp.find_module() (on which modulefinder._find_module() is based) respond to namespace packages?

[1] https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.loader

caje731 · 2020-05-02T17:42:50Z

Reproduced on Python3.9.0a5+

imp.find_module() simply raised an ImportError in my tests with an implicitly namespaced package (without an __init__.py)

About the "type_", I think it should be consistent with _PKG_DIRECTORY, since PEP-420 states the following[1]:

A namespace package is not fundamentally different from a regular package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference between it and a regular package.

I'd be happy to submit a patch if you think this is alright.

[1] https://www.python.org/dev/peps/pep-0420/#id24

caje731 · 2020-05-05T08:47:09Z

Turns out using _PKG_DIRECTORY as a type for namespace packages ended up making _find_module try to search for an init.py within them, since it had no understanding of the diff. between a namespace package and a regular one (the lack of init.py), and caused it to break.

I've raised a PR with a new type _NSP_DIRECTORY for namespace-directories.

dkasak · 2020-10-31T18:26:00Z

Anything still left to do that is stalling this? I just got bitten by it when trying to use modulefinder.

FFY00 · 2021-10-23T21:13:12Z

I opened up a new PR that should fix this properly. The root issue was that PathFinder was not setting the loader attribute for namespace packages in the spec, which it should. After fixing that, _find_module just needed to be updated to deal with NamespaceLoader.

$ tree namespace
namespace/
└── a.py

0 directories, 1 file
$ cat test-bpo-40350.py
import namespace.a

$ ./python
Python 3.11.0a1+ (heads/main:9e05da6224, Oct 23 2021, 20:36:14) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> modulefinder.ModuleFinder('test-bpo-40350.py')
<modulefinder.ModuleFinder object at 0x7f66f7647480>
>>> f = modulefinder.ModuleFinder('test-bpo-40350.py')
>>> f.
KeyboardInterrupt
>>> m = modulefinder.ModuleFinder()
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
>>> f.modules.items()
dict_items([('__main__', Module('__main__', 'test-bpo-40350.py')), ('namespace', Module('namespace', _NamespacePath(['/home/anubis/git/cpython/namespace'])))])

Previously:

$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/modulefinder.py", line 161, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/usr/lib/python3.9/modulefinder.py", line 357, in load_module
    self.scan_code(co, m)
  File "/usr/lib/python3.9/modulefinder.py", line 430, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/lib/python3.9/modulefinder.py", line 375, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/usr/lib/python3.9/modulefinder.py", line 173, in import_hook
    q, tail = self.find_head_package(parent, name)
  File "/usr/lib/python3.9/modulefinder.py", line 229, in find_head_package
    q = self.import_module(head, qname, parent)
  File "/usr/lib/python3.9/modulefinder.py", line 316, in import_module
    fp, pathname, stuff = self.find_module(partname,
  File "/usr/lib/python3.9/modulefinder.py", line 508, in find_module
    return _find_module(name, path)
  File "/usr/lib/python3.9/modulefinder.py", line 77, in _find_module
    if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'

The modulefinder._find_module has an unpatched bug (python/cpython#84530) in py3.8, 3.9, 3.10. Neither imp.find_module, nor the buggy modulefinder._find_module, handles namespace packages... This fixes #130. #140, #141.

nabheet · 2024-02-15T23:49:35Z

So I found another use case where this same error occurs. I recently learnt about python freeze.py to package our application into a single binary. I think this module finder error is preventing me from creating our binary.

$ pipenv run python script/freeze.py -p /usr/local/  src/main.py
['/workspaces/api/src', '/usr/share/doc/python3.11/examples/freeze/', '/workspaces/api/script', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload', '/workspaces/api/.venv/lib/python3.11/site-packages']
Traceback (most recent call last):
  File "/workspaces/api/script/freeze.py", line 512, in <module>
    main()
  File "/workspaces/api/script/freeze.py", line 386, in main
    mf.run_script(scriptfile)
  File "/usr/local/lib/python3.11/modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/usr/local/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/usr/local/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/local/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/usr/local/lib/python3.11/modulefinder.py", line 166, in import_hook
    m = self.load_tail(q, tail)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 243, in load_tail
    m = self.import_module(head, mname, m)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 315, in import_module
    m = self.load_module(fqname, fp, pathname, stuff)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/usr/local/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/local/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/usr/local/lib/python3.11/modulefinder.py", line 166, in import_hook
    m = self.load_tail(q, tail)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 243, in load_tail
    m = self.import_module(head, mname, m)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 315, in import_module
    m = self.load_module(fqname, fp, pathname, stuff)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/usr/local/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/local/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/usr/local/lib/python3.11/modulefinder.py", line 165, in import_hook
    q, tail = self.find_head_package(parent, name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 221, in find_head_package
    q = self.import_module(head, qname, parent)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 308, in import_module
    fp, pathname, stuff = self.find_module(partname,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 489, in find_module
    return _find_module(name, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/modulefinder.py", line 69, in _find_module
    if spec.loader.is_package(name):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'is_package'

I am really hoping this can be resolved soon-ish. 🤞

FFY00 · 2024-02-15T23:57:43Z

While this is still a valid bug, I'd recommend you use something like PyInstaller for that use-case, instead of freeze.py.

nabheet · 2024-02-16T00:26:33Z

Thank you!

When compiling multi-file projects that contain a namespace pacakge (folder without __init__.py), we hit Python bug python/cpython#84530. This adds a try/except block to catch the error and raise a more helpful error message. Issue: pybricks/support#1602

bjude · 2024-11-04T06:32:08Z

Are there any plans to rebase the existing PRs or otherwise fix this bug? We're currently having to hackily reimplement modulefinder._find_module() and monkey patch it which doesn't feel like the right solution for the standard library

GregWhiteley mannequin added 3.8 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Apr 21, 2020

ericsnowcurrently added 3.9 only security fixes labels May 1, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

thebjorn mentioned this issue Apr 21, 2022

tensorflow/keras/numpy breaks pydeps thebjorn/pydeps#141

Closed

FFY00 mentioned this issue May 2, 2022

Set loader for namespace packages in PathFinder.find_spec #92205

Open

thebjorn mentioned this issue May 19, 2022

py3.9: failing tests: "bad marshal data" thebjorn/pydeps#71

Open

serhiy-storchaka added the topic-importlib label May 22, 2022

TomJuri mentioned this issue Apr 17, 2024

[Feature] Support submodules pybricks/support#1602

Open

dlech mentioned this issue May 5, 2024

pybricksdev/compile.py: Add better error message pybricks/pybricksdev#85

Merged

micdet-delen mentioned this issue May 23, 2024

Support for editable packages / direct_url.json thebjorn/pydeps#222

Closed

bedevere-app bot mentioned this issue Dec 3, 2024

gh-84530: Introduce a new type for namespace packages #19917

Open

bedevere-app bot mentioned this issue Dec 3, 2024

gh-84530: fix namespace package support in modulefinder #29196

Open

picnixz added 3.12 bugs and security fixes 3.13 bugs and security fixes 3.14 new features, bugs and security fixes and removed 3.9 only security fixes 3.8 (EOL) end of life labels Dec 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modulefinder chokes on numpy - dereferencing None in spec.loader #84530

modulefinder chokes on numpy - dereferencing None in spec.loader #84530

GregWhiteley mannequin commented Apr 21, 2020

GregWhiteley mannequin commented Apr 21, 2020

ericsnowcurrently commented May 1, 2020

caje731 mannequin commented May 2, 2020

caje731 mannequin commented May 5, 2020

dkasak mannequin commented Oct 31, 2020

FFY00 commented Oct 23, 2021

nabheet commented Feb 15, 2024

FFY00 commented Feb 15, 2024

nabheet commented Feb 16, 2024

bjude commented Nov 4, 2024

modulefinder chokes on numpy - dereferencing None in spec.loader #84530

modulefinder chokes on numpy - dereferencing None in spec.loader #84530

Comments

GregWhiteley mannequin commented Apr 21, 2020

GregWhiteley mannequin commented Apr 21, 2020

ericsnowcurrently commented May 1, 2020

caje731 mannequin commented May 2, 2020

caje731 mannequin commented May 5, 2020

dkasak mannequin commented Oct 31, 2020

FFY00 commented Oct 23, 2021

nabheet commented Feb 15, 2024

FFY00 commented Feb 15, 2024

nabheet commented Feb 16, 2024

bjude commented Nov 4, 2024