pkg_resources-style namespace package issue

[frenzymadness]

Hello.

I have one issue related to namespace package. The project I am working on [0] is using pkg_resources-style namespace packages in a correct way described by documentation [1].

the process is:

1. During a build of a first subpackage as RPM package setuptools skip installation of <namespace-package>/__init__.py with message Skipping installation of /builddir/build/BUILDROOT/python-moksha-common-1.2.5-1.fc27.x86_64/usr/lib/python3.6/site-packages/moksha/__init__.py (namespace package)
2. Then I install newly created RPM from the step one.
3. When I try to build a second subpackage as RPM package (which has the first one as a dependency) build fails because the second subpackage needs to import the first one installed from RPM, but the first one doesn't contain init.py file and that is the reason why the namespace logic fails and the other sub-packages is not importable.

In docs [1] there is:

Every distribution that uses the namespace package must include an identical __init__.py. If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional code in __init__.py will be inaccessible.

The question is: Why setuptools skip installation of pkg_resources-style __init__.py file in namespace package when it must be included in every part of a namespace package?

My current workaround is to add missing files to RPM package manually and then everything works. I also tried to delete __init__.py files from the second subpackage before running tests with python setup.py test (where import of the first subpackage is needed) but then it fails with error in moksha.hub setup command: Distribution contains no modules or packages for namespace package 'moksha'.

I thought that the problem comes from Python 3.6 (bug report with more details and examples [2]) but it seems that Python 3.6 is only more strict in requiring __init__.py files everywhere as described in docs.

Maybe I just misunderstood something.

Thank you for help and have a nice day.

[0] https://github.com/mokshaproject/moksha
[1] https://packaging.python.org/guides/packaging-namespace-packages/#pkg-resources-style-namespace-packages
[2] http://bugs.python.org/issue29144

[cas--]

I have encountered this same issue with a debian package on Python 3 missing the namespace __init__.py files causing a ModuleNotFoundError with the plugins. Manually copying the __init__.py from source to relevant directories fixes the issue so I guess will have to do that in the debian package.

[cas--]

I may found part of the problem, where setuptools is no longer using nspkg.pth files: #805

However the removal of __init__.py files predates that change so perhaps when not using nspkg.pth the namespace __init__.py files should not be skipped?

I tracked down the original commit with the reason for skipping __init__.py: 92a6157

Just to add that I have tried in the past to move our application to native namespace but since we use eggs for plugins we need to use pkg_resources namespace so this is an annoying issue.

[jaraco]

Thanks @cas for summary and renewed interest in this issue. Sorry for not having responded to this issue sooner.

Just to add that I have tried in the past to move our application to native namespace but since we use eggs for plugins we need to use pkg_resources namespace so this is an annoying issue.

I would consider eggs deprecated. The PyPA has a distinct interest in removing all support for eggs. Is there a reason you couldn't use wheels (even zip wheels) for your plugins? Would you consider filing a ticket with pypa/packaging-problems and mention me, describing your use-case and what feature of eggs you rely upon?

setuptools is no longer using nspkg.pth

This is not quite true. Consider, for example:

draft $ python -m venv env
draft $ env/bin/pip install -U setuptools
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/e7/16/da8cb8046149d50940c6110310983abb359bbb8cbc3539e6bef95c29428a/setuptools-40.6.2-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed setuptools-40.6.2
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
draft $ env/bin/pip install jaraco.functools --no-deps
Collecting jaraco.functools
  Using cached https://files.pythonhosted.org/packages/c2/f3/ce1bd8d5434227d26c2c2e07a9f89229a6b81122437452871d7e67f0ec76/jaraco.functools-1.20-py2.py3-none-any.whl
Installing collected packages: jaraco.functools
Successfully installed jaraco.functools-1.20
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
draft $ cat env/lib/python3.7/site-packages/jaraco.functools-1.20-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('jaraco',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('jaraco', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('jaraco', [os.path.dirname(p)])));m = m or sys.modules.setdefault('jaraco', types.ModuleType('jaraco'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
draft $ env/bin/python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['jaraco']
<module 'jaraco' (namespace)>

The work in #805 ended up creating nspkg.pth files that perform the same logic as PEP-420 namespace packages, using the same API calls (importlib.util.module_from_spec) on Python 3.5 and later. Still, as you can see, setuptools produces nspkg.pth files when creating wheels and pip installs them.

As you've noticed, the omission of of __init__.py for namespace packages goes back a very long time, to 0.6a11, so I dare say this behavior is by design. The declare_namespace call in __init__.py (and the similar call in pkgutil) was meant as a compatibility shim until namespace packages had proper support (which originally was meant to be pkg_resources in stdlib, but later evolved into PEP 420 as implemented in Python 3.3 and improved in Python 3.5).

What's surprising to me is that you shouldn't need the -nspkg.pth files on Python 3.5 or later at all (continuing from prior example):

draft $ env/bin/pip install jaraco.functools
Requirement already satisfied: jaraco.functools in ./env/lib/python3.7/site-packages (1.20)
Collecting more-itertools (from jaraco.functools)
  Using cached https://files.pythonhosted.org/packages/79/b1/eace304ef66bd7d3d8b2f78cc374b73ca03bc53664d78151e9df3b3996cc/more_itertools-4.3.0-py3-none-any.whl
Collecting six<2.0.0,>=1.0.0 (from more-itertools->jaraco.functools)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, more-itertools
Successfully installed more-itertools-4.3.0 six-1.11.0
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
draft $ rm env/lib/python3.7/site-packages/jaraco.functools-1.20-py3.6-nspkg.pth
draft $ ls env/lib/python3.7/site-packages/jaraco
__pycache__  functools.py
draft $ env/bin/python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import jaraco.functools
>>>

As you can see, even after removing the -nspkg.pth file, the namespace package imports just fine, even though there's no __init__.py.

You should be able to replicate this behavior very simply:

draft $ mkdir foo
draft $ python -q
>>> import foo
>>> foo.__spec__
ModuleSpec(name='foo', loader=<_frozen_importlib_external._NamespaceLoader object at 0x10c8a94a8>, submodule_search_locations=_NamespacePath(['/Users/jaraco/draft/foo']))

No -nspkg.pth file and no __init__.py. That's how it's meant to work.

The issue becomes if you install not to a one of the known site-packages directories on a Python 3.4 or earlier, the nspkg.pth file won't be loaded and the package won't load properly (on 3.3 or 3.4, it might work or there may be other issues).

So I'm unable to replicate the issue in the original report.

I suspect there's another factor. Maybe you have a mixed environment, where some of the paths have the namespace package with __init__.py and others do not. If that's the case, I'll need some assistance recreating the environment in which you're encountering the issue.

Would you do something for me? Would you work to distill the issue to as minimal a case as you can muster? If you're familiar with Docker, an easy way to share the replication is to build a Dockerfile that replicates the steps, especially if it's unique to a particular Linux environment.

[cas--]

@jaraco Thanks for the comprehensive reply!

I have created an issue to describe the plugins egg usage: pypa/packaging-problems#212

[cas--]

I went back and checked the debian package and realised that the nspkg.pth file was not being copied from source to the install. This never seemed to be an issue on Python 2 however apparently it does on Python 3, likely due to pkg_resources vs native namespace.

Therefore the proper packaging fix was not to restore the __init__.py files but ensure the nspkg.pth file exists since we are using pkg_resources namespace.