Sphinx dependency interferes with local, non-installed sphinxcontrib packages

[other-mickk]

Issue description

The Sphinx documentation system comes with extension support. Third-party extensions don't need to by installed by the user in order to be used, they can instead by placed locally and found by the Python runtime (from the docs). This is convenient for building documentation on other systems: ship the extension with the software that is being documented, no installation required.

I used to be able to build a Sphinx project of my own using a third-party extension before, but now I end up with the following Sphinx error:

Extension error:
Could not import extension sphinxcontrib.fulltoc (exception: No module named 'sphinxcontrib.fulltoc')

Steps to reproduce

(Roughly speaking.)

• put a third-party extension (in my case I'm using sphinxcontrib-fulltoc) somewhere without installing it either purely or impurely.
• create a Sphinx project making sure conf.py is set-up as per the official instructions to find this third-party extension
• drop into nix-shell -p python36.setuptools python36.sphinx and try to build the docs

Likely cause

d58e1f1 added the sphinxcontrib-websupport extension as a dependency to Sphinx. (Meanwhile sphinxcontrib-websupport was itself added to nixpkgs with 7726a1e.)

My understanding is that in the Sphinx ecosystem sphinxcontrib is meant to be a namespace package--both sphinxcontrib-websupport and sphinxcontrib-fulltoc provide their functionality as sphinxcontrib.websupport and sphinxcontrib.fulltoc modules respectively. Because Sphinx is a relatively long-lived project targeting both Python 2 and Python 3 its ecosystem uses pkg_resources-style namespace packages.

Because of its dependency, installing python36Packages.sphinx does a Nixified setuptools-style installation of sphinxcontrib-websupport. At this point setuptools turns the pkg_resources-style namespace package into a PEP 420-style namespace package. This is incompatible with the pkg_resources-style namespace package that is found at runtime in the local third-party extension, and this incompatibility is enforced by the interpreter starting from CPython 3.6.

That setuptools turns one kind of namespace package into another is possibly a bug:

• the python issue
• the setuptools issue

As a workaround I've changed my local copy of sphinxcontrib-fulltoc to use a PEP 420-style namespace package.

Technical details

• System: NixOS 18.03pre115134.d6f9c0b353 (Impala)
• Nix version: nix-env (Nix) 1.11.14
• Nixpkgs version: 18.03pre115134.d6f9c0b353
• Sandboxing enabled: no

[FRidh]

From what you describe this would seem to me entirely an upstream issue (CPython/Setuptools) and would have nothing to do with Nixpkgs. Do you agree? If yes, then we can close this issue. In any case, thank you for the extensive issue description. There's always something going on with namespace packages...

[other-mickk]

Do you agree?

Honestly there are too many moving parts for me to confidently assert anything. I would wait until someone in the setuptools project addresses the issue before deciding on anything: if they choose to tackle it, then presumably it's all good.

On the other hand if they kick the can then it's more complicated as I suppose the burden would fall next on Sphinx devs and third-party devs of the Sphinx ecosystem. (I expect the CPython folks will abide by PEP 420.) That's a lot of code and many maintainers, which could be time consuming.

[FRidh]

Well, I don't see how this is a Nixpkgs issue so I am closing this.