__pycache__ in the wheel

[Siecje]

In the wheel on PyPI there are the following directories:

jupyter_core/tests/__pycache__/
jupyter_core/tests/dotipython/profile_default/__pycache__/
jupyter_core/tests/dotipython_empty/profile_default/__pycache__/

Should they be there?

I noticed this because we only deploy .pyc files so after the pip install I pull the .pyc files out of __pycache__ and into the location of the .py files using this script.

https://bitbucket.org/Siecje/restore_pyc/src/9834a5c2c16acf29b5b330ebd7590585bc20126b/restore_pyc.py?at=default&fileviewer=file-view-default

I noticed that the resulting .pyc had a different magic code. There is likely a bug in the restore_pyc.py script linked above as it should be finding the .pyc file that was created during the pip install.

But I figured I would report that there are __pycache__ directories in the wheel.

[takluyver]

Not really; it's probably because of this function which includes all directories under tests as package data.

[Siecje]

It seems that function is not responsible, it doesn't seem to change the wheel.

If you have a __pycache__/ directory in your package and the package is listed in packages then it will be included in the wheel. I guess every project has this problem.

[coldfix]

This function is responsible. Via print debugging I found that it generates the following entries to package_data['jupyter_core.tests']:

*.*
dotipython_empty/profile_default/*.*
dotipython_empty/profile_default/static/custom/*.*
dotipython_empty/profile_default/__pycache__/*.*
__pycache__/*.*
dotipython/profile_default/*.*
dotipython/profile_default/static/custom/*.*
dotipython/profile_default/__pycache__/*.*
dotipython/nbextensions/*.*

The easiest workaround is to replace:

     for parent, dirs, files in os.walk(test_dir):
-        if files:
+        if files and os.path.basename(parent) != '__pycache__':
             test_files.append(pjoin(parent[prefix_len:], '*.*'))

as a first step.

Alternatively, make sure to build the wheel on a clean repository (e.g. git clean -xdf).

However, I think the real issue is that wildcard inclusion is pretty dangerous and probably not the best idea in general. Instead, one should list explicitly what kinds of files should be in the archive. For example set include_package_data = True and then let MANIFEST.in do the rest (this would require setuptools though, but I don't think that's an issue now that you have universal wheels and users generally don't come in contact with the source package).

I tried this approach with your current MANIFEST.in, and discovered that the global-exclude directive doesn't seem to work. Might be related to pypa/setuptools#849, which is supposed to have been fixed for over 2 years.

Note: Make sure to remove your build folder before rebuilding the wheel -- otherwise it will use the list of files generated during earlier builds and still include the .pyc files.