Support mounting wheels that use extensions without an EXTENSIONS file

[stewartmiles]

Is your feature request related to a problem? Please describe.

To test out distlib.Wheel.mount() I tried downloading and mounting the wrapt wheel for Python 3.8 on Linux. Unfortunately, Wheel.mount() fails to extract the extension module from the wheel and load it.

Describe the solution you'd like

I would like to be able to mount wheels with extension modules.

Describe alternatives you've considered

I'm considering building a fork of distlib.Wheel.mount() that searches the wheel for all possible extension modules (.so, .dylib, .dll), unpacks them and loads them when imported.

Additional context

I downloaded the wrapt wheel as it's simple and contains a single extension module:

pip download wrapt

I attempted to mount it using the script:

import os

from distlib import wheel

THIS_DIRECTORY = os.path.dirname(os.path.realpath(__file__))

wrapt_wheel = wheel.Wheel(
  filename=os.path.join(
    THIS_DIRECTORY,
    'wrapt-1.16.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.'
    'manylinux_2_17_x86_64.manylinux2014_x86_64.whl'))
wrapt_wheel.mount(append=False)

from wrapt import _wrappers  # This is an extension module

However this fails with:

python mount_wheel.py
Traceback (most recent call last):
  File "mount_wheel.py", line 14, in <module>
    from wrapt import _wrappers
ImportError: cannot import name '_wrappers' from 'wrapt' (.../wrapt-1.16.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl/wrapt/__init__.py)

The problem is due to Wheel._get_extensions() which requires an EXTENSIONS file in the dist-info directory.

distlib/distlib/wheel.py

Line 746 in 888c48b

arcname = posixpath.join(info_dir, 'EXTENSIONS')

However, wrapt for Python 3.8 on Linux only contains:

unzip -l wrapt-1.16.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Archive:  wrapt-1.16.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2023-11-09 06:14   wrapt-1.16.0.dist-info/
        0  2023-11-09 06:14   wrapt/
        0  2023-11-09 06:14   wrapt.libs/
        6  2023-11-09 06:14   wrapt-1.16.0.dist-info/top_level.txt
      217  2023-11-09 06:14   wrapt-1.16.0.dist-info/WHEEL
     6602  2023-11-09 06:14   wrapt-1.16.0.dist-info/METADATA
     1304  2023-11-09 06:14   wrapt-1.16.0.dist-info/LICENSE
     1097  2023-11-09 06:14   wrapt-1.16.0.dist-info/RECORD
    27137  2023-11-09 06:14   wrapt/wrappers.py
    21333  2023-11-09 06:14   wrapt/decorators.py
     3881  2023-11-09 06:14   wrapt/weakrefs.py
     1238  2023-11-09 06:14   wrapt/__init__.py
    10997  2023-11-09 06:14   wrapt/importer.py
      443  2023-11-09 06:14   wrapt/__wrapt__.py
   204768  2023-11-09 06:14   wrapt/_wrappers.cpython-38-x86_64-linux-gnu.so
     1746  2023-11-09 06:14   wrapt/arguments.py
     5204  2023-11-09 06:14   wrapt/patches.py
---------                     -------
   285973                     17 files

Interestingly according to the - AFAIK - official wheel spec the required EXTENSIONS file is not part of the spec (see here) so it seems a bit unfair to expect wheels using extensions to publish it.

[stewartmiles]

@vsajip it looks like you were the original author any thoughts on an approach to solving this?

[stewartmiles]

stewartmiles@28a8c39 is a sketch of a solution, of course it needs a test. Using my manual test mount_wheel.py referenced in #222 (comment) I can load the wrapt extension module.

[vsajip]

I've been away for a few weeks, so I've only just come across this. Will look into it soon. Thanks!

[stewartmiles]

@vsajip thanks for your reply. I've been working on a branch https://github.com/stewartmiles/distlib/tree/wheel_mount_extensions that adds source and build script for the test wheel minimext and am at the point where I've updated the tests to use the new wheels. However, I changed the test wheel to embed the extension within a package i.e the distribution now contains minimext.calculate*.so and minimext.calculate_py.py. With this change WheelTestCase.test_mount_extensions() fails. I'm debugging at the mo' but lmk if you have thoughts.

[stewartmiles]

@vsajip I've sent a few pull requests related to this and opened a few other issues associated with the patch series. If you want to see all patches applied and everything working end-to-end you can pull https://github.com/stewartmiles/distlib/tree/wheel_mount_extensions . The only patch I haven't added to a pull request yet is the final one in the series 0ad2e29 .

Any idea when you'll be able to take a look? I would prefer to not have to maintain a fork.

[vsajip]

I hope to be able to look at this early next week. Thanks for looking into this!