Python import of automesh module gives import error.

[hovey]

Describe the bug
ImportError on Python CLI with loading automesh module

To Reproduce

(.venv)  (ost) chovey@machine/Users/chovey/recon3d> python
Python 3.11.9 (v3.11.9:de54cf5be3, Apr  2 2024, 07:12:50) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import automesh as am
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chovey/recon3d/.venv/lib/python3.11/site-packages/automesh/__init__.py", line 1, in <module>
    from .automesh import *
ImportError: dlopen(/Users/chovey/recon3d/.venv/lib/python3.11/site-packages/automesh/automesh.cpython-311-darwin.so, 0x0002): Library not loaded: /opt/homebrew/opt/xz/lib/liblzma.5.dylib
  Referenced from: <D27B9254-7A13-3D51-AC56-FE702D6503B9> /Users/chovey/recon3d/.venv/lib/python3.11/site-packages/automesh/automesh.cpython-311-darwin.so
  Reason: tried: '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file)
>>>

Expected behavior
The module loads and then I can use the module in Python.

Screenshots
n/a - python command line dump above

Desktop (please complete the following information):

• OS: macOS 14.7
• Browser [e.g. chrome, safari]
• Version: automesh 0.2.2, python 3.11.9

[hovey]

Python package automesh is trying to load a shared library (liblzma.5.dylib) that it cannot find. This library is part of the xz compression library, which is often used for data compression.

https://crates.io/crates/xz

updated about 4 years ago

[mrbuche]

that is weird, I dont get the same errors on Ubuntu, and the Mac runners dont seem to have any issues

[hovey]

yeah, maybe my mac is deficient some how, I will tinker with it more... stay tuned 🤓

[hovey]

Why does the above error look for the library here: /opt/homebrew/opt/xz/lib/liblzma.5.dylib?

I haven't installed a Python with brew, viz.,

 chovey@machine/Users/chovey> brew list
==> Formulae
cmake	fish	gh	git-lfs	ncurses	pcre2	ripgrep

To find liblzma.5.dylib on macOS

sudo find / -name "liblzma.5.dylib" 2>/dev/null

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/PIL/.dylibs/liblzma.5.dylib
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PIL/.dylibs/liblzma.5.dylib
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cv2/.dylibs/liblzma.5.dylib
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib
/System/Volumes/Data/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/PIL/.dylibs/liblzma.5.dylib
/System/Volumes/Data/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PIL/.dylibs/liblzma.5.dylib
/System/Volumes/Data/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cv2/.dylibs/liblzma.5.dylib
/System/Volumes/Data/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib
/System/Volumes/Data/Users/chovey/recon3d/.venv/lib/python3.11/site-packages/PIL/.dylibs/liblzma.5.dylib
...

[hovey]

It appears liblzma in Python 3.11 is working just fine:

import lzma

# Example data to compress
data = b'This is some data to compress. ' * 10  # Create a larger byte string

# Compress the data
compressed_data = lzma.compress(data)
print(f'Compressed data: {compressed_data}')

# Decompress the data
decompressed_data = lzma.decompress(compressed_data)
print(f'Decompressed data: {decompressed_data.decode()}')  # Decode bytes to string

>>> import lzma
>>>
>>> # Example data to compress
>>> data = b'This is some data to compress. ' * 10  # Create a larger byte string
>>>
>>> # Compress the data
>>> compressed_data = lzma.compress(data)
>>> print(f'Compressed data: {compressed_data}')
Compressed data: b"\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3\xe0\x015\x00%]\x00*\x1a\t'd\x1c\x87\x8f\x1d-\x1b1\x1a\xe0\x19o\x90\x95[\xb0\xc9\xa8G\xe6\xbb\xe85\xa7\xed\xa8\xa0+V\x0f\\\x00\x00\x00\x00\x00\x00\xb8\xfc\x8e\x95>\x0e;2\x00\x01A\xb6\x02\x00\x00\x00+8\xdf2\xb1\xc4g\xfb\x02\x00\x00\x00\x00\x04YZ"
>>>
>>> # Decompress the data
>>> decompressed_data = lzma.decompress(compressed_data)
>>> print(f'Decompressed data: {decompressed_data.decode()}')  # Decode bytes to string
Decompressed data: This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress. This is some data to compress.

[hovey]

@mrbuche
This seems like a really nasty one to debug. Basically, I think there is something with how automesh is compiled, that it wants to seek out liblzam only in homebrew locations where I don't have it installed...

automesh.cpython-311-darwin.so, 0x0002): Library not loaded: /opt/homebrew/opt/xz/lib/liblzma.5.dylib
  Referenced from: <D27B9254-7A13-3D51-AC56-FE702D6503B9> /Users/chovey/recon3d/.venv/lib/python3.11/site-packages/automesh/automesh.cpython-311-darwin.so
  Reason: tried: '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file)

Do you know if the macOS runners use homebrew?

[mrbuche]

I do not think the runner use homebrew, this is definitely a strange one. Dependencies of dependencies are fun! From cargo tree it looks like it might be xz2, a dependency of dependency vtkio. Does the CLI work fine? I would think there would be an issue there as well since the Python build would not compile those parts differently, but maybe not.

[mrbuche]

#203