py3.9: failing tests: "bad marshal data" #71
Comments
|
It looks like python 3.9 will change the .pyc format (or perhaps just the marshal/pickle protocol). The solution probably involves comparing https://github.com/python/cpython/blob/master/Lib/modulefinder.py to pydeps' version... A cursory look indicates that py3.9 skips the first 16 bytes (https://github.com/python/cpython/blob/master/Lib/modulefinder.py#L348) while our version skips the first 8 (https://github.com/thebjorn/pydeps/blob/master/pydeps/mf27.py#L324). There seems to be a number of other changes as well... I'll gladly accept a PR from anyone that wants to work on this :-) |
|
Why does pydeps bundle a version of the modulefinder library anyway - using the one from the stdlib should always load the pyc files from the current version correctly? Your guess that this is caused by the byte offset seems to be correct. At least installing pydeps into a venv and changing that single detail resolved the issue for me. |
|
I'm interested what the motivation is to provide a custom modulefinder (mf27.py) ? |
|
IIRC, it's almost entirely due to the bug that is fixed in https://github.com/thebjorn/pydeps/blob/master/pydeps/mf27.py#L503 (original line commented out below). It doesn't stop Python's modulefinder.py from finding all modules, but it doesn't correctly represent the parent/child caller/callee importer/imported relationship - which is kind of important for pydeps. |
|
Do you think it would be possible to subclass the module finder from the stdlib and only overwrite that single method? |
This fixes thebjorn#71 The vendored and outdated version is removed. Instead the one from the stdlib is used and subclassed.
|
As the changes from #129 were effectively overwritten by some other commits this again fails (at least on py3.10) |
|
Yes, unfortunately the bundled modulefinder has an issue in Python 3.8-10 (python/cpython#84530) that prevents direct usage, see #139, #140, #141. |
|
But even more unfortunate is that it is currently broken with every module on newer python versions |
|
I guess that depends on which python version you are on, but I agree it's sub-optimal. Just pushing in the native modulefinder will also break (with the bug above) on newer python versions however, so it requires a better fix in pydeps. Unfortunately I haven't had time to look at it yet (nor write testcases to prevent regressions). |
|
FWIW, resilient magic number parsing and .pyc loading has been added to v1.12.17, which hopefully fixes this issue as well. |
Hello,
A few tests are failing while building pydeps for Fedora with py3.9:
The complete build logs are attached:
pydeps-build.txt
pydeps-root.txt
The text was updated successfully, but these errors were encountered: