'ImportError: DLL load failed while importing xxxxxxx: The filename or extension is too long' with LongPathsEnabled

[Andrej730]

Bug report

Bug description:

Python 3.11.9

In Python there there is a known limitation on Windows for paths being limited at 260 symbols, unless LongPathsEnabled is set, see https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation.

Without that registry key set import of module at long path fails with ModuleNotFoundError: No module named 'xxxxxxxxx' or opening file fails with FileNotFoundError: [Errno 2] No such file or directory. With registry key set the limit is extended and 260 symbols long paths are working fine.

The issue I've found that importing binary Python extension seems to break on long paths (250 symbols seems to be max length) even with LongPathsEnabled, see the snippet below.
To reproduce this, you can setup some very long path (>250 symbols) in your site-packages and put in that directory one empty .py file (in my case it's "yyyy...yyyyyyyyyyy.py") and some .pyd file that it's importing fine without any other binary dependencies - in my case I was using _yaml.cp311-win_amd64.pyd I got from https://pypi.org/project/PyYAML/#files.

# Importing just .py module.
import xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy as script
print(len(script.__file__)) # 298. Doesn't seem to have limit at 250-260 symbols.
# Importing .pyd.
import xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx._yaml as yaml
print(len(yaml.__file__)) # 250. seems to be the max.

Found this issue working with Blender (related issue) - Blender comes with it's own Python and there's an option to store some of Blender addons dependencies next to blender.exe, so it can be virtually anywhere on the computer and paths can get very long.

CPython versions tested on:

3.11

Operating systems tested on:

Windows

[zooba]

Yes, even with long paths enabled, Windows still has a per-segment limit: (from https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)

This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters)

So no individual directory or file name can have more than that many characters, even though the entire path might exceed that length. That's just an OS limit, and there's no getting around it.

Having a quick look through the Blender issue, it does seem that the issue there is some executables probably aren't correctly manifested to allow long paths. It's annoying that you have to do that, but again, that's the way Windows made it work (ironically, the first release of Windows with the long paths feature didn't check the manifests, so everything got it automatically and life was great! 😆 ).

[zooba]

I'm not sure there's anything we can do here, but let's leave the issue open for a couple of days in case someone has an idea. (I don't particularly want to try and generate a better error message than Windows itself, though in this case it's possible because the actual limit is available at runtime.)

[Andrej730]

@zooba

Yes, even with long paths enabled, Windows still has a per-segment limit: (from learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)

This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters)

So no individual directory or file name can have more than that many characters, even though the entire path might exceed that length. That's just an OS limit, and there's no getting around it.

The issue I report here is about .pyd filepaths being limited to around 250 symbols in overall size, not in a single segment.

Having a quick look through the Blender issue, it does seem that the issue there is some executables probably aren't correctly manifested to allow long paths

It was indeed the original issue with Blender with importing .py modules that's resolved with changing manifest, resolving helped figuring that there is another issue specifically with .pyd's.

[zooba]

Ah, thanks for clarifying.

I can reproduce, but it's deep inside Windows's dependency loader. As far as I can tell, it can only be fixed by an OS update.

It does appear to only occur when the DLL has an embedded manifest. Doesn't seem to matter what's in the manifest, but if it's there then it apparently tries to load something through the path and hits a too-long error.

I've reached out to the team who owns this part of the OS to see what they say. Removing the manifest entirely may be the right option, but I want to confirm that with the part of the OS that relies on it before making that general advice.