Figure out how to handle long paths on Windows

[asmeurer]

Context

This was originally discussed at #559 (comment)

Windows support is still in progress (#559), but an issue that comes up is that Windows by default has a filesystem path length limit of 260 characters. You end up with errors like

ERROR:root:[WinError 206] The filename or extension is too long: 'C:\\Users\\Aaron\\Quansight\\conda-store\\conda-store-server\\conda-store-state\\default\\99108419ad0fd922fdeff9bbc434b58d41f68e3f923a83f6a7ab19568463bc82-20230915-201619-848782-1-test\\Lib\\site-packages\\sympy\\parsing\\autolev\\test-examples\\pydy-example-repo'

See https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation

It is possible to remove this limitation in Windows 10 and later if you set a registry value https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later. However, this (presumably) requires administrator privileges so I don't know if it's a viable thing to require.

The exact length of the path depends on the specific conda package. However, it's pretty obvious that the issue is the long UUID generated by conda-store: 99108419ad0fd922fdeff9bbc434b58d41f68e3f923a83f6a7ab19568463bc82-20230915-201619-848782-1-test. That's 90 characters (plus the 4 character environment name). Shortening or removing this UUID from the path would solve the issue for this specific package. One could also store the environment closer to the root (see also #554). However, I'm not really sure what the longest "real world" path in a conda package is. The conda issue mentions that this has been fixed upstream in many conda-forge packages with longer paths, but I don't know just how long of a path is considered "OK" for a Windows conda package, but it's still something that could pop up even if we fix both of those things. At that point, that would probably be considered an upstream issue with conda and/or the specific package, but it's worth being aware of this problem.

This is a known issue in conda. See conda/conda#7203.

Based on this comment, conda/conda#7203 (comment), if we want to try to make the path no longer than a typical conda install default, the part of the path between C:\Users\Username and the individual package files (like \Lib\site-packages\...) should be no longer than \AppData\Local\conda\conda\pkgs, which is 31 characters. But \.conda-store\conda-store-state, as proposed in #554, is already 31 characters.

It might be possible to try a scheme where we store packages in paths with very short hashes. But otherwise, we might just have to require people to set that registry key if they want to use things on Windows.

Value and/or benefit

Some packages inherently have long paths in them, and this would make them unusable on Windows. I haven't done a thorough investigation of how long the longest paths are on various packages on Windows, but this could be something that could be done.

Anything else?

Regardless of what we do, we need to make sure error messages are properly shown to the user (but this is sort of moot at this point because Windows support itself is still not truly functional).

[trallard]

@jaimergp since you are our conda-forge wizard I'd like to hear what you'd suggest here.

At a first glance it seems the easiest path would be to replace/shorten the hash used (UUID).

[asmeurer]

It would be useful to get a feel for how long real-world paths are in packages on Windows. How hard would it be to compute a list of

package: longest path in that package

for every Windows package on conda-forge?

[jaimergp]

Some Jupyter stuff can get easily in the hundred characters (e.g. notebook)

[jaimergp]

conda does the following:

• \\?\ prefixes
• Make the registry call to enable long paths with these params

I would say that it should be a pre-requisite for Windows users to enable long paths and be in a recent enough Windows versions (this can be done with a script we ship). Alternatively, if the users cannot do that (e.g. insufficient permissions, locked Windows versions), we can provide a global parameter to shorten the UUIDs to a certain amount while informing of the collision risks. These users should also be instructed to install to a very short path (e.g. C:\conda-store instead of C:\Users\username\anaconda\envs\conda-store) to save a few characters.

[asmeurer]

Some Jupyter stuff can get easily in the hundred characters (e.g. notebook)

For clarify, the longest path in that package is

Lib/site-packages/notebook/static/components/MathJax/fonts/HTML-CSS/STIX-Web/woff/STIXMathJax_DoubleStruck-BoldItalic.woff

which is 122 characters.

conda does the following:

IIRC, the error I saw was coming from conda somewhere. So conda doesn't completely take care of this internally. If it did, I suppose this wouldn't be an issue. I can try to reproduce the problem to see the full traceback if that helps.

[asmeurer]

Long paths can also be an issue on other platforms. For me, the test_conda_store_register_environment_workflow test fails locally on my Mac. The actual failure output isn't helpful (just a timeout) but if you dig into it, the failure comes from

conda-lock install --validate-platform --log-level INFO --prefix /private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/pytest-of-aaronmeurer/pytest-12/test_conda_store_register_envi0/conda-store-state/pytest-namespace/35f604188f69ceb5d9e3fae2c93ffd48c2971d192f21c776e3ebfed7e1196868-20231005-205956-556312-1-pytest-name /private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/tmpqckd1epr/conda-lock.yml
INFO:root:Downloading and Extracting Packages
INFO:root:Downloading and Extracting Packages
INFO:root:Preparing transaction: ...working... done
INFO:root:Verifying transaction: ...working... failed
ERROR:root:Placeholder of length '255' too short in package /private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/pytest-of-aaronmeurer/pytest-12/test_conda_store_register_envi0/conda-store-state/pytest-namespace/35f604188f69ceb5d9e3fae2c93ffd48c2971d192f21c776e3ebfed7e1196868-20231005-205956-556312-1-pytest-name/bin/bunzip2.
ERROR:root:The package must be rebuilt with conda-build > 2.0.
ERROR:root:
ERROR:root:PaddingError: Placeholder of length '255' too short in package /private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/pytest-of-aaronmeurer/pytest-12/test_conda_store_register_envi0/conda-store-state/pytest-namespace/35f604188f69ceb5d9e3fae2c93ffd48c2971d192f21c776e3ebfed7e1196868-20231005-205956-556312-1-pytest-name/bin/bunzip2.
ERROR:root:The package must be rebuilt with conda-build > 2.0.
ERROR:root:
Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1489, in install
    install_func(file=lockfile)
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 225, in do_conda_install
    _conda(
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/invoke_conda.py", line 129, in _invoke_conda
    raise subprocess.CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/Users/aaronmeurer/anaconda3/condabin/mamba', 'create', '--file', '/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/tmpzxedckcj', '--yes', '--prefix', '/private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/pytest-of-aaronmeurer/pytest-12/test_conda_store_register_envi0/conda-store-state/pytest-namespace/35f604188f69ceb5d9e3fae2c93ffd48c2971d192f21c776e3ebfed7e1196868-20231005-205956-556312-1-pytest-name']' returned non-zero exit status 1.

In other words, some package can't be installed because it has a has a placeholder prefix of 255 characters, which is shorter than the install prefix.

So I think we should definitely shorten/remove the hash from the env path. It's way too problematic.

[dcmcand]

My take is that this hash doesn't need to be cryptographically secure, it is just to make collisions of environments less likely. I would personally be in favor of truncating the has to 8 characters and adding the datetime, or hashing the sha of the environment plus the datetime and truncating that hash to 8 characters. Either way, that should result in a pretty significant shortening of the path.
I have admittedly made some assumptions about a code base that I am not super familiar with above. If I am incorrect about the function of the hash, please let me know.

[costrouc]

@nkaretnikov needs to take a look on this. Research needed.

[jaimergp]

We are looking at two separate issues related to paths here:

• conda-build will use a default prefix placeholder of 255 characters. This means that the prefix base path (e.g. ~/.local/conda-store/envs/env-name-1234abcd) always needs to be under 255 characters. There should be a check that validates the configuration and ensures that base installation path + intermediate directories + default length of the hashed environment <= 255. This is for all platforms. Issue for this is ENH - Shorten hash used in environment path #611.
• On top of that: Windows might have issues with long paths in general. This is, the prefix length (again, capped at 255 because conda) + the relative path length of the extracted file in the environment.
  • When long path support is not enabled, the maximum length is already 256 characters!
  • Users with admin permissions can get by with: cmd -> regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem -> LongPathsEnabled -> 1. This can be done programmatically with conda-store I guess, but we'll need to elevate the prompt. Apparently apps might also need to "manifest" compatibility (not sure how much that applies to us). Ray also noted that some old libraries might have assumed MAX_PATH in several places of their logic, so allowing that might break or cause crashes.
  • If not possible, we need to severely restrict the base prefix length and the cache to accommodate for as much path as possible in the 256 chars limit.

[jaimergp]

I have created more issues to split this problem in actionable items, all of them covered by this meta: #650

IMO this issue should be focused on the following subset:

• How to enable long paths programmatically on Windows (but make the user aware of the caveats)
• If not possible, building on top of [ENH] - Ensure that the length of all environment paths <= 255 characters #649, suggest configurations to reduce the prefix length (and extracted cache!) to accommodate for as much as possible:
  • Choose a different base path for the prefixes (e.g. C:\envs)
  • Choose a different base path for the extracted cache (e.g. C:\pkgs)
  • Reduce the hash length to the minimum possible
  • Add checks in the installation to anticipate (or diagnose) whether a file is (going to be) too long and have a more informative message than the default exception.

[nkaretnikov]

The extended-length path prefix "\\?\" has been already mentioned above, here's my comment on it from another issue: #611 (comment)

[jaimergp]

One possible workaround we haven't considered yet. Windows has a command to map directories to letter drives (e.g. C:\Long\Path\With\Issues -> D:\): subst.

We could use that to dramatically shorten the path used, or at least mention it in the documentation (drives are very visible on Windows, so some users might not like that permanent drive in their "My Computer").

[nkaretnikov]

Status update: I need to read through the proposals here again and propose the final plan that we're going to implement

[nkaretnikov]

@jaimergp To address the Windows-specific limitation (this issue), I propose we implement the "\\?\" support, which will be configured via an additional command-line parameter. How does it sound?

For completeness, I'll mention that other things discussed above have already been addressed:

• we have a build prefix version with a truncated hash now, which should minimize issues with the conda prefix (not specific to Windows), configured via CondaStore.build_key_version, landed in Use shorter build_key #652
• we have documentation on how to configure MAX_PATH via the registry on Windows in docs/administration.md, landed in Windows support #640
• there are errors that are thrown when the build prefix length is exceeded, those are visible to users via UI error messages as well, see Check the size of build_path #653 (size checks), UI part: Expose status_info in the UI conda-store-ui#332.

[jaimergp]

I propose we implement the "\?" support, which will be configured via an additional command-line parameter. How does it sound?

Sure, this needs to be done but as I mentioned above, it's not a final solution either. See this comment upstream.

So I'd say yes, let's proceed but:

• Why a command-line option and not a regular setting?
• We need to make sure the Windows registry has that flag set up. We can try to do that for the user, but it'll require administrator access and we might not be able to elevate easily. So we need a fallback workflow where the user will do that manually and then come back to us.
• Thus, the settings UI needs a section for long path support on Windows with:
  • Detection of the registry setting: ✅ "Windows is correctly configured", ❌ "Please enable long path support in Windows. Try now or read docs for manual instructions".
  • A checkbox to make sure they have understood the caveats mentioned above and how it's preferable to have a shorter base prefix to begin with (C:\envs). This should either link to docs or have a little popup report the potential caveats. It can also be an extended description in the UI.
  • If the registry setting is green, then we can enable a toggle for conda-store using \\?\.

Let me know if you need further clarification.

[nkaretnikov]

Why a command-line option and not a regular setting?

Yeah, I meant the standard way we set options, via traitlets.