Detect POSIX-style paths on Windows #415

cr1901 · 2020-03-29T21:43:42Z

I have a git binary installed on Windows (the MSYS2 variant, specifically) that does not understand Windows-style paths. When trying to install software using setuptools_scm, this leads to cryptic errors such as the following:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    setup(
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "C:/msys64/mingw64/lib/python3.8/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 447, in __init__
    _Distribution.__init__(self, {
  File "C:/msys64/mingw64/lib/python3.8/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 740, in finalize_options
    ep.load()(self)
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/integration.py", line 17, in version_keyword
    dist.metadata.version = _get_version(config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 147, in _get_version
    parsed_version = _do_parse(config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 103, in _do_parse
    version = _version_from_entrypoints(config) or _version_from_entrypoints(
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 63, in _version_from_entrypoints
    version = _call_entrypoint_fn(root, config, ep.load())
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 44, in _call_entrypoint_fn
    return fn(root, config=config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/git.py", line 98, in parse
    wd = GitWorkdir.from_potential_worktree(config.absolute_root)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/git.py", line 33, in from_potential_worktree
    if not samefile(real_wd, wd):
  File "C:\msys64\mingw64/lib/python3.8/genericpath.py", line 100, in samefile
    s1 = os.stat(f1)
FileNotFoundError: [WinError 3] The system cannot find the path specified: '/home/William/Projects/FPGA/nmigen'

The error comes from the path git returns from the rev-parse command:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ git rev-parse --show-toplevel
/home/William/Projects/FPGA/nmigen

One potential solution is to pipe the output of git rev-parse through cygpath is we detect that the git being run is MSYS2 (or otherwise doesn't understand Windows-style paths), as unfortunately conversion between POSIX and Windows path is an involved problem:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ git rev-parse --show-toplevel | cygpath -w -f -
C:\msys64\home\William\Projects\FPGA\nmigen

The following workaround allows me to successfully run the egg-info, develop, and install successfully (with warning: no previously-included files found matching '.travis.yaml'). Obviously this patch isn't acceptable as-is, and this may not be the correct repo to bring this issue up, depending on the scope of supported git variants. However, I figure it's worth a shot.

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$ git diff
diff --git a/src/setuptools_scm/file_finder_git.py b/src/setuptools_scm/file_finder_git.py
index 8b81d2c..f68c91a 100644
--- a/src/setuptools_scm/file_finder_git.py
+++ b/src/setuptools_scm/file_finder_git.py
@@ -17,6 +17,13 @@ def _git_toplevel(path):
                 universal_newlines=True,
                 stderr=devnull,
             )
+
+            out = subprocess.check_output(
+                ["cygpath", "-w", out],
+                cwd=(path or "."),
+                universal_newlines=True,
+                stderr=devnull,
+            )
         trace("find files toplevel", out)
         return os.path.normcase(os.path.realpath(out.strip()))
     except subprocess.CalledProcessError:
diff --git a/src/setuptools_scm/git.py b/src/setuptools_scm/git.py
index afefa34..43cc6c5 100644
--- a/src/setuptools_scm/git.py
+++ b/src/setuptools_scm/git.py
@@ -27,6 +27,9 @@ class GitWorkdir(object):
     @classmethod
     def from_potential_worktree(cls, wd):
         real_wd, _, ret = do_ex("git rev-parse --show-toplevel", wd)
+        if ret:
+            return
+        real_wd, _, ret = do_ex("cygpath -w {}".format(real_wd), wd)
         if ret:
             return
         trace("real root", real_wd)

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$

The text was updated successfully, but these errors were encountered:

RonnyPfannschmidt · 2020-03-30T05:49:14Z

at first glance this will require quite a number of back and forth path transformations,

its not something that i can hope to build and test (i have neither the env nor the msys expertise, plus i use linux)

you can use the SETUPTOOLS_SCM_PRETEND_VERSION env var to work around git not providing data for the windows env, but i would most strongly suggest to use a windows git on the windows side and a msys git on the msys side

cr1901 · 2020-03-30T06:23:13Z

@RonnyPfannschmidt

i would most strongly suggest to use a windows git on the windows side and a msys git on the msys side

What about an msys git with an msys python (which also has the path mismatch problem)? Is that something you'd consider supporting? If you don't support this configuration, my workaround is probably better suited as a patch for MINGW-packages, and that you only explicitly support a windows git with a windows python.

RonnyPfannschmidt · 2020-03-30T06:53:29Z

im not sure whats the best approach there

if msys python uses windows paths instead of msys paths that sounds like a bug in msys python

the problem with supporting the configuration you mention is that i do not use windows or msys at all
and i have no idea how to set up a correct reliable CI for testing them either

so right now msys is not a system/setup i can support by myself, it needs help by someone using those systems and knowing them

wjblanke · 2020-04-01T19:36:17Z

@cr1901 we are trying to build wheels using msys2. We are seeing the exact same path issue you report. I've forked setuptools_scm and applied your patch. How did you replace setuptools_scm from msys2 with the forked version? Thanks!

cr1901 · 2020-04-02T00:35:58Z

@wjblanke According to pip list:

setuptools-scm      0.1.dev814+g69eb0fa.d20200329 c:/msys64/home/william/projects/py-packages/setuptools_scm/src

Meaning I did the following from my setuptools_scm checkout:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$ python setup.py develop
running develop
running egg_info
writing src/setuptools_scm.egg-info/PKG-INFO
writing dependency_links to src/setuptools_scm.egg-info/dependency_links.txt
writing entry points to src/setuptools_scm.egg-info/entry_points.txt
writing requirements to src/setuptools_scm.egg-info/requires.txt
writing top-level names to src/setuptools_scm.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching '.travis.yaml'
writing manifest file 'src/setuptools_scm.egg-info/SOURCES.txt'
running build_ext
Creating c:/msys64/mingw64/lib/python3.8/site-packages/setuptools-scm.egg-link (link to src)
setuptools-scm 0.1.dev814+g69eb0fa.d20200401 is already the active version in easy-install.pth

Installed c:/msys64/home/william/projects/py-packages/setuptools_scm/src
Processing dependencies for setuptools-scm==0.1.dev814+g69eb0fa.d20200401
Finished processing dependencies for setuptools-scm==0.1.dev814+g69eb0fa.d20200401

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$

wjblanke · 2020-04-02T04:00:57Z

Thanks @cr1901! We were able to get it to run in the toml with

requires = ["setuptools_scm@git+https://github.com/Chia-Network/setuptools_scm.git@master#egg=setuptools_scm", "setuptools>=42", "wheel", "setuptools_scm[toml]>=3.5.0"]

Unfortunately it doesn't look like pypi accepts mingw wheels.

HTTPError: 400 Client Error: Binary wheel 'chiapos-0.0.0-cp38-cp38-mingw.whl' has an unsupported platform tag 'mingw'. for url: https://test.pypi.org/legacy/

cr1901 · 2020-04-06T00:58:02Z

@wjblanke I'm guessing "mingw" is explicitly for packages that are part of the MSYS2 Project, and you otherwise have to use any tag:

$ python -c "import wheel.pep425tags as w; print(w.get_supported())"
[('cp38', 'cp38', 'mingw'), ('cp38', 'none', 'mingw'), ('cp38', 'none', 'any'), ('cp3', 'none', 'any'), ('cp37', 'none', 'any'), ('cp36', 'none', 'any'), ('cp35', 'none', 'any'), ('cp34', 'none', 'any'), ('cp33', 'none', 'any'), ('cp32', 'none', 'any'), ('cp31', 'none', 'any'), ('cp30', 'none', 'any'), ('py3', 'none', 'mingw'), ('py38', 'none', 'any'), ('py3', 'none', 'any'), ('py37', 'none', 'any'), ('py36', 'none', 'any'), ('py35', 'none', 'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')]

RonnyPfannschmidt · 2020-05-02T07:39:40Z

personally i wont work on this as i dont have the system needed, im happy to review pr's with tests

i wonder if a gitlab ci build could be added with such a environment to ensure its not breaking

until someone comes up with a working co it cant be declared a supported feature

cr1901 · 2020-05-02T08:13:17Z

@RonnyPfannschmidt Sorry, I wasn't expecting you to fix this, especially if you don't have the requisite system.

I've been trying to tie up a bunch of loose ends, and I haven't gotten around to adding a fix to this. Calling a binary to do parsing is a horrible approach. In the past I've used ctypes to load cygwin1.dll, but I also consider that unacceptable for a widely-used library.

im happy to review pr's with tests

I will take a look, though it may be a few days. You have a utils.py module- I'm sure I can stick something there.

working co

Working checkout?

RonnyPfannschmidt · 2020-05-02T08:22:31Z

Whops, I meant ci for continuous integration

xiaozhongtian · 2020-10-13T13:44:02Z

Same issue!

in MSYS2:

when i install dependencies by pip install -r requirement.txt. it works well.

pip install -e '.[dev]' by setup.py it crashed with error:

 s1 = os.stat(f1)
    FileNotFoundError: [WinError 3] Le chemin d▒acc▒s sp▒cifi▒ est introuvable: '/c/Users/XIAOZHO/digibiz/DE/digibiz_data_validation'

lazka · 2021-01-31T19:29:09Z

We've patched this now in MSYS2 by calling git in a way so it outputs relative paths: https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-python-setuptools-scm/0001-git-Use-show-prefix-instead-of-show-toplevel-for-fin.patch

If this is something that's considered acceptable I can try to upstream it.

RonnyPfannschmidt · 2021-01-31T19:44:22Z

Can you open a pr for more detailed discussion

I believe it looks okish enough

This should fix Cygwin type git which will return posix path when `git rev-parse --show-toplevel` is called which isn't parsed properly. This call `git rev-parse --show-prefix` which return relative paths which is later parsed. Fixes pypa#415

this takes the idea from https://github.com/msys2/MINGW-packages/blob/bf4d6f85053b66db09977ecffac0c1b3db735a0f/mingw-w64-python-setuptools-scm/0001-git-Use-show-prefix-instead-of-show-toplevel-for-fin.patch and simplified the logic additionally msys2 testing is added Fixes #415

RonnyPfannschmidt added the help wanted label Mar 30, 2020

wjblanke added a commit to Chia-Network/setuptools_scm that referenced this issue Apr 1, 2020

https://github.com/pypa/setuptools_scm/issues/415

c3f4c28

RonnyPfannschmidt added the deferred label May 2, 2020

RonnyPfannschmidt mentioned this issue Jun 21, 2020

Relying on git rev-parse output fragile on Cygwin+Conda installations #456

Closed

This was referenced Jan 28, 2021

Add support for MSVC cocotb/cocotb#1798

Merged

[Tracking] Support installing to msys2 environment cocotb/cocotb#2378

Closed

[Tracking] Support installing to mingw environments cocotb/cocotb#2379

Closed

naveen521kk mentioned this issue Feb 8, 2021

Use show prefix instead of show toplevel for finding #520

Merged

RonnyPfannschmidt closed this as completed in #520 Feb 9, 2021

dependabot bot mentioned this issue Mar 18, 2021

Update setuptools-scm requirement from <6.0 to <7.0 Backblaze/b2-sdk-python#207

Closed

tkalmar mentioned this issue Oct 7, 2021

Fix #1025 using git rev-parse to get the relative path inside git repo lint-staged/lint-staged#1026

Merged

cr1901 mentioned this issue Sep 5, 2023

NotADirectoryError: [WinError 267] The directory name is invalid when installing via pip install -e . using msys2 git. pdm-project/pdm-backend#197

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect POSIX-style paths on Windows #415

Detect POSIX-style paths on Windows #415

cr1901 commented Mar 29, 2020 •

edited

Loading

RonnyPfannschmidt commented Mar 30, 2020 •

edited

Loading

cr1901 commented Mar 30, 2020

RonnyPfannschmidt commented Mar 30, 2020

wjblanke commented Apr 1, 2020

cr1901 commented Apr 2, 2020

wjblanke commented Apr 2, 2020

cr1901 commented Apr 6, 2020

RonnyPfannschmidt commented May 2, 2020

cr1901 commented May 2, 2020

RonnyPfannschmidt commented May 2, 2020

xiaozhongtian commented Oct 13, 2020 •

edited

Loading

lazka commented Jan 31, 2021

RonnyPfannschmidt commented Jan 31, 2021

Detect POSIX-style paths on Windows #415

Detect POSIX-style paths on Windows #415

Comments

cr1901 commented Mar 29, 2020 • edited Loading

RonnyPfannschmidt commented Mar 30, 2020 • edited Loading

cr1901 commented Mar 30, 2020

RonnyPfannschmidt commented Mar 30, 2020

wjblanke commented Apr 1, 2020

cr1901 commented Apr 2, 2020

wjblanke commented Apr 2, 2020

cr1901 commented Apr 6, 2020

RonnyPfannschmidt commented May 2, 2020

cr1901 commented May 2, 2020

RonnyPfannschmidt commented May 2, 2020

xiaozhongtian commented Oct 13, 2020 • edited Loading

lazka commented Jan 31, 2021

RonnyPfannschmidt commented Jan 31, 2021

cr1901 commented Mar 29, 2020 •

edited

Loading

RonnyPfannschmidt commented Mar 30, 2020 •

edited

Loading

xiaozhongtian commented Oct 13, 2020 •

edited

Loading