sparse-checkout does not appear to work on windows. "error: invalid path "

[wdtj]

When I attempt to checkout a repository from github I get the error:

error: invalid path 'configs/perl-modules/DIST.64/perl-HTML-Tree-1:5.03-1.el6.noarch.rpm'

I suspect the issue is that the path contains a : which is illegal on Windows.

After researching the error, I've found 2 possible answers:
1) Change the path on the repository file. Unfortunately, this is is a team resource and can not be fixed in the foreseeable future.
2) Use sparse-checkout. I've tried this with no effect as evidenced in the following:

$ git clone -n [email protected]:XXXXXX/deploy.git          [[[Name obscured because it is a private repo.]]]
Cloning into 'deploy'... 
    remote: Enumerating objects: 57, done.
    remote: Counting objects: 100% (57/57), done.
    remote: Compressing objects: 100% (49/49), done.
    remote: Total 86457 (delta 10), reused 22 (delta 8), pack-reused 86400
    Receiving objects: 100% (86457/86457), 1.50 GiB | 4.73 MiB/s, done.
    Resolving deltas: 100% (59779/59779), done.
    $ cd deploy/ 
    $ git config core.sparsecheckout true
    $ echo www >> .git/info/sparse-checkout
    $ git checkout centos6
    error: invalid path 'configs/perl-modules/DIST.64/perl-HTML-Tree-1:5.03-1.el6.noarch.rpm'
    error: invalid path 'configs/perlbrew/perls/perl-5.24.1/man/man3/App::Cpan.3'
    ... (repeat for many files in the above path)

This was done with Git for Windows "git version 2.28.0.windows.1". I have also tried both types of line endings and using various version of .git/info/sparse-checkout such as:

    *
    !/configs/perl-modules/
    !/configs/perlbrew/perls/perl-5.24.1/man/man3

Checkout works fine on Linux, MacOS and WSL, only problem is that my IDEs don't work in those environments. Why isn't sparse-checkout working on Windows. Is there any other possibilities? Given the number of hits I have encountered searching with Google, maybe the there should be an enhancement to git or fir-for-windows to allow filepath mapping.

Further dialog is ocuring at https://stackoverflow.com/questions/63727594/github-git-checkout-returns-error-invalid-path-on-windows

[dscho]

This has come up before; the path names are checked when the corresponding files enter the Git index, even if the entries are not written out. See #2777 for more information (of which I think this here ticket is a duplicate).

[PhilipOakley]

issue is that the path contains a : which is illegal on Windows.

As an aside, I wondered if the clean and smudge style filters may offer a route to at least creating a valid local filename/path that could at least allow users to view the files. Maybe the file in the filesystem gets some sort of a symref/gitlink line to show it's 'real' path (and act as a pseudo flag for the smudge/clean filter). 'real' path being the repository path, rather than the local FS path.

The 'problem' at the moment is that Git retains the filename/pathname, which makes it a no-go, but it could be sold to some in the linux community as a mechanism for avoiding having Word.doc files pollute their pristine FS, being replaced by a panadoc variant 😉 .

just a thought. probably too many obstacles for the path mapping.

[wdtj]

@dscho was correct that#2777 was my issue and offers a bypass to the problem. At stated in the other issue: I've spent about 2 days trying to resolve the issue. I whole hardheartedly suggest that something be changed to prevent this in the future. Either a warning to set core.protectNTFS when sparse-checkout is specified or similar an enhancement to remap bad filepaths.

With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to accept this issue or reopen #2777 .

[dscho]

With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to accept this issue or reopen #2777 .

I will leave this ticket open all right, at the same time I encourage you to think about ways how you could bring the change about that you want to see.

To give you a couple of pointers:

• https://github.com/git/git/blob/v2.28.0/read-cache.c#L967 is the location of the call to is_valid_path().

• That is_valid_path() is actually a macro that points to is_valid_win32_path() on Windows: https://github.com/git/git/blob/v2.28.0/compat/mingw.h#L474

• The is_valid_win32_path() function skips any validation when core.protectNTFS is set to false: https://github.com/git/git/blob/v2.28.0/compat/mingw.c#L2602

• The verify_path() function is actually called in quite a few places: https://github.com/git/git/search?q=verify_path&unscoped_q=verify_path

• The index entries corresponding to the files that are excluded from the sparse checkout are marked with the CE_SKIP_WORKTREE bit: https://github.com/search?q=%22CE_SKIP_WORKTREE%22+repo%3Agit%2Fgit&type=Code&ref=advsearch&l=&l=

• If I read the code correctly, it is the apply_sparse_checkout() function (https://github.com/git/git/blob/v2.28.0/unpack-trees.c#L492-L554) that ultimately determines whether an index entry (data type struct cache_entry, for hysterical raisins) is to be skipped from the worktree or not.

Note: the index entries are created only if their corresponding paths pass the verify_path() test, and only quite a bit later do we actually know whether they have the CE_SKIP_WORKTEE bit set.

Also note: the verify_path() test is completely bypassed when reading an index file from disk. In other words, the test is only performed when a file is added to the index.

... is to figure out how to delay the call to is_valid_path() safely in the make_cache_entry()/make_transient_cache_entry() code paths (and only from those code paths, not from, say, apply.c's check_unsafe_path() code path!) to the time when the file is to be written (i.e. when a file is not marked with CE_SKIP_WORKTREE nor CE_NEW_SKIP_WORKTREE at all, or when it was marked with CE_SKIP_WORKTREE but now is no longer marked with that bit).

This, of course, has to be done with great care, as any regression in the safeguards would open users up for vulnerabilities that are no longer unknown (because they have been fixed as part of Git v2.24.1).

[kaovilai]

@dscho What about using WSL git directly?
https://docs.microsoft.com/en-us/windows/wsl/interop#run-linux-tools-from-a-windows-command-line
Like replacing use of git.exe [param] with wsl.exe git [param]
This would resolve the issue for my use case. I just need to learn a way to do it.
I asked a question on Superuser about this.

[kaovilai]

I can imagine Git-for-Windows becoming a proxy to git in WSL. It would resolve this issue. All interoperability between different file systems will then be delegated to WSL team. Maybe as an experimental flag.

[dscho]

What about using WSL git directly?

What about it? If it works for you, that's good. In general, it is not really a solution for this here ticket.

[ExE-Boss]

Git for Windows should probably use the U+F000 – U+F0FF range to escape the reserved characters, like what WSL does¹²³.

Footnotes

1. Enable filenames with colon (:) on drvfs microsoft/WSL#1514 ↩

2. When browsing a WSL folder in Explorer, there's no way to name a file/folder with some characters microsoft/WSL#5515 ↩

3. When browsing a WSL directory, some characters are not properly displayed microsoft/WSL#5516 ↩

[dscho]

Git for Windows should probably use the U+F000 – U+F0FF range to escape the reserved characters, like what WSL does​microsoft/WSL#1514​microsoft/WSL#5515​microsoft/WSL#5516.

And how would that help? Git would now understand those, but everything else, most notably the build system, would not.

Which means that Git would all of a sudden pretend that it succeeded, but the users would now be faced with a totally broken checkout that none of their tooling handles (if they used Cygwin for their tooling, they would most likely use Cygwin's git.exe, not Git for Windows').

[trusktr]

I'm getting this error on this path: damage/damage/util/prn.py. But, as you can see, there doesn't seem to be any invalid character there.

I was able to run git config core.protectNTFS false to work around the issue, but it doesn't seem like that should be needed in this case.

This happened to me after cloning a repo in windows. After clone, all files were deleted and all deletions staged, then this fails:

PS C:\Users\user\path\to\project> git reset .
error: invalid path 'damage/damage/util/prn.py'
fatal: make_cache_entry failed for path 'damage/damage/util/prn.py'

and this works:

PS C:\Users\user\path\to\project> git config core.protectNTFS false
PS C:\Users\user\path\to\project> git reset .
Unstaged changes after reset:
D       damage/damage/util/prn.py
...

Now if I turn it back on and try to checkout the root:

PS C:\Users\user\path\to\project> git config core.protectNTFS true
PS C:\Users\user\path\to\project> git checkout .
error: unable to create file damage/damage/util/prn.py: Invalid argument
Updated 85 paths from the index

turning it back off does not work this time:

PS C:\Users\user\path\to\project> git config core.protectNTFS false
PS C:\Users\user\path\to\project> git checkout .
error: unable to create file damage/damage/util/prn.py: No such file or directory
Updated 1 path from the index
PS C:\Users\jopea\src\nasa+dsa+dsa_modeling> git status
...elided...
        deleted:    damage/damage/util/prn.py

Is this a related to this issue? Or a different one?

[rimrul]

I'm getting this error on this path: damage/damage/util/prn.py. But, as you can see, there doesn't seem to be any invalid character there.

Not an invalid character, but an invalid name. PRN and a few other names can't be used as filenames (with or without file extensions like .py)

(CON, PRN, AUX, CLOCK$, NUL COM0, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9)

[trusktr]

Ah, thank you!

[jtmoon79]

Problem

Currently causes failures during github action uses: action/checkout@v2 on os: windows-latest.

Reproduction steps

I reproduced the error.

1. in a Windows Sandbox
2. install git for windows Git-2.38.1-64-bit.exe
3. open MinGW Git shell
4. commands:

$ mkdir super-speedy-syslog-searcher

$ git.exe init ./super-speedy-syslog-searcher/
Initialized empty Git repository in C:/Users/WDAGUtilityAccount/Temp/super-speedy-syslog-searcher/.git/

$ cd super-speedy-syslog-searcher/

$ git.exe remote add origin https://github.com/jtmoon79/super-speedy-syslog-searcher

$ git.exe config --local gc.auto 0

$ git.exe  config --local --name-only --get-regexp core\.sshCommand

$ git.exe submodule foreach --recursive "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"

$ git.exe config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader

$ git.exe submodule foreach --recursive "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"

$ git.exe  -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +371df442ef7ea8d8c77c4039990774144d0e8e11:refs/remotes/origin/main
remote: Enumerating objects: 621, done.
remote: Counting objects: 100% (621/621), done.
remote: Compressing objects: 100% (521/521), done.
remote: Total 621 (delta 50), reused 548 (delta 47), pack-reused 0
Receiving objects: 100% (621/621), 1.48 MiB | 7.07 MiB/s, done.
Resolving deltas: 100% (50/50), done.
From https://github.com/jtmoon79/super-speedy-syslog-searcher
 * [new ref]         371df442ef7ea8d8c77c4039990774144d0e8e11 -> origin/main

$ git.exe checkout --progress --force -B main refs/remotes/origin/main
error: invalid path 'logs/Ubuntu18/samba/log.'

(those git.exe copied from the build output in the embedded picture)

Workaround

According to SO Answer https://stackoverflow.com/a/63742773/471376 , a workaround is git config core.protectNTFS false.

I started a Q&A Discussion for calling git config before git checkout actions/checkout#976

Successful fix is here jtmoon79/super-speedy-syslog-searcher@2598719

[dscho]

According to SO Answer https://stackoverflow.com/a/63742773/471376 , a workaround is git config core.protectNTFS false.

For some definition of "workaround".

It's like telling somebody to simply throw away their front door just because it creaks. Yes, the door won't creak anymore. But there are other consequences, too.

[bberberov]

I have run into this issue as well, since I have filenames that contain a :

Right now I'm trying to run with this git config:

[core]
    protectNTFS = false
    longpaths = true

Checkout (git worktree add ...) will succeed on NTFS, but the filenames are truncated to right before the :. Not really workable for me.

I tested using UDF as well, formatted with format D: /FS:UDF /Q, but that did not work either.

Interestingly, running touch test:test in the bash shell succeeds just fine.

Is there any way to successfully checkout (git worktree add ...) files with special characters, not reserved names like aux, con, etc?

[dscho]

I have run into this issue as well, since I have filenames that contain a :

That's because : is not a legal character in Win32 file names, see e.g. https://cygwin.com/cygwin-ug-net/using-specialnames.html for an explanation.

Checkout (git worktree add ...) will succeed on NTFS, but the filenames are truncated to right before the :. Not really workable for me.

The file name is not really truncated before the :, but that character is interpreted as separator between the actual file name and the name of the Alternate Data Stream.

Interestingly, running touch test:test in the bash shell succeeds just fine.

That's because the Bash is not really a Win32 program. It is an MSYS program that derives certain hacks from Cygwin. For example the hack that characters that are not allowed in file names are automatically 'translated" into a private Unicode page. That works if you access the file via another MSYS program. But if you try to open that file from a regular Win32 using the name containing the : character, it will fail.

So no, there is no good workaround. The problem is that there is no way to write files with names containing colons on Windows, and trying to do that anyway can only lead to a steady source of problems.

[dscho]

I tested using UDF as well, formatted with format D: /FS:UDF /Q, but that did not work either.

The colon character is not allowed in file names on Windows. That is an operating system limitation, not a file system limitation. You could actually write a file whose name contains a colon on an NTFS file system (say, by mounting the file system in Linux, or using other tricks). But that would still not make it an allowed file name on Windows.

[bberberov]

Thank you for the detailed and clear explanation! 👍

The PUA (Private Use Area) workaround is interesting. Certainly explains what I observed with File Explorer after touch test:test. Maybe a Cygwin-only environment is more appropriate for what I, and others, are trying/need to do. My workplace has a lot of restrictions, around OS and software, that lead us to look for unusual workarounds to situations like this.