Some repositories fail to get status when using GitStatusCache

[tmercswims]

System Details

• posh-git version/path: 1.0.0 beta4 ~\Documents\PowerShell\Modules\posh-git\1.0.0
• PowerShell version: 7.0.0
• git version 2.26.0.windows.1 hub version 2.14.2
• OS: Microsoft Windows NT 10.0.18363.0

Issue Description

I have git-status-cache-posh-client installed and working. I have posh-git configured to use it ($GitPromptSettings.EnableFileStatusFromCache = $true).

I also have posh-git integrated into my prompt. I use it to print status information to a PowerLine block in my PowerShell profile like this:

# ...
{ New-PromptText { & { Write-VcsStatus } 6>&1 } -BackgroundColor Yellow -ForegroundColor Black },
# ...

Side note: I'm unsure if there is a better way to do this, as this was the only way I could see to get the repository information in the "prompt format" while also taking advantage of the caching. Either way, the end result, most of the time, looks like this (name blacked because it's a private org repo, sorry 😬):

However, if I clone or create a new repository on my system, then it looks like this:

If I do Get-GitStatus then I get a little bit more detail:

$ Get-GitStatus
InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.

InvalidOperation: C:\Users\thomas.mercurio\Documents\PowerShell\Modules\posh-git\1.0.0\GitUtils.ps1:179
Line |
 179 |              $hash[$path] = 1
     |              ~~~~~~~~~~~~~~~~
     | Index operation failed; the array index evaluated to null.


HasUntracked : False
Upstream     :
Index        : {}
UpstreamGone :
GitDir       : C:\Users\thomas.mercurio\dev\code\tmercswims\dotfiles\.git
Working      : {}
Branch       : master
HasWorking   : False
StashCount   : 0
HasIndex     : False
RepoName     : dotfiles
BehindBy     :
AheadBy      :

And Get-GitStatusFromCache just bombs:

$ Get-GitStatusFromCache


Version Error
------- -----
      1 Failed to retrieve status of git repository at provided 'Path'.

I know that this is due to the caching because if I turn it off, then everything works fine. But turning it back on again results in the same problem.

I have been living with it for a long time like this, well before PowerShell 7, so I know that has nothing to do with it. I can also reproduce it on more than one machine, though they are all set up pretty much identically - same PowerShell profile, same GitStatusCache.exe version (the latest release), etc. Rebooting, re-cloning, removing and reinstalling git-status-cache-posh-client, and essentially every other normal troubleshooting tactic have not helped.

It's also definitely fair to say that this issue might be somewhere in the caching stack, and not with posh-git itself. I am certainly happy to open an issue somewhere else if it's deemed appropriate. I guess I gravitated toward filing the issue here because the git-status-cache and git-status-cache-posh-client repositories both seem pretty lifeless at the moment.

[dahlbyk]

I guess I gravitated toward filing the issue here because the git-status-cache and git-status-cache-posh-client repositories both seem pretty lifeless at the moment.

git status doesn't really change often, so it's reasonable to consider them "done" unless we find a bug. We can loop in @cmarcusreid if necessary.

The error looks like it's in here:

posh-git/src/GitUtils.ps1

Lines 174 to 181 in 1a49d26

	function GetUniquePaths($pathCollections) {
	$hash = New-Object System.Collections.Specialized.OrderedDictionary
	foreach ($pathCollection in $pathCollections) {
	foreach ($path in $pathCollection) {
	$hash[$path] = 1
	}
	}

posh-git/src/GitUtils.ps1

Lines 348 to 349 in 1a49d26

	$indexPaths = @(GetUniquePaths $indexAdded,$indexModified,$indexDeleted,$indexUnmerged)
	$workingPaths = @(GetUniquePaths $filesAdded,$filesModified,$filesDeleted,$filesUnmerged)

It seems likely the cache version ends up with a $null value in one of those file path collections. Can you see the problem if you look at Get-GitStatusFromCache directly?

All the cache response paths are processed similarly; we should probably replace the $castStringSeq.Invoke(...) with a PowerShell function in which we can also fix that $null values are being included:

posh-git/src/GitUtils.ps1

Lines 257 to 271 in 1a49d26

	$indexAdded.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.IndexAdded))))
	$indexModified.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.IndexModified))))
	foreach ($indexRenamed in $cacheResponse.IndexRenamed) {
	$indexModified.Add($indexRenamed.Old)
	}
	$indexDeleted.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.IndexDeleted))))
	$indexUnmerged.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.Conflicted))))
	$filesAdded.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.WorkingAdded))))
	$filesModified.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.WorkingModified))))
	foreach ($workingRenamed in $cacheResponse.WorkingRenamed) {
	$filesModified.Add($workingRenamed.Old)
	}
	$filesDeleted.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.WorkingDeleted))))
	$filesUnmerged.AddRange($castStringSeq.Invoke($null, (,@($cacheResponse.Conflicted))))

[dahlbyk]

The solution might be as simple as using OfType instead of Cast, which should automatically exclude $null.

[tmercswims]

If I replace Cast with OfType, like this:

- $castStringSeq = [Linq.Enumerable].GetMethod("Cast").MakeGenericMethod([string])
+ $castStringSeq = [Linq.Enumerable].GetMethod("OfType").MakeGenericMethod([string])

Then it definitely works better, but still not totally correctly:

There's no error and the branch is there now, but the rest is missing. The comparison to the remote branch, file status, etc. Right now I have a remote tracking branch and an unstaged change:

But those aren't reflected.

If I run Get-GitStatusFromCache manually, I get this:

Which is most likely an issue. I've tried Restart-GitStatusCache but that didn't seem to improve anything.

[dahlbyk]

If I run Get-GitStatusFromCache manually, I get this

Sorry, just noticed you said that in the issue. How about Get-GitStatusCacheStatistics?

cc @cmarcusreid

[tmercswims]

Sure, here you go:

[cmarcusreid]

Bummed to hear you're hitting this issue!

If you set $global:GitStatusCacheLoggingEnabled = $true and call Restart-GitStatusCache, it'll create a log file at %temp%\GitStatusCache_[timestamp].log. There may be a hint as to what's going wrong there.

That error seems to suggest that one of the calls to libgit2 failed. See Git::GetStatus in https://github.com/cmarcusreid/git-status-cache/blob/e1ea01573e7b248c77fcb1876f04cf2f6bee2d26/src/GitStatusCache/src/Git.cpp if interested in the implementation.

This code hasn't changed since 2017, but it's very possible there's something about your repository that this code can't handle. The log file would be a great starting point for investigation. I'd also be interested if you have a minimal repository that reproduces the issue.

[tmercswims]

Thank you for jumping in!

I followed your steps, and right after got a new prompt, to force the call which appears to be failing. Here is the log file.

As for the second question, I don't have a repository which consistently reproduces this problem. In fact, the same exact repository (the one I've been using here) fails on one of my machines but works on the other. The only difference at all between them might be that on the one where it works, the repository had been cloned before I had GitStatusCache as part of the setup, and on the one where it doesn't, it was cloned after. It's "might be" because I honestly don't really remember. 😅

I can consistently reproduce the problem with any repository which I clone right now, in fact. It doesn't matter how old the repository itself is, if I clone (or re-clone) it right this moment, it will be broken.

[cmarcusreid]

Here's the error:

2020-04-06 22:34:51.533328	GitStatusCache.exe (0x00008404)	0x00006154	Spam	Git.GetFileStatus.377	Call completed. Reporting timing. { "milliseconds": 2.2844, "call": "git_status_list_new(&statusList.get(), repository.get(), &statusOptions)" }
2020-04-06 22:34:51.533328	GitStatusCache.exe (0x00008404)	0x00006154	Error	Git.GetFileStatus.FailedToCreateStatusList	Failed to create status list. { "repositoryPath": "C:/Users/thomas.mercurio/dev/code/tmercswims/dotfiles/.git/", "result": "Generic Error", "lastError": "Invalid data in index - incorrect header version" }

I see libgit2/libgit2@5625d86, and I am guessing that git-status-cache requires an update to a more recent version of libgit2.

[dahlbyk]

Maybe try fiddling with index.version and/or feature.manyFiles?

[tmercswims]

That was it! I have feature.manyFiles turned on in my ~/.gitconfig, which makes the index version 4. That also explains why it was working for old repositories - it wasn't about when they were cloned relative to caching being on, it was about when they were cloned relative to feature.manyFiles being on.

Confirmed by setting it to false and then cloning a repository - everything works as expected! In addition, it works without the modification to GitUtils.ps1 that we did from above. So that change may or may not be necessary?

I hope that git-status-cache will be updated to support the new index version, but my immediate issue is solved by simply keeping feature.manyFiles off for the time being.

Thank you both for your help, I really appreciate it!

[cmarcusreid]

I'm attempting the update now, but building libgit2 is proving tricky. I'll update when a release is available for your testing.

[cmarcusreid]

I just published the v1.1.6 release of git-status-cache. This release updates libgit2 to v1.0.0. You can update by calling Update-GitStatusCache, after which repositories cloned with feature.manyFiles should work.

[dahlbyk]

In addition, it works without the modification to GitUtils.ps1 that we did from above. So that change may or may not be necessary?

Now that we know what an Error cache response looks like, it wouldn't be terrible to handle that more gracefully than a Hashtable error.

[cmarcusreid]

I agree. I will put together a PR to add a nicer error on the posh-git side later today.

[tmercswims]

I just tested using version 1.1.6, and everything is working as expected! Both existing clones which had previously been cloned with feature.manyFiles = true and brand new clones with feature.manyFiles = true.

Thank you again for the quick response and fix on this!

[cmarcusreid]

@dahlbyk: #759

[dahlbyk]

Thanks for the report and help troubleshooting!