Git Tab Completion Slow on Larger repositories for certain commands #1533

nwhansen · 2018-03-02T21:16:41Z

I was not able to find an open or closed issue matching what I'm seeing

Setup

Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

64 bit installer
git version 2.16.1.windows.4
cpu: x86_64
built from commit: ef6d451bbfef86a529ebf12620289e0f15a93d8e
sizeof-long: 4

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
Windows 10

Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.

What options did you set as part of the installation? Or did you choose the
defaults?

Editor Option: VIM
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

Any other interesting things about your environment that might be related
to the issue you're seeing?

This repository is very large. 5gb in size however this is replicatable with linux kernal repository. I am not using a solid state for the repository.

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.

git status -- ar[TAB]

Or

git mv ar[TAB]

What did you expect to occur after running these commands?
Tab completed to

git status -- arch/

What actually happened instead?
After approximately 10-20 seconds it completes
If the problem was occurring with a specific repository, can you provide the
URL to that repository to help us with testing?
Reproduced using: https://github.com/torvalds/linux

h3. Additional investigation:
Enabling

GIT_TRACE_PERFORMANCE=<logfile>

Creates the following log entries

13:59:59.608590 trace.c:417             performance: 0.002119112 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'rev-parse' '--git-dir' '--is-inside-git-dir' '--is-bare-repository' '--is-inside-work-tree' '--short' 'HEAD'
14:00:16.755165 trace.c:417             performance: 0.001872594 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'config' 'status.showUntrackedFiles'
14:00:28.706112 trace.c:417             performance: 11.796146578 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' '-C' '.' 'ls-files' '--exclude-standard' '--cached' '--directory' '--no-empty-directory' '--others'

Running the command

"C:\Program Files\Git\mingw64\bin\git.exe" -C ./ ls-files --exclude-standard --cached --directory --no-empty-directory --others

Returns 62913 lines. Attached as ls-files-result.txt
ls-files-result.txt

Git status on freshly checkout respository is the file checkoutStatus.txt
checkoutStatus.txt

Some further investigation shows that running set -x to gain some insight into the completion script shows that we spend a lot of time opening the completion script with

++case "$file" in
++echo drivers
++read -r file

being repeated what seems thousands of times.

The text was updated successfully, but these errors were encountered:

drizzd · 2018-03-10T12:16:21Z

I can confirm your observation on Windows 10 64-bit with Msys2 mingw64 git version 2.15.1.windows.2:

~/src/linux$ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git

real    0m11.876s
user    0m4.685s
sys     0m6.808s

Most of the time is spent in the while read loop in __git_index_files. We can improve much by replacing the loop with an sed script:

~/src/linux$ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time
_git

real    0m1.372s
user    0m0.263s
sys     0m0.167s

Here's the change:

diff --git a/.git-completion.bash b/.git-completion-mod.bash
index a54cc2c..f20945f 100644
--- a/.git-completion.bash
+++ b/.git-completion-mod.bash
@@ -351,12 +351,7 @@ __git_index_files ()
        local root="${2-.}" file

        __git_ls_files_helper "$root" "$1" |
-       while read -r file; do
-               case "$file" in
-               ?*/*) echo "${file%%/*}" ;;
-               *) echo "$file" ;;
-               esac
-       done | sort | uniq
+       sed -e '/^\//! s#/.*##' | sort | uniq
 }

 # Lists branches from the local repository.

I cannot reproduce the issue in WSL. Maybe the mingw64 version of bash has a performance issue with while read loops.

drizzd · 2018-03-11T11:42:25Z

I have prepared a branch with the changes here https://github.com/drizzd/git/tree/completion-optimize-ls-files-filter. Still need to setup an email client to contribute this to the git mailing list.

dscho · 2018-03-13T16:00:38Z

@drizzd if you want, I can contribute this for you. Or you can follow @derrickstolee's excellent guide and set up git send-email: https://github.com/git-for-windows/git/blob/master/CONTRIBUTING.md#submit-your-patch

dscho · 2018-03-13T16:00:59Z

Maybe the mingw64 version of bash has a performance issue with while read loops.

FWIW this is very likely...

nwhansen · 2018-04-03T17:43:46Z

I modified the git completion script to include that change and the performance difference is quite significant. I will continue to test out the change and and report further findings.

drizzd · 2018-04-04T07:28:36Z

Thanks. Please note that the patch has been discussed on the mailing list: https://public-inbox.org/git/[email protected]/

The patch removes a sorting operation, which is wrong. I have a prepared a patch with the fix here:
https://github.com/drizzd/git/commits/completion-optimize-ls-files-filter4.

dscho · 2018-04-04T10:36:44Z

@drizzd do you think the updated patch is ready to be cherry-picked into Git for Windows?

nwhansen · 2018-04-04T14:26:53Z

My apologies I am not currently following that mailing list. I appreciate the effort and time looking into this bug report :)

dscho · 2018-04-04T14:44:40Z

My apologies I am not currently following that mailing list.

Only start following it if you like drinking from a fire hose...

drizzd · 2018-04-04T16:31:04Z

@dscho It's working fine for me on Windows. But who knows if there are more comments on the list. I can make a note for you here when the patch is accepted.

From the output of ls-files, we remove all but the leftmost path component and then we eliminate duplicates. We do this in a while loop, which is a performance bottleneck when the number of iterations is large (e.g. for 60000 files in linux.git). $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m11.876s user 0m4.685s sys 0m6.808s Replacing the loop with the cut command improves performance significantly: $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m1.372s user 0m0.263s sys 0m0.167s The measurements were done with Msys2 bash, which is used by Git for Windows. When filtering the ls-files output we take care not to touch absolute paths. This is redundant, because ls-files will never output absolute paths. Remove the unnecessary operations. The issue was reported here: git-for-windows#1533 Signed-off-by: Clemens Buchacher <[email protected]> Signed-off-by: Junio C Hamano <[email protected]>

dscho · 2018-04-25T11:52:32Z

A new snapshot is building right now and should become available in about half an hour: https://wingit.blob.core.windows.net/files/index.html

dscho · 2018-04-25T11:53:08Z

(Please test, the easiest way is to install the portable Git, start its git-bash.exe and verify that the tab completion is faster in your repository.)

Tab completion of `git status -- <partial-path>` [is now a lot faster](git-for-windows/git#1533). Signed-off-by: Johannes Schindelin <[email protected]>

dscho · 2018-04-28T12:10:03Z

Please test

Can y'all please test?

nwhansen · 2018-04-28T14:00:22Z

It is working so far. The performance difference is outstanding, the native commands are faster but it is much harder to tell the difference. and they are well within a second of each other.

The double tab list is in the same order so I don't know if there is anything else I can really test. Thanks so much for getting a fix for this @drizzd @dscho

dscho · 2018-04-29T10:52:28Z

The performance difference is outstanding

That is really what I was hoping.

The double tab list is in the same order

Very good. Thank you for verifying, this is really crucial information to have confidence in this improvement.

From the output of ls-files, we remove all but the leftmost path component and then we eliminate duplicates. We do this in a while loop, which is a performance bottleneck when the number of iterations is large (e.g. for 60000 files in linux.git). $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m11.876s user 0m4.685s sys 0m6.808s Replacing the loop with the cut command improves performance significantly: $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m1.372s user 0m0.263s sys 0m0.167s The measurements were done with Msys2 bash, which is used by Git for Windows. When filtering the ls-files output we take care not to touch absolute paths. This is redundant, because ls-files will never output absolute paths. Remove the unnecessary operations. The issue was reported here: #1533 Signed-off-by: Clemens Buchacher <[email protected]> Signed-off-by: Junio C Hamano <[email protected]>

Shell completion (in contrib) that gives list of paths have been optimized somewhat. * cb/bash-completion-ls-files-processing: completion: improve ls-files filter performance This fixes #1533 Signed-off-by: Johannes Schindelin <[email protected]>

From the output of ls-files, we remove all but the leftmost path component and then we eliminate duplicates. We do this in a while loop, which is a performance bottleneck when the number of iterations is large (e.g. for 60000 files in linux.git). $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m11.876s user 0m4.685s sys 0m6.808s Replacing the loop with the cut command improves performance significantly: $ COMP_WORDS=(git status -- ar) COMP_CWORD=3; time _git real 0m1.372s user 0m0.263s sys 0m0.167s The measurements were done with Msys2 bash, which is used by Git for Windows. When filtering the ls-files output we take care not to touch absolute paths. This is redundant, because ls-files will never output absolute paths. Remove the unnecessary operations. The issue was reported here: #1533 Signed-off-by: Clemens Buchacher <[email protected]> Signed-off-by: Junio C Hamano <[email protected]>

Shell completion (in contrib) that gives list of paths have been optimized somewhat. * cb/bash-completion-ls-files-processing: completion: improve ls-files filter performance This fixes #1533 Signed-off-by: Johannes Schindelin <[email protected]>

dscho added this to the v2.17.0(2) milestone Apr 25, 2018

dscho closed this as completed in 8dd7bac Apr 25, 2018

dscho added a commit to git-for-windows/build-extra that referenced this issue Apr 25, 2018

Mention Bug Fix in release notes

33ac105

Tab completion of `git status -- <partial-path>` [is now a lot faster](git-for-windows/git#1533). Signed-off-by: Johannes Schindelin <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Git Tab Completion Slow on Larger repositories for certain commands #1533

Git Tab Completion Slow on Larger repositories for certain commands #1533

nwhansen commented Mar 2, 2018 •

edited

Loading

drizzd commented Mar 10, 2018

drizzd commented Mar 11, 2018

dscho commented Mar 13, 2018

dscho commented Mar 13, 2018

nwhansen commented Apr 3, 2018

drizzd commented Apr 4, 2018

dscho commented Apr 4, 2018

nwhansen commented Apr 4, 2018

dscho commented Apr 4, 2018

drizzd commented Apr 4, 2018

dscho commented Apr 25, 2018

dscho commented Apr 25, 2018

dscho commented Apr 28, 2018

nwhansen commented Apr 28, 2018

dscho commented Apr 29, 2018

Git Tab Completion Slow on Larger repositories for certain commands #1533

Git Tab Completion Slow on Larger repositories for certain commands #1533

Comments

nwhansen commented Mar 2, 2018 • edited Loading

Setup

Details

drizzd commented Mar 10, 2018

drizzd commented Mar 11, 2018

dscho commented Mar 13, 2018

dscho commented Mar 13, 2018

nwhansen commented Apr 3, 2018

drizzd commented Apr 4, 2018

dscho commented Apr 4, 2018

nwhansen commented Apr 4, 2018

dscho commented Apr 4, 2018

drizzd commented Apr 4, 2018

dscho commented Apr 25, 2018

dscho commented Apr 25, 2018

dscho commented Apr 28, 2018

nwhansen commented Apr 28, 2018

dscho commented Apr 29, 2018

nwhansen commented Mar 2, 2018 •

edited

Loading