connectivity issues with github. https silently fails but SSH works fine

[ghost]

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git clone https://github.com/19379/wsh_marc2003.git test
Cloning into 'test'...

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git ls-remote https://github.com/19379/wsh_marc2003.git

the cursor flashes for a few seconds then nothing. i tried to enable GIT_TRACE but i don't think it's helping...

User@HP-JMHBEJRPM4NE MINGW32 ~
$ export GIT_TRACE=1

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git clone https://github.com/19379/wsh_marc2003.git test
19:22:04.306661 git.c:348               trace: built-in: git 'clone' 'https://github.com/19379/wsh_marc2003.git' 'test'
Cloning into 'test'...
19:22:05.753664 run-command.c:335       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/19379/wsh_marc2003.git'
User@HP-JMHBEJRPM4NE MINGW32 ~
$ git ls-remote https://github.com/19379/wsh_marc2003.git
19:22:12.062698 git.c:348               trace: built-in: git 'ls-remote' 'https://github.com/19379/wsh_marc2003.git'
19:22:12.067698 run-command.c:335       trace: run_command: 'git-remote-https' 'https://github.com/19379/wsh_marc2003.git' 'https://github.com/19379/wsh_marc2003.git'

i can connect to github using curl...

User@HP-JMHBEJRPM4NE MINGW32 ~
$ curl -i https://github.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/1.1 200 OK
Server: GitHub.com
Date: Mon, 14 Sep 2015 18:23:15 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Status: 200 OK
<snip>

and finally, i can clone with SSH just fine...

User@HP-JMHBEJRPM4NE MINGW32 ~
$ export GIT_TRACE=0

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git clone [email protected]:19379/wsh_marc2003.git test
Cloning into 'test'...
remote: Counting objects: 702, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 702 (delta 14), reused 0 (delta 0), pack-reused 670
Receiving objects: 100% (702/702), 1.82 MiB | 387.00 KiB/s, done.
Resolving deltas: 100% (433/433), done.
Checking connectivity... done.
Checking out files: 100% (90/90), done.

given SSH works i don't have any real issue but i'm really curious about no errors being reported.

just to muddy the waters a little, the pc is configured to use a proxy but that's only via group policy/Internet Explorer and a PAC file. any application configured with no proxy can access/download anything they like. although browsing in IE is restricted, github is an allowed site and because i don't have any issues with curl or SSH i don't think it's that.

OS: windows 7 embedded 32bit. maybe an issue with not having a full fat OS? i'm totally clueless about this sort of thing!!

[dscho]

What Git for Windows version is this? And what does echo $? show after the failed commands? Does it work if you export the environment variable CHARSET (e.g. to cp1252)?

[ghost]

sorry i forgot to mention it was the latest 2.5.2 release 2.

User@HP-JMHBEJRPM4NE MINGW32 ~
$ export GIT_TRACE=1

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git clone https://github.com/19379/wsh_marc2003.git test
20:18:30.101436 git.c:348               trace: built-in: git 'clone' 'https://github.com/19379/wsh_marc2003.git' 'test'
Cloning into 'test'...
20:18:31.696450 run-command.c:335       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/19379/wsh_marc2003.git'

User@HP-JMHBEJRPM4NE MINGW32 ~
$ echo $?
128

User@HP-JMHBEJRPM4NE MINGW32 ~
$ export CHARSET=cp1252

User@HP-JMHBEJRPM4NE MINGW32 ~
$ git clone https://github.com/19379/wsh_marc2003.git test
20:23:09.864230 git.c:348               trace: built-in: git 'clone' 'https://github.com/19379/wsh_marc2003.git' 'test'
Cloning into 'test'...
20:23:11.487241 run-command.c:335       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/19379/wsh_marc2003.git'

User@HP-JMHBEJRPM4NE MINGW32 ~
$ echo $?
128

[dscho]

128 most likely means: access violation...

[ghost]

well given it seems to be specific to only me and SSH is working, i'll close this.

thanks for the prompt replies.

[dscho]

Indeed, this is what I got:

$ git version
git version 2.5.2.windows.2

$ git clone https://github.com/19379/wsh_marc2003.git test
Cloning into 'test'...
remote: Counting objects: 702, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 702 (delta 14), reused 0 (delta 0Receiving objects: 100% (702/702)), pack-reused 670
Receiving objects: 100% (702/702), 1.82 MiB | 1.39 MiB/s, done.
Resolving deltas: 100% (433/433), done.
Checking connectivity... done.

However, I think it is a mistake to leave it at that. If you have that problem, so will others. This is your chance to contribute to the project.

[ghost]

i must confess i forgot to mention something else rather important. before using v2.5.2, i used this package of 1.9.5. this works fine over https on the same pc/internet connection.

https://github.com/sheabunge/GitPortable/releases/tag/v1.9.5-devtest.2

perhaps i could try some earlier 2.x builds and see if they work?

[dscho]

128 most likely means: access violation...

And since I cannot reproduce, I fear that you will have to hunt mostly on your own.

This is how I would start investigating, if I could reproduce:

1. install the Git for Windows SDK
2. verify that git ls-remote https://github.com/19379/wsh_marc2003.git still fails with exit code 128, even when running with the just-built Git.
3. modify the remote-curl.c file in /usr/src/git/ by inserting error("%s:%d: here", __FILE__, __LINE__); fflush(stderr); lines in strategic places (e.g. at the beginning and the end of the main() function), probably also around the curl_..._init() calls in http.c.
4. run make install in /usr/src/git/ and test again.

At some point, I would expect that the location of the crash could be pin-pointed to one particular line. This should be sufficient information to guess the true cause of the access violation.

[ghost]

right, i will try the above but it won't be for a few weeks. i don't have admin rights on the pc i use so i can't run any installer. i'm currently building my own pc but it won't be ready until the end of the month.

[dscho]

i don't have admin rights on the pc i use so i can't run any installer.

Did you know that you can unpack the SDK installer with .7z? It is set up that way so that you can run that installer by unpacking it and then running the setup-git-sdk.bat script inside.

[ghost]

cool. i was simply put off by the filename. i'll give that a go tonight.

edit: doh, just noticed 7z in the filename. i'll blame not being awake yet... or something.

[dscho]

Very good. I will help you as much as I can, of course. If you run into troubles, just add a comment here.

[ghost]

can i ask how long this takes to build? there's no way i can use the pc i have connectivity issues with because it's a thin client with the weakest CPU known to man. even browsing the web is a fairly painful experience on it!! i do have access to a decent spec pc at my local library but time there is limited and given how big this project is, i can imagine it taking some time. it has a quad core i5 if that helps.

also, will i get a choice of what to build when running the make file? i only want to build the 32bit portable install.

[dscho]

i do have access to a decent spec pc at my local library

Hmm. Given that I cannot reproduce this issue on any machine I have access to, I am rather doubtful that you can use that machine to investigate: simply switching the machine will make the symptom go away.

can i ask how long this takes to build?

IIRC it will download a fair amount of data, like around 250MB. Depending on your machine, the build may take a long time; it takes around 5-20 minutes here, I guess, I never timed it precisely. But given your account, I imagine that it will take substantially longer on your machine to build (if you can clone the Git repository itself at all).

also, will i get a choice of what to build when running the make file? i only want to build the 32bit portable install.

There is no choice, but it already builds what you need: the Git executables. You do not even need a portable install.

In fact, you only need a rebuilt git-remote-https.exe that is instrumented to show how far it gets before crashing.

[ghost]

Hmm. Given that I cannot reproduce this issue on any machine I have access to, I am rather doubtful that you can use that machine to investigate: simply switching the machine will make the symptom go away.

i'm only intending to use the other pc to build. i'd obviously be testing it on the problematic pc.

(if you can clone the Git repository itself at all).

i did notice that line near the end of the installer batch file. it's no problem for me to clone the repositories using SSH.

[dscho]

i'm only intending to use the other pc to build. i'd obviously be testing it on the problematic pc.

That makes a lot of sense.

it's no problem for me to clone the repositories using SSH

That should do, indeed.

[ghost]

i managed to build but still no output and i get the same 128 exit code. i put the changes i made here for you to double check...

https://github.com/19379/git/commit/78c874a6f9eef8c84ae8490f23ba85e4ddd8c2de

i just copied the new git-remote-https.exe like you said.

[dscho]

Okay, this establishes that it is not similar to the CHARSET problem as I thought earlier.

In addition to this change, you could also instrument the start_command() and finish_command() functions before rebuilding git-fetch.exe or git-ls-remote.exe, to figure out where exactly it exits with exit code 128.

I have a hunch that it is in this line, but it could also be this one.

The exit code should really be set here.

[ghost]

^ i'm sorry, i don't understand what you want me to do??

also, i added a comment in reply to yours which you probably got a notification for...

https://github.com/19379/git/commit/78c874a6f9eef8c84ae8490f23ba85e4ddd8c2de#commitcomment-13271936

[dscho]

I meant to suggest instrumenting the code, like so: https://github.com/19379/git/pull/1 then rebuilding git-ls-remote.exe and executing that like ./git-ls-remote.exe https://github.com/19379/wsh_marc2003.git.

[ghost]

new build, same result...

[dscho]

Did you copy the git-ls-remote.exe file you built to /mingw32/libexec/git-core? You should see at least one line, where start_command() tries to start git-remote-https.exe...

Ah, slaps-his-head, bummer! The git-ls-remote.exe is not the one you want to copy. You need to build git.exe, copy that into /mingw32/bin/git.exe and then run /mingw32/bin/git ls-remote.exe .... I completely forgot that git-ls-remote.exe is just the Git wrapper, sorry!

[ghost]

see the huge bump in filesize!!!

as i forgot to include it on the screenshot, echo $? returns 128

[dscho]

see the huge bump in filesize!!!

That is to be expected, because you built it with debug symbols.

So now we're getting somewhere! git-remote-https was started. But it was not finished? And even worse: it seems that https://github.com/19379/git/commit/63ef029364b6e65090a14a77666d5564b8f6c4f4#diff-7577a5178f8cdc0f719e580577889f04R537 is hit, but not https://github.com/19379/git/commit/63ef029364b6e65090a14a77666d5564b8f6c4f4#diff-7577a5178f8cdc0f719e580577889f04R543?

Oh, but I forgot to put in fflush(stderr); after the error() calls. Under certain circumstances, stderr is actually not flushed, but cut off, in mintty if not flushed explicitly. Could you add those and try again?

[dscho]

Also, I am pretty certain that the start_command() function is called here. You could instrument the surrounding code there, too, to figure out where things start to break down.

[ghost]

Also, I am pretty certain that the start_command() function is called here.

is that link right? it's pointing at a line which you've already modified.

i've just built a fresh copy with fflush added but won't be able to test until i get home later.

[dscho]

Also, I am pretty certain that the start_command() function is called here.

is that link right? it's pointing at a line which you've already modified.

Bah, copy-paste error. This is the location I meant.

[ghost]

is this ok?

error("Before start_command %s", cmd->argv[0]); fflush(stderr);
code = start_command(helper);
error("After start_command %s", cmd->argv[0]); fflush(stderr);

[dscho]

error: compat/mingw.c:2314: got status -1073741511

Yep, I think that remote-https won't start. Can you start it directly, i.e. using ./git-remote-https.exe?

$ ./git-remote-https.exe
error: remote-curl.c:961: here
error: remote-curl: usage: git remote-curl <remote> [<url>]

[dscho]

(Assuming that it does not start correctly, could you verify the hunch that it misses a .dll using depends.exe?)

[ghost]

User@HP-2DU932P158BD MINGW32 /mingw32/libexec/git-core
$ ./git-remote-https.exe

User@HP-2DU932P158BD MINGW32 /mingw32/libexec/git-core
$ echo $?
127

this looks potentially interesting.

https://cloud.githubusercontent.com/assets/12451008/9976938/08179e3c-5eed-11e5-93b4-496a5c949415.png

remember earlier when i said i was running windows 7 embedded.

edit: earlier in the thread, not today!

[dscho]

I uploaded the picture here ;-)

The problem is the .dll files, like libcurl-4.dll, right? Are they not in libexec/git-core/?

For the record, this is what I get:

$ ls /mingw32/libexec/git-core/*.dll
/mingw32/libexec/git-core/edit.dll*
/mingw32/libexec/git-core/libatomic-1.dll*
/mingw32/libexec/git-core/libbz2-1.dll*
/mingw32/libexec/git-core/libcares-2.dll*
/mingw32/libexec/git-core/libcharset-1.dll*
/mingw32/libexec/git-core/libcurl-4.dll*
/mingw32/libexec/git-core/libeay32.dll*
/mingw32/libexec/git-core/libexpat-1.dll*
/mingw32/libexec/git-core/libffi-6.dll*
/mingw32/libexec/git-core/libgcc_s_dw2-1.dll*
/mingw32/libexec/git-core/libgmp-10.dll*
/mingw32/libexec/git-core/libgmpxx-4.dll*
/mingw32/libexec/git-core/libgnutls-30.dll*
/mingw32/libexec/git-core/libgomp-1.dll*
/mingw32/libexec/git-core/libgomp-plugin-host_nonshm-1.dll*
/mingw32/libexec/git-core/libhogweed-4-1.dll*
/mingw32/libexec/git-core/libiconv-2.dll*
/mingw32/libexec/git-core/libidn-11.dll*
/mingw32/libexec/git-core/libintl-8.dll*
/mingw32/libexec/git-core/libminizip-1.dll*
/mingw32/libexec/git-core/libnettle-6-1.dll*
/mingw32/libexec/git-core/libp11-kit-0.dll*
/mingw32/libexec/git-core/libpcre-1.dll*
/mingw32/libexec/git-core/librtmp-1.dll*
/mingw32/libexec/git-core/libssh2-1.dll*
/mingw32/libexec/git-core/libssp-0.dll*
/mingw32/libexec/git-core/libstdc++-6.dll*
/mingw32/libexec/git-core/libsystre-0.dll*
/mingw32/libexec/git-core/libtasn1-6.dll*
/mingw32/libexec/git-core/libtre-5.dll*
/mingw32/libexec/git-core/libvtv_stubs-0.dll*
/mingw32/libexec/git-core/libvtv-0.dll*
/mingw32/libexec/git-core/libwinpthread-1.dll*
/mingw32/libexec/git-core/ssleay32.dll*
/mingw32/libexec/git-core/tcl86.dll*
/mingw32/libexec/git-core/tk86.dll*
/mingw32/libexec/git-core/zlib1.dll*

after installing a fresh PortableGit-2.5.3-32-bit.7z.exe by double-clicking it...

[dscho]

(Note in particular the presence of libcurl-4.dll, libiconv-2.dll, libintl-8.dll and libpcre-1.dll...)

This is actually expected, as per these lines in 13-copy-dlls.post which gets called from post-install.bat.

[ghost]

i had 0 dlls and now i know why.... it's because i'm using a FAT32 formatted thumb drive. installing to NTFS works fine. the link or copy does not copy when the link fails.

[dscho]

That's good to know, thanks! I will try to reproduce and if I can, I think I will be able to fix it.

Thanks for your persistence with this ticket!

[dscho]

Oops, I think I found it: the echo in this line is a left-over from debugging and needs to go.

@19379 would you maybe able to test by unpacking the portable Git using 7-Zip, editing the 13-copy-dlls.post file by deleting said echo, running post-install.bat and verifying that this fixes things on FAT (I have to go to bed now...)?

[ghost]

yes, that fixes it. i think we can close this now!

[dscho]

Not so fast ;-) The fix is not committed yet... Care to open a Pull Request? 😀

[ghost]

done.

edit: just realised i could have named it better. i didn't reference the actual modified file.

[dscho]

Thanks!

[dscho]

@19379 no worries about the PR! It helped a lot because I only had to adjust it slightly to resolve the issue. Thank you so much!

[bviktor]

So what exactly is the purpose of a portable version if it requires manual "installation" steps? I used to be able to just extract the portable and deploy it everywhere, now I get things like this when starting Git bash:

mkdir: cannot change permissions of '/dev/shm': Permission denied
mkdir: cannot change permissions of '/dev/mqueue': Permission denied
'C:\\Windows\\System32\\drivers\\etc\\hosts' -> '/etc/hosts'
'C:\\Windows\\System32\\drivers\\etc\\protocol' -> '/etc/protocols'
'C:\\Windows\\System32\\drivers\\etc\\services' -> '/etc/services'
'C:\\Windows\\System32\\drivers\\etc\\networks' -> '/etc/networks'

If Git Portable now depends on its path or modifies files outside the Git folder, then it's not portable.

If what the scripts does is the same for any path you put it in and doesn't modify files outside its own folder, then why isn't it done once, before packaging, instead of relying on users doing it?

TBH what the script does it pretty much gibberish to me so if someone cares to explain, that would be helpful.

I deploy Git Portable in a network share, so there's at least 3 paths where it's available: local path on the server (S:\tools\git2), network path on the server (\whatever.domain\tools\git2), and local path when synced to the workstation (X:\whatever\path\they\use\tools\git2). Besides, they're also pushed to Jenkins nodes with almost identical paths but different drive letters in some cases. These I could deploy by simply robocopy'ing them to any workstation and I obviously ain't gonna run scripts on every one of them. What do I do now?

[dscho]

So what exactly is the purpose of a portable version if it requires manual "installation" steps?

I fail to see the relationship with "connectivity issues with github". Wrong ticket?

[bviktor]

No, I don't think so. The second half is about this "post-install" thingie. But I can open a new ticket if you wish.

[dscho]

But I can open a new ticket if you wish.

Oh, come on. It is not like you are doing me a favor here. You are actually not doing yourself a favor, either, by dragging this out. How is your concern possibly related to connectivity issues with GitHub? It is not at all related to that, and you are intelligent enough to realize that.

And you are also smart enough to realize that a bug tracker has to be well organized in order to be effective. Discussing unrelated things in the same ticket (even if there might be some overlap in the code that is touched by those discussions) is doing everybody a disservice.

[bviktor]

A simple "yes, please" will do next time.

[dscho]

A simple "yes, please" will do next time.

You still pretend that you are doing me a favor.

[bviktor]

Nope, I didn't consider my part a favor, the "please" part wasn't about "favors", only about human decency.

I wasn't just randomly picking an issue to whine about a random thing. The release notes about 2.6.0 say "Portable Git's README is now clearer about the need to run post-install.bat when unpacking manually". This text links to pr 83, which is merely a few comments and it explains the changes by linking to this issue.

So I came to this issue to explain what kind of problems this change has caused. But I even said that I can open a new ticket if it's preferable.

Then you come and tell me it's not relevant (???), then 2 paragraphs of blah-blah about how I should treat you and your assaults to get a response out of your highness that's different from GTFO. Way to go, above and beyond.

[dscho]

You know, between replying to your rants, I actually also do real work. And I am getting a bit annoyed, I have to admit, that you still keep doing what you are doing, despite my clear indications that it is inappropriate.

[dscho]

(And yes, I also indicated what would be appropriate instead, so I was even constructive in my criticism.)