Git can't work as network service with OpenSSH

[davekran]

Hi,

I'm trying to run Git.exe as a network service account when connecting to a private repository with
OpenSSH.
This issue seems to be a regression since I could do the above in Git for windows version 1.9.5
In order to reproduce you can download PsExec utility from the following official Microsoft link:
https://technet.microsoft.com/en-us/sysinternals/psexec.aspx

I created a ".ssh" directory and keys inside network service account home folder "%systemroot%\ServiceProfiles\NetworkService"

The command I run looks like:
psexec.exe -i -u "nt authority\network service" "C:\Program Files (x86)\Git\cmd\git.exe" ls-remote "[email protected]/username/repo"

If I run the above git command as a normal user account, everything works just fine.

I had a search for similar issues in github and found a similar one which had a fix for local system account, please see: #435
Indeed, when I run my command as a local system and ".ssh" directory under local system home folder at "C:\Windows\System32\Config\systemprofile" it works fine with a command like:
psexec.exe -s "C:\Program Files (x86)\Git\cmd\git.exe" ls-remote "[email protected]/username/repo"

This is a critical issue for me since my application runs as network service and executes git as
external process, can you please advise?

Thanks
Dave

[kostix]

Your report misses an error message you get.

Two more things:

1. Please copy-and-paste what Git outputs when you perform that ls-remote command, but first make sure you have the GIT_TRACE environment variable set to 1.

   Like this:

    set GIT_TRACE=1
    psexec.exe ...
   

2. Please copy and paste the environment a process run under the network system account has on your system.

   To do this, run something like

    psexec.exe  ... cmd /c set >c:\env.txt
   

[davekran]

The error I get is: (the trace doesn't give any extra info)

Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

There was no change after adding the /c argument to PsExec

[Radrik5]

Please try "C:\Program Files (x86)\Git\usr\bin\ssh.exe" -v [email protected] to understand which keys it uses. Most probably it doesn't have a proper HOME path in the environment.

[dscho]

@Radrik5 good point. This has been fixed recently. Which points out a serious short-coming of this bug report: it is pretty sloppy, e.g. missing out the exact architecture and version of Git for Windows used. Here is an article that should help address those short-comings: https://github.com/git-for-windows/git/wiki/Issue-reporting-guidelines

[rimrul]

Are you using DSA keys? Have a look at #375 and/or #421.

This is a critical issue for me since my application runs as network service and executes git as
external process, can you please advise?

nt authority\network service and nt authority\local service are not exactly intended to run anything but services. There might be a valid reason to do it anyways, but this is one of the possible side effects. If your application is not a service, consider runing it under a User, that is not intended for services. If it is a service and you have access to the source code, how about using libgit2?

The error I get is: (the trace doesn't give any extra info)
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Can you paste the trace info anyways? It might contain details, that don't seem usefull to you, but are usefull to us. The Permission denied (publickey). sounds like either a DSA key or the server rejecting your key to me. Are you sure, you published the right key to the server?

[kostix]

@rimrul, the OP stated that

If I run the above git command as a normal user account, everything works just fine.

so I beleive the DSA key should not be the reason.

I tend to suspect some crucial difference in the environment between normal users and that service account which makes OpenSSH fail to locate the key (or something like this).

I think @Radrik5's idea is a spot-on: an explicit call to ssh could clear up much.

[rimrul]

@rimrul, the OP stated that

If I run the above git command as a normal user account, everything works just fine.

so I beleive the DSA key should not be the reason.

Right. I missed that. It might still be, if OP accidentially copied the wrong key or tested against a different host or something, but it's less likely.

I think @Radrik5's idea is a spot-on: an explicit call to ssh could clear up much.

That should deffinitly help. I was trying to provide additional assistance.

[kostix]

@rimrul, oh, I definitely did not intend to bash your insight. The more ideas are thrown at a problem, the better!

[davekran]

Some clarifications and answers:

1. The issue occurs on Git for Windows version 2.6.3-64 bit on Windows 7. It works fine on Git for Windows version 1.9.5
2. My application is a service and I need to use network service account.
3. I tried to run ssh directly with PsExec by running the command psexec -i -u "nt authority\network service" "C:\Program Files\Git\usr\bin\ssh.exe" -v [email protected]
   and the error I got was:

Error establishing communication with PsExec service on My-PC:
The system cannot find the file specified.

[kostix]

1. I tried to run ssh directly with PsExec by running the command
   psexec -i -u "nt authority\network service" "C:\Program Files\Git\usr \bin\ssh.exe" -v [email protected] and the error I got was:

Error establishing communication with PsExec service on My-PC:
The system cannot find the file specified.

Are you positive C:\Program Files\Git\usr\bin\ssh.exe exists?

If yes, there might be some problem with ssh.exe fails to load some
DLL it needs for some reason (or one of DLLs it links to fails to load
another DLL -- and so on -- this is transitive).

[rimrul]

In the first post you wrote

The command I run looks like:
psexec.exe -i -u "nt authority\network service" "C:**Program Files (x86)**\Git\cmd\git.exe" ls-remote "[email protected]/username/repo"

In your clarifications and answers post you wrote

I tried to run ssh directly with PsExec by running the command psexec -i -u "nt authority\network service" "C:*Program Files*\Git\usr\bin\ssh.exe" -v [email protected]
and the error I got was:
Error establishing communication with PsExec service on My-PC:
The system cannot find the file specified.

The issue is a missing (x86).

[davekran]

The path "C:\Program Files (x86)\Git\cmd\git.exe" from my initial post was taken from the command I ran on Git 1.9.5 which was working fine.
Sorry for confusion but the issue is not related to a missing (x86)

[davekran]

Are you positive C:\Program Files\Git\usr\bin\ssh.exe exists?

Yes ssh.exe exists, it executed when I run without PsExec

[kostix]

Are you positive C:\Program Files\Git\usr\bin\ssh.exe exists?
Yes ssh.exe exists, it executed when I run without PsExec

Is it possible to run a shell (say, cmd.exe) under that "network
system" account? I mean, a shell which will be physically available
for you to type commands into?

If yes, the next step will be to attempt running that ssh.exe (using
its full path) in such a console instance while looking for abnormal
error messages.

If my initial hypothesis is correct, the next logical step would be
either trying depends.exe from that console on the SSH program or
firing up WinDbg, attaching to that console window, somehow making
it debug the child processes spawned, then spawning ssh.exe and
looking for problems there. Unfortunately, I don't know if it's even
possible but I'd try.

[dscho]

The error I get is:

psexec.exe -i -u "nt authority\network service" "C:\Program Files\Git\cmd\git.exe" ls-remote "[email protected]/dscho/images"

PsExec v2.11 - Execute processes remotely
Copyright (C) 2001-2014 Mark Russinovich
Sysinternals - www.sysinternals.com


Error establishing communication with PsExec service on MYLAPTOP:
The system cannot find the file specified.

I wish the bug report was more thorough so I would not be stuck on the preliminaries such as reproducing the issue.

[kostix]

psexec.exe -i -u "nt authority\network service" "C:\Program Files\Git\cmd\git.exe" ls-remote "ssh://[email protected]/whatever.git"

worked for me with GfW 2.6.1 on WinXP SP3 32-bit. Git started, and was able to use plink.exe from TortoiseGit (via the GIT_SSH env. variable).

On the other hand, IIUC much has been changed since XP in regard to running services.

[kostix]

Just tried to run

psexec.exe -i -u "nt authority\network service" C:\WINDOWS\system32\cmd.exe

on a nearby Windows 7 64-bit and it worked OK if I run the command in a shell started with elevated privileges (otherwise it fails with "Permission denied" error).

The environment of the shell spawned looks quite okay to me (some inessential bits elided):

C:\Windows\system32>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=MYHOST
ComSpec=C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK=NO
lib=C:\Program Files\SQLXML 4.0\bin\
LOCALAPPDATA=C:\Windows\ServiceProfiles\NetworkService\AppData\Local
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=...
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=AMD64 Family 16 Model 5 Stepping 3, AuthenticAMD
PROCESSOR_LEVEL=16
PROCESSOR_REVISION=0503
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC=C:\Users\Public
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Windows\SERVIC~2\NETWOR~1\AppData\Local\Temp
TMP=C:\Windows\SERVIC~2\NETWOR~1\AppData\Local\Temp
USERDOMAIN=MYDOMAIN
USERNAME=MYHOST$
USERPROFILE=C:\Windows\ServiceProfiles\NetworkService
windir=C:\Windows
windows_tracing_flags=3
windows_tracing_logfile=C:\BVTBin\Tests\installpackage\csilogfile.log

So I'm inclined to think this issue is somehow specific to GfW.

[dscho]

On my Windows 7, 64-bit VM:

>psexec.exe -i -u "nt authority\network service" C:\WINDOWS\system32\cmd.exe

PsExec v2.11 - Execute processes remotely
Copyright (C) 2001-2014 Mark Russinovich
Sysinternals - www.sysinternals.com


Error establishing communication with PsExec service on VIRTUALBOX-PC:
The system cannot find the file specified.

I.e. no dice.

Maybe you need to be domain joined or some such. If so, that is information that really would have helped me start the bug hunt already.

[kostix]

Maybe you need to be domain joined or some such. If so, that is information that really would have helped me start the bug hunt already.

Yeah, the machine I tested on is joined into a Windows domain and the currently logged in user was actually a domain admin (I still had to ask the user to get me an elevanted shell for the invocation to succeed). Have you run your example from an elevated shell as well?

[dscho]

I tested on is joined into a Windows domain and the currently logged in user was actually a domain admin

I have not a single setup available to me where I am domain admin. I think that explains that. I wish the original report would have been substantially clearer on that detail, it would have saved me something like 1h to stab around and not being a single step closer to reproducing, let alone fixing. I guess I outspent my time quota on this one for a month or so.

Have you run your example from an elevated shell as well?

Yes.

[kostix]

I tested on is joined into a Windows domain and the currently
logged in user was actually a domain admin
I have not a single setup available to me where I am domain admin.
[...]
Have you run your example from an elevated shell as well?
Yes.

I'll see if I would be able to reproduce this in a domain-joined machine
but under a local admin account. If that would exhibit the bug, I
think we'll be able to provision a Win7 64-bit VM for you -- as we did
back then with a WinXP VM.
Of course, if you wish to spend time on debugging this. ;-)

[kostix]

I tested on is joined into a Windows domain and the currently
logged in user was actually a domain admin
I have not a single setup available to me where I am domain admin.

Just tried to run "C:\Program Files\Git\bin\git.exe" of 2.6.4 on a Win7 64-bit machine joined to a domain using an elevated shell running under a domain admin account -- worked for me.

The only weird thing is ssh.exe telling me it wants to manupulate
its ".ssh" directory under "/":

Could not create directory '/.ssh'.
The authenticity of host 'git.domai.local (192.168.2.2)' can't be established.

RSA key fingerprint is SHA256:/5lFie/ot7+jg6tQlZmdPiuZz5JHGeghdjw0MwErvva3g.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/.ssh/known_hosts).
[email protected]'s password:

At the same time the environment of the ssh.exe process looks OK: the HOME and USERPROFILE variables point under the directory supposed to be the network service's account home dir.

GfW has been installed for the first time on this box. I ticked the "Run Git commands in the shell" flag (I always do), selected to use cmd.exe as the console and enabled filesystem caching (should not affect the matter).

All-in-all, if the message @davekran has in their second round of tests really was

Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.

the explanation might indeed lie in that weird /.ssh path the SSH client tries to access.

So, @davekran, could you start with running

psexec ... C:\WINDOWS\system32\cmd.exe /k "C:\Program Files\Git\bin\git.exe" ls-remote ssh://user@host/repo

from an elevated whell and see whether ssh will issue warning similar to these in my test run?

[Radrik5]

@kostix are you sure ssh has correct HOME variable in the environment? It's absent in the list of environment variables you posted in #569 (comment).

[kostix]

@Radrik5, the environment I showed was from an instance of cmd.exe. I'm pretty sure it would be strange to find the HOME env. variable there.

In my comment about running git.exe on Win7, I actually studied the environment of ssh.exe, and there was the HOME variable. May be it gets set by git.exe?

[dscho]

@kostix I would love to have access to your box again, to investigate that incorrect HOME setting.

[dscho]

Let's just close this ticket before it is so stale that it starts to smell.