Git SSH keys failing to authenticate

[btrepp]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Argocd is failing to authenticate with a valid private key

argocd@argocd-server-c9958cfc8-cwvg2:~$ argocd repo add [email protected]:/~btrepp/infrastructure --ssh-private-key-path ./id --loglevel debug
FATA[0002] rpc error: code = Unknown desc = error testing repository connectivity: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 

Using ssh on the same server shows the key is good

argocd@argocd-server-c9958cfc8-cwvg2:~$ ssh -i id [email protected] 
PTY allocation request failed on channel 0
Hi btrepp! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

To Reproduce

Run argocd add repo

Expected behavior

Argocd adds the repo with it's credentials

Screenshots

Version

argocd@argocd-server-c9958cfc8-cwvg2:~$ argocd version
argocd: v2.3.3+07ac038
  BuildDate: 2022-03-30T00:59:49Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/arm64
argocd-server: v2.3.3+07ac038
  BuildDate: 2022-03-30T00:59:49Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/arm64
  Ksonnet Version: vdev-2022-03-30T00:42:07+0000
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.0+gd141386
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

Logs

time="2022-04-17T05:32:08Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims="{\"exp\":1650259033,\"iat\":1650172633,\"iss\":\"argocd\",\"jti\":\"baeab803-9cc0-44a5-a33a-192203dc5ed9\",\"nbf\":1650172633,\"sub\":\"admin\"}" grpc.request.content= grpc.service=version.VersionService grpc.start_time="2022-04-17T05:32:08Z" span.kind=server system=grpc
time="2022-04-17T05:32:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2022-04-17T05:32:08Z" grpc.time_ms=30.641 span.kind=server system=grpc
time="2022-04-17T05:32:10Z" level=error msg="finished unary call with code Unknown" error="rpc error: code = Unknown desc = error testing repository connectivity: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain" grpc.code=Unknown grpc.method=ValidateAccess grpc.service=repository.RepositoryService grpc.start_time="2022-04-17T05:32:08Z" grpc.time_ms=1892.31 span.kind=server system=grpc

[nikhleshgoenka]

Did you try cloning the repository with same private key and that works?

ssh-agent
ssh-add
git clone

[btrepp]

Clones fine

argocd@argocd-server-7d74dbfbc7-4vklw:~$ eval $(ssh-agent)
Agent pid 66
argocd@argocd-server-7d74dbfbc7-4vklw:~$ ssh-add id
Identity added: id (beau@beau-pc)
argocd@argocd-server-7d74dbfbc7-4vklw:~$ git clone [email protected]:/~btrepp/infrastructure
Cloning into 'infrastructure'...
remote: Enumerating objects: 1490, done.
remote: Counting objects: 100% (1048/1048), done.
remote: Compressing objects: 100% (1014/1014), done.
remote: Total 1490 (delta 610), reused 0 (delta 0), pack-reused 442
Receiving objects: 100% (1490/1490), 297.37 KiB | 378.00 KiB/s, done.
Resolving deltas: 100% (776/776), done.

Unfortunately argcod doesn't really give any logs. If I had to guess, it's not using the SSH key private for some reason and ignoring it... but it would be great if I knew why, the key exists entirely for argocd, so it's easy enough to generate a new one.
A bit strange as if i give it a malformed key on purpose... i get an error that it couldn't find a key. So it seems it will parse it, but not use it.

I'm unclear how the cli/server and what not work. I would hope running the cli on the server would give some idea, but maybe it's xferring the key to another component, and that is failing.

I've tried stripping and running dos2unix, but still the same error. I would be happy enough to regenerate a key, but it would be random flags until argocd decides it likes that key :(

From #1894 I've tried generating a new key with -m pem, as maybe argocd doesn't support non-pem keys? but this is the same behaviour, git and ssh work, argocd fails

[btrepp]

ssh-keygen -t ed25519 -C "argocd" -f argocd

from #7600
Has allowed me to finally craft a key argocd will use, it's not the same host (e.g not github).

I think argocd isn't supporting the default flags of whatever ssh-keygen arch installs.

[ghost]

Argo CD only half supports PEM keys. The error is confusing because of history like #2890 and from that history you'll notice mention that Argo does not consistently handle the git URLs and the disparity is visible when attempting to use PEM formatted private keys, which you might have if you're migrating from fluxcd.

The error presents as follows for the application:

rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force` failed exit status 128: Load key "/dev/shm/2504980542": invalid format [email protected]: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

Viewing the repo status, it has "Successful" for the status.

[MaiCabrera]

any one have update/workaround and or a solution on this bug?
I've been literally suck in the same situation. @protosam did you get any luck changing the ssh key ?

[ghost]

@MaiCabrera yes, just don't use PEM formatted keys. Use OpenSSH formatted keys. These commands will change the files in-place. Make backups beforehand just in case. (I might have this backwards btw, it's been a while)

# to openssh format
ssh-keygen -e -p -f ~/.ssh/id_rsa-keyname -m RFC4716

# to pem format
ssh-keygen -e -p -f ~/.ssh/id_rsa-keyname -m PEM

I actually stopped using Argo CD after seeing these inconsistencies and have fully adopted Flux v2 at this point. The one thing Argo CD would do for me that Flux v2 doesn't is accept an Application that installs cert-manager via helm + Issuers, and eventually reconcile these things.

Flux v2 would need to be setup with a Kustomization CR (it can dynamically build it's own kustomization.yaml). The entire thing will fail if cert-manager is not installed but Issuers are in the manifests. For chicken and egg issues where CR's don't exist yet, I just built a shim that would wrap the problem inside HelmReleases, which can depend on each other.

In fact, being forced to this state, I've found that I actually like the way my problems had to be addressed in the Flux v2 eco system. My Kustomization does gitops by using a repo where all objects are required to be a HelmRelease. Automation for pull requests validate HelmReleases with kubeconform and yq checking the "kind". PR's can't be merged if validation fails. Using only HelmReleases is more akin to config management, which my team desires. Maintaining pods, configmaps, and replicasets is more like app development, which my team does not desire in the cluster management repo.

Not saying Argo CD is bad by any stretch here, but having more field of view over this, it does fall a bit short in ways I didn't expect.

[Xosmond]

@protosam Thank you.
It's very weird that so many people have to deal with this issue without a proper response from the maintainer, clear error root cause or updated documentation. I'll look for an alternative tool as I don't plan to spent my time debugging a tool instead of my clusters/applications.

[JMarji]

So I was having trouble with creating an app in ArgoCD after successfully connecting to my repo over ssh in ArgoCD.

I was using bitbucket as a repository. by reading the error message that ArgoCD output when I hit app create. Which went something like "[email protected] unable to "git fetch... etc" (Public Key)". I realized that Argo was trying to authenticate with bitbucket using the username argocd. This would not work in my setup as we did not have a user account set up for argocd.

TLDR; I selected https instead of ssh in ArgoCD and used bitbucket app passwords in account settings on bitbucket and used my bitbucket username and the app password provided through the app password process. solved all my auth problems with the repo.

[diptripa]

ssh-keygen -t ed25519 -C "argocd" -f argocd

from #7600 Has allowed me to finally craft a key argocd will use, it's not the same host (e.g not github).

I think argocd isn't supporting the default flags of whatever ssh-keygen arch installs.

This really helped solve this problem! Thanks

[blakepettersson]

See my comment in #7600.

This issue might be related to golang/go#37278, specifically golang/go#37278 (comment). golang/crypto#211 might address this, but unknown when/if this will be merged...

[blakepettersson]

golang/crypto@6fad3dfc may have fixed this. Argo CD 2.7.x upgraded to golang.org/x/crypto v0.6.0 so might be worth trying out there?

[horlyk]

In my case I had to remove passphrase from a key and then it started working. I figured that out by installing argo cli and then adding repo manually specifying a key path and there was a clear error about passphrase protection.

argocd repo add REPO_URL_HERE --ssh-private-key-path ~/.ssh/argocd

FATA[0000] rpc error: code = Unknown desc = error testing repository connectivity: ssh: this private key is passphrase protected