Permission denied (publickey). fatal: Could not read from remote repository

[XDavidT]

Describe the bug
Fresh new installation - added a local git repo and then added a new app.
Found that it's weird that the branch list is working, but typing path isn't auto-complete me, but did it anyway. after clicking add got an error :

Unable to create application: application spec for iid is invalid: InvalidSpecError: Unable to generate manifests in argocd/my-app: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force failed exit status 128: Load key "/dev/shm/3523402803": error in libcrypto git@gitserver: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

To Reproduce

1. Install new argocd
2. Add local repo - mark 'skip server verification'
3. add a new application

or using the CLI:
argocd app create iid --repo ssh://git@gitserver:7999/my/repo.git --path argocd/my-app --dest-namespace default --dest-server https://kubernetes.default.svc --directory-recurse
and output is:

FATA[0000] rpc error: code = InvalidArgument desc = application spec for myapp is invalid: InvalidSpecError: Unable to generate manifests in argocd/my-app: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force` failed exit status 128: Load key "/dev/shm/95645104": error in libcrypto
git@gitserver: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Expected behavior

1. Autocomplete me when typing path while adding a new app
2. See the new app created.

Screenshots

Version

argocd: v2.3.3+07ac038
  BuildDate: 2022-03-30T01:46:59Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.4+fe80bdc
  BuildDate: 2022-07-07T07:11:00Z
  GitCommit: fe80bdcfdc847372f268f5d711d3112c7d1e39bf
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0```

[simbelmas]

Exactly same behavior here trying to pull from gitea repository on the same k3s cluster.

Argocd version:

argocd-server: v2.4.0+5c08527
  BuildDate: 2022-07-20T21:48:11Z
  GitCommit: 5c08527132d1da9538fb736783f45d92615907e7
  GitTreeState: clean
  GoVersion: go1.18.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.5 2022-05-20T20:25:40Z
  Helm Version: v3.9.0+g7ceeda6
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Gitea version:

Gitea version 1.16.9 built with GNU Make 4.3, go1.16.15 : bindata

Tried scenarii:

1. reach repo by ingress (traefik)
2. reach repo directly on service (using svc.cluster.local)

Obviously the key work outside of argocd.
Diffed argocd ssh-privatekey b64 stanza with another secret used to pull from comman line.

On git server side, it seems that ssh connection is stuck so it's killed:

2022/07/21 20:15:12 modules/ssh/ssh.go:154:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:XXX from 10.42.4.120:39310
2022/07/21 20:15:12 modules/ssh/ssh.go:235:publicKeyHandler() [D] Handle Public Key: 10.42.4.120:39310 Fingerprint: SHA256:XXX is not a certificate
2022/07/21 20:15:12 ...s/asymkey/ssh_key.go:158:searchPublicKeyByContentWithEngine() [I] [SQL] SELECT `id`, `owner_id`, `name`, `fingerprint`, `content`, `mode`, `type`, `login_source_id`, `created_unix`, `updated_unix`, `verified` FROM `public_key` WHERE (content like ?) LIMIT 1 [ssh-rsa XXX] - 5.444798ms
2022/07/21 20:15:12 modules/ssh/ssh.go:252:publicKeyHandler() [D] Successfully authenticated: 10.42.4.120:39310 Public Key Fingerprint: SHA256:XXX
2022/07/21 20:15:13 Started GET /api/internal/serv/command/38/user/repo?mode=1&verb=git-upload-pack for [::1]:40770
2022/07/21 20:15:13 [email protected]/engine.go:1139:Get() [I] [SQL] SELECT `id`, `lower_name`, `name`, `full_name`, `email`, `keep_email_private`, `email_notifications_preference`, `passwd`, `passwd_hash_algo`, `must_change_password`, `login_type`, `login_source`, `login_name`, `type`, `location`, `website`, `rands`, `salt`, `language`, `description`, `created_unix`, `updated_unix`, `last_login_unix`, `last_repo_visibility`, `max_repo_creation`, `is_active`, `is_admin`, `is_restricted`, `allow_git_hook`, `allow_import_local`, `allow_create_organization`, `prohibit_login`, `avatar`, `avatar_email`, `use_custom_avatar`, `num_followers`, `num_following`, `num_stars`, `num_repos`, `num_teams`, `num_members`, `visibility`, `repo_admin_change_team_access`, `diff_view_style`, `theme`, `keep_activity_private` FROM `user` WHERE `lower_name`=? LIMIT 1 [user] - 3.979216ms
2022/07/21 20:15:13 [email protected]/engine.go:1139:Get() [I] [SQL] SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `num_projects`, `num_closed_projects`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `trust_model`, `avatar`, `created_unix`, `updated_unix` FROM `repository` WHERE `owner_id`=? AND `lower_name`=? LIMIT 1 [1 repo] - 3.866219ms
2022/07/21 20:15:13 ...s/asymkey/ssh_key.go:145:GetPublicKeyByID() [I] [SQL] SELECT `id`, `owner_id`, `name`, `fingerprint`, `content`, `mode`, `type`, `login_source_id`, `created_unix`, `updated_unix`, `verified` FROM `public_key` WHERE `id`=? LIMIT 1 [38] - 3.421983ms
2022/07/21 20:15:13 [email protected]/engine.go:1139:Get() [I] [SQL] SELECT `id`, `key_id`, `repo_id`, `name`, `fingerprint`, `mode`, `created_unix`, `updated_unix` FROM `deploy_key` WHERE `key_id`=? AND `repo_id`=? LIMIT 1 [38 21] - 2.505926ms
2022/07/21 20:15:13 Completed GET /api/internal/serv/command/38/user/repo?mode=1&verb=git-upload-pack 200 OK in 23.463268ms
2022/07/21 20:15:13 ...ters/private/serv.go:412:ServCommand() [D] Serv Results:
	IsWiki: false
	DeployKeyID: 34
	KeyID: 38	KeyName: [email protected]
	UserName: user
	UserID: 1
	OwnerName: user
	RepoName: repo
	RepoID: 21
2022/07/21 20:15:13 modules/ssh/ssh.go:144:sessionHandler() [E] SSH: Wait: signal: killed
2022/07/21 20:15:13 modules/ssh/ssh.go:148:sessionHandler() [E] Session failed to exit. EOF
2022/07/21 20:15:13 modules/ssh/ssh.go:263:sshConnectionFailed() [W] Failed connection from 10.42.4.120:39316 with error: [ssh: no auth passed yet]
2022/07/21 20:15:13 modules/ssh/ssh.go:265:sshConnectionFailed() [W] Failed authentication attempt from 10.42.4.120:39316

[simbelmas]

Answering my question.
By digging into open issue, i found #9122

This solves the issue for me.