ssh: handshake failed: ssh: no authorities for hostname: quis.cx:22

[sgielen]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am trying to add a private Git repository on quis.cx using the UI (Settings -> Repositories, Connect repo using SSH).

First, I tried to add the repository "[email protected]:argocd". This only gave the error "repository not found", with no outgoing connections to quis.cx. Even with the reposerver logging bumped up to debug (and a recreate of the reposerver Pod), there was no further information in the reposerver log other than:

time="2021-08-25T08:37:52Z" level=error msg="finished unary call with code Unknown" error="repository not found" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2021-08-25T08:38:52Z" grpc.service=repository.RepoServerService grpc.start_time="2021-08-25T08:37:52Z" grpc.time_ms=66.486 span.kind=server system=grpc

(Should I file a separate issue for this?)

After this, I tried to add the repository ssh://[email protected]/argocd. This way, I saw SSH connections being established but failing (no additional information in the reposerver logs):

time="2021-08-25T09:00:23Z" level=error msg="finished unary call with code Unknown" error="ssh: handshake failed: ssh: no authorities for hostname: quis.cx:22" grpc.code=Unknown grpc.method=TestRepository grpc.request.deadline="2021-08-25T09:01:23Z" grpc.service=repository.RepoServerService grpc.start_time="2021-08-25T09:00:23Z" grpc.time_ms=101.509 span.kind=server system=grpc

I have added the host keys from ssh-keyscan quis.cx to Settings -> Certificates and they show up in the argocd-ssh-known-hosts-cm. As you can see, I have added various formats to no avail:

    quis.cx ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIW+voCxUw9DtHFePZMGcj4pNz19G8iEUzGGWNMSLqOd6lzXeJ/KuZR5KAEIM2CogbP0i2/jVimFLsNO8kUEqi8=
    37.252.124.223 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIW+voCxUw9DtHFePZMGcj4pNz19G8iEUzGGWNMSLqOd6lzXeJ/KuZR5KAEIM2CogbP0i2/jVimFLsNO8kUEqi8=
    quis.cx ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6APdXTo3hQJz0x9T1cOswmedVIKbmNSht8qz+H/AA4UzWggL1tH5faxNT5wSi6+MRidSlHfooZJpLqB7Emx9BpAvl0ZRnUnNDCxJ7kunAnXAjEsScJRGgzDGzbr1hV2UlZIQp6sO4c8pedTTyU68iLY7Hak2GLJ4CfuIdCj7hZv88LDp1ZHDdsDKxLFsXaXUar6gbmI3PN6eY/WFtTRMJqmx5FvtVNqr/m+26A5EILmVJ5IH0wQjapVIDk+G9UJyiDrXiX6N2KP8HIjfpT2Q2H0QlfvPjH72cVfh8XmboRy/MWneYbwvlKC7gzAo3BFGyZNaWkTTx+6rwxq8GFSZH
    [quis.cx]:22 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6APdXTo3hQJz0x9T1cOswmedVIKbmNSht8qz+H/AA4UzWggL1tH5faxNT5wSi6+MRidSlHfooZJpLqB7Emx9BpAvl0ZRnUnNDCxJ7kunAnXAjEsScJRGgzDGzbr1hV2UlZIQp6sO4c8pedTTyU68iLY7Hak2GLJ4CfuIdCj7hZv88LDp1ZHDdsDKxLFsXaXUar6gbmI3PN6eY/WFtTRMJqmx5FvtVNqr/m+26A5EILmVJ5IH0wQjapVIDk+G9UJyiDrXiX6N2KP8HIjfpT2Q2H0QlfvPjH72cVfh8XmboRy/MWneYbwvlKC7gzAo3BFGyZNaWkTTx+6rwxq8GFSZH
    [quis.cx]:22 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIW+voCxUw9DtHFePZMGcj4pNz19G8iEUzGGWNMSLqOd6lzXeJ/KuZR5KAEIM2CogbP0i2/jVimFLsNO8kUEqi8=
    [quis.cx]:22 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINuw4/qY8/l1Rl5xGcS0Oh5LOuAN1ZzbK0f2MA5ApYg4
    quis.cx ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINuw4/qY8/l1Rl5xGcS0Oh5LOuAN1ZzbK0f2MA5ApYg4

However, when I tick the "Skip server verification" box, the connection succeeds.

To Reproduce

1. Add the host keys above to the argocd-ssh-known-hosts-cm and restart the reposerver pod.
2. In the webinterface, add the ssh://[email protected]/argocd repository and observe "no authorities for hostname: quis.cx:22" error.

Expected behavior

A repository to be added even with "skip server verification" disabled.

Screenshots

Version

argocd: v2.1.0+d0b2d55
  BuildDate: 2021-08-20T05:30:54Z
  GitCommit: d0b2d55e3fb7fe8b17385d6687886de41651f31b
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64

[hyww]

This might be related to that Golang issue golang/go#33366