Attempting to add a remote to podman results in key mismatch

[sprig]

Issue Description

Running on macos 14.6:

$ podman --log-level=trace system connection add host ssh://user@host
INFO[0000] podman filtering at log level trace
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.LtDz5CJvJ7/Listeners", ssh-agent signer enabled
DEBU[0000] SSH Agent Key SHA256:<key-id> ssh-rsa
DEBU[0000] SSH Agent Key SHA256:<another-id> [email protected]
WARN[0000] ssh host key mismatch for host host:22, got key SHA256:<third-id> of type ecdsa-sha2-nistp256
Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch

whereas

$ ssh host
user@host ~ $

Steps to reproduce the issue

See above.

Describe the results you received

See above.

Describe the results you expected

I expect the connection to be established.
I expect to see where podman looks for host keys in the debug log so that I can confirm/remedy.
In case of another problem (not ssh related) I expect to be notified of this appropriately

podman info output

Macos 14.6


$ podman info
OS: darwin/arm64
provider: applehv
version: 5.1.2

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v5.1.2/libpod/_ping": dial unix /var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/storage-run-501/podman/podman.sock: connect: no such file or directory

(note - I intentionally did not start the VM)

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[EDIT]

I just checked the releases and saw 5.2.0 was available. Upgraded. Exactly the same results.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[sprig]

Nothing stale about this issue, it's just being ignored.

[rhatdan]

You could always attempt to fix it and open a PR.

The contributors to this project are working on hundreds/thousands of issues and features. Whether or not an issue becomes a priority to one is tough to gauge.

[sprig]

If I knew how to fix this, I would. Unfortunately it is not even clear where to look other than spraying the code with print statements.

Unfortunately most of the documentation for podman that I found is not very technical and is more of a tutorial level - particularly on Red Hat's site - and, e.g. searching the source for "host key mismatch" does not reveal much either.

I did not come here with demands for anyone's time. And yet, calling an issue "stale" when it has not received the slightest drop of attention, e.g. pointing a layman where to dig further, is disingenuous at best.

And, from a distinguished container engineer working at the company developing this technology for profit - read, not volunteer - I would have appreciated the vaguest hint at what to look for much much more than this "intro into how open source works".

[rhatdan]

This is no longer my primary job and I don't work on Mac and Windows side of the operations.

If you were a paying customer and running on RHEL you should have opened an issue with Red Hat and they would have raised the priority. As this is a community project the contributors like me do our best to keep up, but sometimes issues get lost in the cracks. If you look at any very active projects you will find thousands of unanswered issues.

@l0rd PTAL

[sprig]

I'm not a paying customer and I did not expect support much less in any given timeframe, yet having the issue autoclosed as "stale" simply because it did not get any response is somewhat frustrating.

That being said, my apologies for the snark - it was pointless and unwarranted. Thank you for your reasonable comments.

[rhatdan]

Stale is not closing issues. we leave issue open indefinitely unless the reporter disappears. I usually only look at issues which go stale, since they ping my email.

[sprig]

I understand, thanks for the explanation.

[l0rd]

@sprig I am able to reproduce your problem on my Mac. I will try to figure out if there is a workaround (i.e. cleaning up the keyring where podman finds the other key) and the problem's root cause.

[sprig]

Thanks!

[l0rd]

@sprig can you check if in your ~/.ssh/known_hosts you have a line for the host you are trying to connect to?

In my case, after removing that particular line in the ~/.ssh/known_hosts, I stopped getting the key mismatch error.
I suspect that podman compares SSH keys of different types (Ed25519 and ECDSA in my case). And of course they don't match and the handshake is aborted. Whereas running the command line SSH client works just fine.

[sprig]

@l0rd Looks like this was this issue! I did not even suspect. Thanks!

I do get a different error now, but it's a bit less cryptic and I can try to debug that but if you have any pointers that would be welcome;

$ podman --log-level=trace ps
INFO[0000] podman filtering at log level trace
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=trace ps)
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.xr9IR7FkPS/Listeners", ssh-agent signer enabled
DEBU[0000] SSH Agent Key SHA256:XXX ssh-rsa
DEBU[0000] SSH Agent Key SHA256:YYY [email protected]
DEBU[0001] DoRequest Method: GET URI: http://d/v5.2.0/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v5.2.0/libpod/_ping": ssh: rejected: connect failed (open failed)
DEBU[0002] Shutting down engines

$ podman --log-level=trace system connection list
INFO[0000] podman filtering at log level trace
Name        URI                                                         Identity    Default     ReadWrite
host        ssh://user@host:22/tmp/podman-run-1000/podman/podman.sock              true        true
DEBU[0000] Shutting down engines

I suspect this is because I use added a connection to user@host where podman is running rootless and so there isn't a socket to connect to since there is no daemon running (there's also a root instance but it does not expose a socket either currently). I see that /tmp/podman-run-1000 exists on the host but not the socket (unless it is supposed to be dynamically created somehow? which I presume is not the case).

Anyway, the original issue is solved. Thank you! I wish the error messages were clearer regarding what is going on (although fetching the correct key would be more ideal :-). I'll see if I can find where the key fetching takes place and whether it is possible to make it compare the correct one.

Feel free to close the issue - I will leave it open for now in the hope you have something to say regarding the second problem.

[l0rd]

Hi @sprig, I had a look at the original issue, and that's a problem with the Golang x/crypto/ssh package. I built a working version of podman using this library. I am considering opening a PR to discuss it with the rest of the maintainers. We need to decide if we want to wait for this proposal implementation and, for the time being, improve the error message as you have suggested. I think we should keep this issue open for now.

[jtognazzi]

I'm on Windows, and I confirm that deleting the existing known keys from the known_hosts file is a valid workaround.

This is the entry added by the windows ssh program:

silverblue ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILLL65WroYP9RR9Zp8V+0me+fTwwhxcmy+RctAUDQ0Ig

and this is the entry added by podman (version 5.2.0)

silverblue ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYFjxg0up06m12Hir3gz4M3MCLqieJBonhmNuW8zRjc8QBdXOk4nhMrVrnmJx5PvS7Pbm7J+nt+1jriq67Fqh8=

[sprig]

I can also confirm that deleting existing keys works.

[rsenna]

Having the same issue on macOS and Linux.
I managed to get it working at some point, by removing the keys from .ssh/known_hosts as mentioned here, but even that stopped working after some time in my case...
Forcing podman to use the required key also doesn't work:

$ podman --identity ~/.ssh/id_ed25519 ps

WARN[0000] ssh host key mismatch for host some-host-ip:22, got key SHA256:... of type ecdsa-sha2-nistp256
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: knownhosts: key mismatch

[l0rd]

I have opened a PR that should address this issue. In this commit message, I describe the podman scenarios I have tested. To be able to merge it, though, I am waiting to see all CI tests green and reviews.