Podman over SSH in native mode does not use custom config `~/.ssh/config` #23831

evolutics · 2024-09-01T13:51:13Z

Issue Description

Connecting to a remote Podman socket over native SSH (e.g., podman --host ssh://my-host --ssh native ps) fails, despite the local ~/.ssh/config file containing an entry for Host my-host (i.e., ssh my-host succeeds).

Steps to reproduce the issue

Set up:

Run a test VM locally using Vagrant:
```
vagrant init bento/fedora-40
vagrant up
```

Set up SSH access to the VM:

vagrant ssh-config --host my-host >>~/.ssh/config
ssh my-host echo 'SSH setup is OK'

Set up Podman on the VM (with Docker API), running an example container:

ssh my-host <<EOF
sudo dnf --assumeyes install podman
podman info

systemctl --user enable --now podman
sudo loginctl enable-linger
sudo ln --force --symbolic /usr/bin/podman /usr/local/bin/docker

podman run --detach docker.io/hashicorp/http-echo:1.0
EOF

Test:

Local Docker can connect to the remote Podman service, showing 1 container:
```
docker --host ssh://my-host ps
```
But local Podman cannot connect to remote Podman – see error below:
```
podman --host ssh://my-host --ssh native ps
```

Tear down:

Remove Vagrant resources:

vagrant destroy
rm --force --recursive .vagrant Vagrantfile

Undo changes to ~/.ssh/config.

Describe the results you received

$  podman --host ssh://my-host --ssh native ps
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp: lookup my-host: device or resource busy: ssh://my-host

Describe the results you expected

$  docker --host ssh://my-host ps
CONTAINER ID   IMAGE                     COMMAND   CREATED              STATUS              PORTS     NAMES
abcdef123456   hashicorp/http-echo:1.0   ""        About a minute ago   Up About a minute             foo_bar

podman info output

$ podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.5
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64

$ uname --all
Linux foo-desktop 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 GNU/Linux


$ vagrant ssh -- podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Wed Aug 21 00:00:00 2024
OS/Arch:      linux/amd64

$ vagrant ssh -- uname --all
Linux fedora 6.8.7-300.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 17 19:21:08 UTC 2024 x86_64 GNU/Linux

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

–

Additional information

Possibly related:

The text was updated successfully, but these errors were encountered:

afbjorklund · 2024-09-01T15:04:33Z

This is a known shortcoming, you have to add the basic ssh config to a podman "system connection"

Host
  HostName
  User
  Port
  IdentityFile

A workaround is to set up the ssh tunnel yourself (with the ssh config), and use unix: instead of ssh:

Podman ignores .ssh/config in remote connection mode #21916

evolutics · 2024-09-01T15:55:03Z

Thanks for your fast explanation!

This is a known shortcoming, you have to add the basic ssh config to a podman "system connection"

Does this mean that I'd have to unpack my (possibly generated) ~/.ssh/config file to translate the supported subset of options to podman system connection add? The manual seems to suggest so.

A workaround would be to extend the $PATH with a folder containing a custom ssh executable that essentially forwards to /usr/bin/ssh -F /my/ssh/config. This works with Docker, however not with Podman – I guess I am misunderstanding what --ssh native does exactly.

afbjorklund · 2024-09-01T15:58:52Z

I think it should be possible for podman to parse the basics, but you would lose out on any other ssh features...

podman tries to lookup hostname which is used for ssh connection. #21113 (comment)

The same issue exists in Lima, and is worse now when Podman Desktop can parse connections but not forwards...

Indicator of Podman Remote in UI podman-desktop/podman-desktop#8281 (comment)

afbjorklund · 2024-09-01T16:13:08Z

I guess I am misunderstanding what --ssh native does exactly.

It seems to be used a little inconsistently around the podman code base.

Some parts are using the "SSHMode", but some are still calling ssh.Dial.

i.e. not using the containers-common wrapper, but golang.org/x/crypto/ssh

afbjorklund · 2024-09-01T16:25:37Z

I think it has been this way since Podman v2, that is before the connections...

e.g. https://boot2podman.github.io/2020/07/22/machine-replacement.html

export CONTAINER_HOST=ssh://[email protected]:2222/run/user/1000/podman/podman.sock
export CONTAINER_SSHKEY=$PWD/.vagrant/machines/default/virtualbox/private_key

But then came Podman Machine v3, and replaced the Fedora and the VirtualBox.

Luap99 · 2024-09-01T17:18:31Z

ssh=native was never properly finished AFAIK, it doesn't seem to be tested either.

I do agree this should work with native mode as we call the ssh binary which should read you ssh config so yes this seems like a bug to me.

afbjorklund · 2024-09-01T17:59:17Z

Most of the current systems seem to not be supporting ssh: at all (even though it was introduced in Docker 18.09), but instead rely on some other system creating a tunnel for them (in case the server does not support the old tcp: that is).

i.e. like Java or JavaScript, or what have you... (interestingly, even Windows supports unix domain sockets these days)
Like the library that Podman Desktop is using, that requires it to create a local tmp socket rather than use it directly.

afbjorklund · 2024-09-01T18:17:46Z

If wanting to support the Docker syntax, then there also needs to be something added for the default path. With the docker client, the $DOCKER_HOST is coming from the environment on the remote side but with podman it is local...

So --host ssh://my-host is not going to fly with podman, because it doesn't have a path to podman.sock - in pretty much the same way that --host ssh://my-host/run/podman/podman.sock fails with a docker client because it does have a path.

afbjorklund · 2024-09-01T18:41:38Z

The patched version looks like this:

Error: unable to connect to Podman socket: Get "http://d/v4.9.3/libpod/_ping": ssh: rejected: connect failed (open failed)

Instead of the default podman version:

Error: unable to connect to Podman socket: failed to connect: dial tcp: lookup lima-podman: no such host

So the parsing of the ssh config goes OK:

DEBU[0000] Alias: lima-podman                           
DEBU[0000] User: anders                                 
DEBU[0000] Hostname: 127.0.0.1                          
DEBU[0000] Port: 39901                                  
DEBU[0000] Identity: "/home/anders/.lima/_config/user"

commit 59c8f28 (https://github.com/kevinburke/ssh_config)

But the connection still fails, due to path.

And it is hard to "guess" (need to read env)

podman --url ssh:// fails wierdly #17452

The remote uid (and thus systemd dir) could be anything.

Something similar to this, connect to the remote server and try to find the socket path.

https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

I think with Podman, you would call podman info on the remote side and parse the output?

  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock

Like so:

podman info --format '{{.Host.RemoteSocket.Path}}'

And use this value, instead of the empty path in url (if none has been provided by the caller)

DEBU[0000] Path: /run/user/1000/podman/podman.sock

commit a928ac1 (https://pkg.go.dev/golang.org/x/crypto/ssh)

Found another bug with the user being required, similar to the earlier bug with port being required...

It tried to authenticate as user "", instead of the current user. The workaround was adding a $USER@

commit ce4ccbd (like 14ef6a9)

Will rebase to HEAD

afbjorklund · 2024-09-01T20:43:37Z

Parsing the basic 4 values and querying for the 5th seems to be working - even with "golang":

$ podman-remote --host ssh://lima-podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp: lookup lima-podman: no such host
$ ./bin/podman-remote --host ssh://lima-podman version
Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.22.3
Git Commit:   a928ac1438058eaed6aacc17fd03dea41df8c432
Built:        Sun Sep  1 22:29:36 2024
OS/Arch:      linux/amd64

Server:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Wed Aug 21 02:00:00 2024
OS/Arch:      linux/amd64

It doesn't use any of the other features of the ssh_config, so that does need the "native" ssh...

The most important one is the ControlMaster auto*

* https://docs.docker.com/engine/security/protect-access/#ssh-tips

Others include using AES hardware crypto (where available), etc etc

Lima handles this automatically, when setting up the port forwarding:

portForwards:
- guestSocket: "/run/user/{{.UID}}/podman/podman.sock"
  hostSocket: "{{.Dir}}/sock/podman.sock"

evolutics · 2024-09-07T09:58:36Z

Thank you very much for your hard work on this.

For anyone stumbling across this issue, above mentioned workaround with the SSH tunnel looks like so:

Keep

ssh -F ssh_config -L "${PWD}/my.sock:/run/user/1000/podman/podman.sock" my-host

running, where -F ssh_config can be omitted if ~/.ssh/config should apply. Also, /run/user/1000/podman/podman.sock is just an example; it may be different on your remote host (see podman info).

Now podman --host "unix://${PWD}/my.sock" ps works.

afbjorklund · 2024-09-07T12:07:57Z

For anyone stumbling across this issue, above mentioned workaround with the SSH tunnel...

It's not only a workaround, but also an optimization that is used by Podman Desktop (and Lima)

...podman.sock is just an example; it may be different on your remote host (see podman info).

i.e. ssh -F ssh_config my-host podman info, not the socket location on the local host (client).

One remaining issue is how to separate a local unix socket, versus a "remote" tunneled unix socket

Remote podman connections shows tmp endpoint and not remote endpoint podman-desktop/podman-desktop#8353

It might need some kind of UUID for the host, to see if we are still on the same machine - or not.

Indicator of Podman Remote in UI podman-desktop/podman-desktop#8281

afbjorklund · 2024-09-08T16:35:30Z

Another (actual) workaround would be to use ssh -G, with the ssh -F if needed.

And let ssh handle the parsing of the ssh_config, and just look for the results:

$ ssh -G lima-podman | grep -E "^(user|hostname|port|identityfile) "
user anders
hostname 127.0.0.1
port 38241
identityfile /home/anders/.lima/_config/user

And then feed that as input to podman system connection add, to look up the path.

Docker supports `~/.ssh/config` but not custom SSH config files (docker/cli#1301). Podman does not support `~/.ssh/config` but might partially do so soon (containers/podman#23831).

github-actions · 2024-10-09T00:09:00Z

A friendly reminder that this issue had no activity for 30 days.

evolutics added the kind/bug Categorizes issue or PR as related to a bug. label Sep 1, 2024

afbjorklund mentioned this issue Sep 3, 2024

Add support for ssh_config for connection #23847

Merged

github-actions bot added the stale-issue label Oct 9, 2024

openshift-merge-bot bot closed this as completed in #23847 Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman over SSH in native mode does not use custom config `~/.ssh/config` #23831

Podman over SSH in native mode does not use custom config `~/.ssh/config` #23831

evolutics commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 •

edited

Loading

evolutics commented Sep 1, 2024

afbjorklund commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

Luap99 commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

evolutics commented Sep 7, 2024

afbjorklund commented Sep 7, 2024 •

edited

Loading

afbjorklund commented Sep 8, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024

Podman over SSH in native mode does not use custom config ~/.ssh/config #23831

Podman over SSH in native mode does not use custom config ~/.ssh/config #23831

Comments

evolutics commented Sep 1, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

afbjorklund commented Sep 1, 2024 • edited Loading

evolutics commented Sep 1, 2024

afbjorklund commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 • edited Loading

afbjorklund commented Sep 1, 2024 • edited Loading

Luap99 commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 • edited Loading

afbjorklund commented Sep 1, 2024

afbjorklund commented Sep 1, 2024 • edited Loading

afbjorklund commented Sep 1, 2024 • edited Loading

evolutics commented Sep 7, 2024

afbjorklund commented Sep 7, 2024 • edited Loading

afbjorklund commented Sep 8, 2024 • edited Loading

github-actions bot commented Oct 9, 2024

Podman over SSH in native mode does not use custom config `~/.ssh/config` #23831

Podman over SSH in native mode does not use custom config `~/.ssh/config` #23831

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024 •

edited

Loading

afbjorklund commented Sep 7, 2024 •

edited

Loading

afbjorklund commented Sep 8, 2024 •

edited

Loading