Compatability with docker client #11668

jeremy-chua · 2021-09-21T05:57:21Z

Pardon me if its the wrong channel to ask this question.
I installed podman on Centos 8 steam using the command

sudo dnf install docker

Here's the output of docker version

Version: 3.3.1
API Version: 3.3.1
Go Version: go1.16.7
Built: Wed Sep 1 01:35:01 2021
OS/Arch: linux/amd64

I have a docker client on a windows which i added a context with ssh

docker context create remote --docker host=ssh://[email protected]

The docker client version is

Client:
Cloud integration: 1.0.17
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:58:50 2021
OS/Arch: windows/amd64
Context: d1
Experimental: true

I got this error when executing docker --context remote ps

error during connect: Get "http://docker.example.com/v1.24/version": command [ssh -l admin -- 192.168.1.121 docker system dial-stdio] has exited with exit status 125, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Error: unrecognized command podman system dial-stdio
Try 'podman system --help' for more information.

Any comments will be greatly appreciated

The text was updated successfully, but these errors were encountered:

vrothberg · 2021-09-21T07:40:11Z

@jwhonce PTAL

rhatdan · 2021-09-21T09:31:05Z

Are you sure podman is configured to listen on /run/docker.sock?

afbjorklund · 2021-09-21T10:57:29Z

When using the docker client, it needs the (real) docker program to be installed on the server as well.

docker-ce-cli (https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-cli-20.10.8-3.el8.x86_64.rpm)

Usage:  docker system dial-stdio

Proxy the stdio stream to the daemon connection. Should not be invoked manually.

You can see when running the client with the -D option, it runs: docker system dial-stdio over ssh...

$ export DOCKER_HOST=ssh://root@localhost:35125
$ docker -D version
DEBU[0000] commandconn: starting ssh with [-l root -p 35125 -- localhost docker system dial-stdio]

But then you can ask it to talk to the podman socket, by setting the $DOCKER_HOST in the bashrc.

$ ssh -l root -p 35125 localhost 'echo $DOCKER_HOST'
unix:///run/podman/podman.sock

rhatdan · 2021-09-21T14:22:01Z

I think this is enough info to allow me to close. Reopen if I am mistaken.

afbjorklund · 2021-09-21T14:39:51Z

Would it be feasible to implement podman system dial-stdio, so that it would work with docker clients when using podman-docker ? I'm not sure exactly what it does, but it seems to be some kind of http proxy.

// Dialer returns a dialer for a raw stream connection, with HTTP/1.1 header, that can be used for proxying the daemon connection.

Then again, CoreOS already includes docker

So docker clients work with podman machine

rhatdan · 2021-09-21T14:51:48Z

Anything is possible, just need volunteers.
docker does not seem to document this, and it seems to be a hidden feature.

jeremy-chua · 2021-09-22T06:28:43Z

@afbjorklund
there's a conflict when i install docker-ce-cli

[admin@node1 ~]$ sudo rpm -i docker-ce-cli-20.10.8-3.el8.x86_64.rpm
error: Failed dependencies:
docker-scan-plugin(x86-64) is needed by docker-ce-cli-1:20.10.8-3.el8.x86_64
docker conflicts with docker-ce-cli-1:20.10.8-3.el8.x86_64

afbjorklund · 2021-09-22T07:33:43Z

I think podman-docker and docker-ce-cli will conflict, since they both provide /usr/bin/docker

# yum localinstall --allowerasing docker-ce-cli-20.10.8-3.el8.x86_64.rpm docker-scan-plugin-0.8.0-3.el8.x86_64.rpm

================================================================================
 Package       Arch   Version                                Repository    Size
================================================================================
Installing:
 docker-ce-cli x86_64 1:20.10.8-3.el8                        @commandline  29 M
 docker-scan-plugin
               x86_64 0.8.0-3.el8                            @commandline 4.2 M
Removing dependent packages:
 podman-docker noarch 3.2.3-0.10.module_el8.4.0+886+c9a8d9ad @appstream   230  

Transaction Summary
================================================================================
Install  2 Packages
Remove   1 Package

Total size: 34 M
Is this ok [y/N]:

As mentioned, you also need to set $DOCKER_HOST to the podman socket. Or you will get error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

matejvasek · 2021-09-23T19:06:25Z

I think that docker system dial-stdio is basically socat - /run/var/docker.sock.

matejvasek · 2021-09-23T19:07:25Z

I don't know why they do it that way when it can be tunneled purely in Go (like the podman).

matejvasek · 2021-09-23T19:12:39Z

Although execing ssh binary also has its advantages: they don't need to care about security updates, also it will pick up system ssh settings (e.g. preferred signature algorithm) and it also checks known_host and so on.

rhatdan · 2021-09-23T19:21:40Z

Do tools other then Docker client require docker system dial-stdio?

matejvasek · 2021-09-23T19:30:00Z

Do tools other then Docker client require docker system dial-stdio?

I cannot tell. From what I've seen it's only docker CLI, and in some discussion somebody mentioned that also Python SDK uses it I think it was @afbjorklund .

matejvasek · 2021-09-23T19:32:44Z

#11643 (comment)

matejvasek · 2021-09-23T19:34:40Z

@jeremy-chua I think that once you install docker-ce-cli it could work, however you might need put something like:
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock into your .bashrc in the VM.

afbjorklund · 2021-09-23T19:41:55Z

@rhatdan docker-py ("pip install docker") also requires docker dial stdio:

https://github.com/docker/docker-py/blob/master/docker/transport/sshconn.py#L46

The other SDK don't support ssh: URLs at all, only unix: and tcp:

Thus the need to continue with the ssh -L workarounds, it was meant to avoid...

i.e. PR #11643

matejvasek · 2021-09-23T19:44:22Z

btw I just tried to replace docker binary by this script

#/bin/bash

socat - /run/user/1000/podman/podman.sock

and it worked 😄

afbjorklund · 2021-09-23T19:46:28Z

however you might need put something like:
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock into your .bashrc in the VM.

The recommendation was to set rootless podman for ~core and root podman for ~root.

Less surprises, that way: #11643 (comment)

You need to do the same kind of configuration, in order to run rootless docker over ssh:

https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

afbjorklund · 2021-09-23T19:48:23Z

I think docker dial stdio was implemented specifically to avoid a dependency on socat 😀

matejvasek · 2021-09-23T19:49:41Z

I think docker dial stdio was implemented specifically to avoid a dependency on socat

quite likely

afbjorklund · 2021-09-23T19:51:49Z

See https://docs.docker.com/engine/release-notes/18.09/#18090 and docker/cli#1014

I do find it valuable if we had a helper binary on daemon side so that we don't have a hard dependency on socat for this or leak a container to docker ps for every connection. Especially because it is very simple logic to implement. I expect that a helper binary/flag in the same host as dockerd can dial into local dockerd to access the API. I don't even mind if socat is the fallback for older daemons.

jeremy-chua · 2021-09-23T20:45:36Z

thanks guys!!!

honestly, i had a hard time getting rootless podman with cgroup2 to work. I needed to limit resource for containers.
i went back to docker-ce

will explore further once i have more time. :)

rhatdan · 2021-09-23T21:17:19Z

Anyone interested in opening a PR for podman system dial-stdio

mheon · 2021-09-28T16:03:50Z

I'll reopen and tag as a feature for adding that command.

AkihiroSuda · 2021-10-05T14:38:43Z

I don't know why they do it that way when it can be tunneled purely in Go (like the podman).

There was a long story: moby/moby#33566 (comment)

Go implementation of ssh was avoided because it can’t easily support ~/.ssh/config
ssh -L was avoided because there was a security concern about exposing API on a socket on the local filesystem
socat was avoided because it is not always installed on the remote host

Adding dial-stdio CLI cmd Signed-off-by: Jake Parks <[email protected]> Made dial-stdio URI configurable Slight refactors Signed-off-by: Jake Parks <[email protected]> Added simple test for existence of `podman system dial-stdio` command Fix 'system dial-stdio' integration tests Changed link in comment to permalink

cmoulliard · 2021-10-20T07:40:15Z

Should we keep this ticket opened in order also to cover the changes needed to simplify the life of the macos, windows users when they will access the podman vm from their laptop as currently this is not evident to figure out what they should do to access properly the DOCKER daemon :

1. Cannot connect to the docker daemon

Error: reported by the docker client when we try to get a ps, images Cannot connect to the docker daemon
Fixed using export DOCKER_HOST=ssh://root@localhost:<VM_PORT>
Question: How can we get the port number to be used ?

2. Host key verification failed

Error: If the docker_host env var is set correctly, then we will got another message : error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22io.x-k8s.kind.cluster%3Dkind%22%3Atrue%7D%7D": command [ssh -l root -p 50370 -- localhost docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Host key verification failed.
Fixed after editing the ~/.ssh/config file to add the follow properties

Host localhost
      HostName 127.0.0.1
      IdentityFile ~/.ssh/<PODMAN_MACHINE_NAME>
      StrictHostKeyChecking no

3. unrecognized command `docker system dial-stdio`

Error: When we try to create a k8s cluster (= it runs as a container), we got this error as the docker client is not installed within the podman VM

ERROR: failed to delete cluster "kind": error listing nodes: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=kind --format '{{.Names}}'" failed with error: exit status 1
Command Output: error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22io.x-k8s.kind.cluster%3Dkind%22%3Atrue%7D%7D": command [ssh -l root -p 50370 -- localhost docker system dial-stdio] has exited with exit status 125, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Error: unrecognized command `docker system dial-stdio`
Try 'docker system --help' for more information.

Fixed: Within the VM, edit the .bashrc file of the core's|root's user and add the following function

docker () {
    if [ "$1" = "system" ] && [ "$2" = "dial-stdio" ]; then
      exec socat - "/run/user/1000/podman/podman.sock"
    fi
    exec /usr/bin/docker $@
}

REMARK:
For root user change exec socat - "/run/user/1000/podman/podman.sock to exec socat - "/run/podman/podman.sock

On the laptop

export DOCKER_HOST=ssh://core@localhost:50370 // rootless
or
export DOCKER_HOST=ssh://root@localhost:50370 // rootfull

4. TCP port issue

Error: If you try to create by example a k8s cluster using kind, then kind will report errors as it will not be able to access the needed ports such as: 80, 6443, ...
Fix: Change the Selinux mode to Permissive using the command setenforce 0

cmoulliard · 2021-10-20T08:01:56Z

Would it be feasible to implement podman system dial-stdio, so that it would work with docker clients when using podman-docker ? I'm not sure exactly what it does, but it seems to be some kind of http proxy.

I'm also getting the same error when I try to create a kind k8s cluster (= container) using the docker daemon running within the podman VM (= fed34)

ssh -l root -p 50370 -- localhost docker system dial-stdio
Error: unrecognized command `docker system dial-stdio`
Try 'docker system --help' for more information.

or

ERROR: failed to delete cluster "kind": error listing nodes: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=kind --format '{{.Names}}'" failed with error: exit status 1
Command Output: error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22io.x-k8s.kind.cluster%3Dkind%22%3Atrue%7D%7D": command [ssh -l root -p 50370 -- localhost docker system dial-stdio] has exited with exit status 125, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Error: unrecognized command `docker system dial-stdio`
Try 'docker system --help' for more information.

matejvasek · 2021-10-20T10:08:42Z

@cmoulliard I believe there is already merged PR implementing dial-stdio #11819. Furthermore @n1hility is working on automatic socket tunneling containers/gvisor-tap-vsock#58.

matejvasek · 2021-10-20T10:09:32Z

And I am working on support of ssh connections in pack buildpacks/pack#1282.

matejvasek · 2021-10-20T10:13:19Z

workaround:
put

if ! which docker; then
  docker () {
    if [ "$1" = "system" ] && [ "$2" = "dial-stdio" ]; then
          if [ "$3" = "--help" ]; then
            echo "\nProxy the stdio stream to the daemon connection."
            exit 0
          else
            exec socat - "/run/user/1000/podman/podman.sock"
          fi
    fi
    exec podman $@
  }
fi

into VM's .bashrc.

cmoulliard · 2021-10-20T12:48:47Z

into VM's .bashrc.

I got an error as fi was not defined

-bash: /var/home/core/.bashrc: line 41: syntax error: unexpected end of file

cat the file

...

if ! which docker; then
docker () {
  if [ "$1" = "system" ] && [ "$2" = "dial-stdio" ]; then
        if [ "$3" = "--help" ]; then
          echo "\nProxy the stdio stream to the daemon connection."
          exit 0
        else
          exec socat - "/run/user/1000/podman/podman.sock"
        fi
  fi
  exec docker $@
}
fi # <!--- MISSING FI

The problem is still there even using the workaround

VM

podman machine ssh fed34
Connecting to vm fed34. To close connection, use `~.` or `exit`
Warning: Permanently added '[127.0.0.1]:50370' (ECDSA) to the list of known hosts.
Fedora CoreOS 34.20211004.2.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Last login: Wed Oct 20 12:51:01 2021 from 192.168.127.1
/usr/local/bin/docker
[core@localhost ~]$ ls -la /usr/local/bin/docker
lrwxrwxrwx. 1 root root 15 Oct 20 07:24 /usr/local/bin/docker -> /usr/bin/podman

From laptop

kind-reg-ingress.sh 
Do you want to delete the kind cluster (yes|no) - Default: no ? yes
Which kubernetes version should we install (1.14 .. 1.22) - Default: 1.21 ? 
What logging verbosity do you want (0..9) - A verbosity setting of 0 logs only critical events - Default: 0 ? 
Deleting kind cluster ...
Deleting cluster "kind" ...
ERROR: failed to delete cluster "kind": error listing nodes: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=kind --format '{{.Names}}'" failed with error: exit status 1
Command Output: error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22io.x-k8s.kind.cluster%3Dkind%22%3Atrue%7D%7D": command [ssh -l root -p 50370 -- localhost docker system dial-stdio] has exited with exit status 125, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Error: unrecognized command `docker system dial-stdio`
Try 'docker system --help' for more information.

cmoulliard · 2021-10-20T13:16:21Z

Problem of the workaround fixed

Within the VM, edit the .bashrc file of the core's|root's user

docker () {
    if [ "$1" = "system" ] && [ "$2" = "dial-stdio" ]; then
      exec socat - "/run/user/1000/podman/podman.sock"
    fi
    exec /usr/bin/docker $@
}

REMARK: For root user change exec socat - "/run/user/1000/podman/podman.sock" to exec socat - "/run/podman/podman.sock"

On the laptop

export DOCKER_HOST=ssh://core@localhost:50370 // rootless
or
export DOCKER_HOST=ssh://root@localhost:50370 // rootfull

matejvasek · 2021-10-20T13:22:34Z

exec socat - "/run/user/1000/podman/podman.sock"

you might want to change the line for root's .bashrc to "/run/podman/podman.sock"

cmoulliard · 2021-10-20T13:28:51Z

you might want to change the line for root's .bashrc

I updated my comment

afbjorklund · 2021-10-20T19:46:39Z

Question: How can we get the port number to be used ?

@cmoulliard : as discussed in chat, to get the address of a podman machine you use podman system connection.

The relationship between the two commands is not that obvious, and it is only the name that links them together.

$ podman4 machine list
NAME                     VM TYPE     CREATED       LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  qemu        39 hours ago  Currently running  1           2.147GB     10.74GB
$ podman4 system connection ls
Name                         Identity                                  URI
podman-machine-default*      /home/anders/.ssh/podman-machine-default  ssh://core@localhost:44133/run/user/1000/podman/podman.sock
podman-machine-default-root  /home/anders/.ssh/podman-machine-default  ssh://root@localhost:44133/run/podman/podman.sock

You can also see the command in the debug log:

$ podman4 --log-level debug machine ssh
INFO[0000] podman4 filtering at log level debug 
...
DEBU[0000] Executing: ssh [-i /home/anders/.ssh/podman-machine-default -p 44133 core@localhost -o UserKnownHostsFile /dev/null -o StrictHostKeyChecking no] 
Warning: Permanently added '[localhost]:44133' (ECDSA) to the list of known hosts.
...

There you also see the other needed ssh config...

However, that warning at the end is a bug. It needs some additional configuration to the ssh command, to hide it.

-o UserKnownHostsFile /dev/null -o StrictHostKeyChecking no

-o LogLevel=quiet -o NoHostAuthenticationForLocalhost=yes

rhatdan closed this as completed Sep 21, 2021

mheon reopened this Sep 28, 2021

mheon added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 28, 2021

afbjorklund mentioned this issue Sep 30, 2021

How to access docker api on a mac #11397

Closed

trynaeat mentioned this issue Sep 30, 2021

Adding dial-stdio CLI cmd #11819

Merged

openshift-merge-robot closed this as completed in #11819 Oct 11, 2021

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Compatability with docker client #11668

Compatability with docker client #11668

Comments

jeremy-chua commented Sep 21, 2021

vrothberg commented Sep 21, 2021

rhatdan commented Sep 21, 2021

afbjorklund commented Sep 21, 2021 • edited Loading

rhatdan commented Sep 21, 2021

afbjorklund commented Sep 21, 2021 • edited Loading

rhatdan commented Sep 21, 2021

jeremy-chua commented Sep 22, 2021

afbjorklund commented Sep 22, 2021 • edited Loading

matejvasek commented Sep 23, 2021

matejvasek commented Sep 23, 2021 • edited Loading

matejvasek commented Sep 23, 2021 • edited Loading

rhatdan commented Sep 23, 2021

matejvasek commented Sep 23, 2021

matejvasek commented Sep 23, 2021

matejvasek commented Sep 23, 2021

afbjorklund commented Sep 23, 2021 • edited Loading

matejvasek commented Sep 23, 2021 • edited Loading

afbjorklund commented Sep 23, 2021 • edited Loading

afbjorklund commented Sep 23, 2021

matejvasek commented Sep 23, 2021

afbjorklund commented Sep 23, 2021 • edited Loading

jeremy-chua commented Sep 23, 2021 • edited Loading

rhatdan commented Sep 23, 2021

mheon commented Sep 28, 2021

AkihiroSuda commented Oct 5, 2021

cmoulliard commented Oct 20, 2021 • edited Loading

1. Cannot connect to the docker daemon

2. Host key verification failed

3. unrecognized command docker system dial-stdio

4. TCP port issue

cmoulliard commented Oct 20, 2021

matejvasek commented Oct 20, 2021 • edited Loading

matejvasek commented Oct 20, 2021

matejvasek commented Oct 20, 2021 • edited Loading

cmoulliard commented Oct 20, 2021 • edited Loading

cmoulliard commented Oct 20, 2021 • edited Loading

matejvasek commented Oct 20, 2021

cmoulliard commented Oct 20, 2021

afbjorklund commented Oct 20, 2021 • edited Loading

afbjorklund commented Sep 21, 2021 •

edited

Loading

afbjorklund commented Sep 21, 2021 •

edited

Loading

afbjorklund commented Sep 22, 2021 •

edited

Loading

matejvasek commented Sep 23, 2021 •

edited

Loading

matejvasek commented Sep 23, 2021 •

edited

Loading

afbjorklund commented Sep 23, 2021 •

edited

Loading

matejvasek commented Sep 23, 2021 •

edited

Loading

afbjorklund commented Sep 23, 2021 •

edited

Loading

afbjorklund commented Sep 23, 2021 •

edited

Loading

jeremy-chua commented Sep 23, 2021 •

edited

Loading

cmoulliard commented Oct 20, 2021 •

edited

Loading

3. unrecognized command `docker system dial-stdio`

matejvasek commented Oct 20, 2021 •

edited

Loading

matejvasek commented Oct 20, 2021 •

edited

Loading

cmoulliard commented Oct 20, 2021 •

edited

Loading

cmoulliard commented Oct 20, 2021 •

edited

Loading

afbjorklund commented Oct 20, 2021 •

edited

Loading