docker from_env and pull is broken on mac

[Samreay]

Unusual issue here - code that was working on Thursday is no longer working today, so this might be tied into some behind-the-scenes update to Docker Desktop. On that note, I've also tried with explicit Docker Desktop versions 4.13.0, 4.12.0, 4.11.1 (and used the clean uninstaller to remove the VMs and temp folders left over) and all report the same issue.

Docker-desktop 4.10.1 does not produce an error, and it is the version I am now migrating to.

Reproduction:

1. conda create -n docker python=3.10 && conda activate docker
2. pip install docker
3. python -c "import docker; client = docker.from_env()"

The final command fails.

Traceback (most recent call last):
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

By specifying version="1.35" when making a client, it will not fail, but then any subsequent client.images.pull will fail regardless:

python -c "import docker; client = docker.from_env(version='1.35'); client.images.pull('docker:latest');"

  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
    pull_log = self.client.api.pull(
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/api/image.py", line 424, in pull
    response = self._post(
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/api/client.py", line 233, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 577, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

I can confirm docker is running during these as well. Screenshot attached.

I separate team member has been able to reproduce this, but on their m1 hardware (I ran it on my x86 based mac).

[pdoorr]

Same here...

[mndeveci]

We see a similar issue when we call client.ping() method. aws/aws-sam-cli#4329

[mndeveci]

I was able to track down this issue. With latest release of docker, the context of the client is changed from default to desktop-linux which uses different endpoint and therefore breaks the docker client.

Run docker info to see which context client uses;

$ docker info
Client:
 Context:    desktop-linux
...

And the run docker context ls to find out its endpoint configuration;

$ docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                  KUBERNETES ENDPOINT   ORCHESTRATOR
default             moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                            swarm
desktop-linux *     moby                                                          unix:///Users/ec2-user/.docker/run/docker.sock

Then setting DOCKER_HOST to the endpoint that you see for the default context should resolve the issue.

But the problem is, how can we know this endpoint configuration, if we can't connect to docker daemon?

Maintainers, we need your feedback to resolve this issue.

[danbf]

this took our docker desktops offline as well. we kinda feel we need to set "disableUpdate": true in the docker desktop config to avoid things breaking our desktops.

[nicks]

Thanks for the report! We're trying to make Docker Desktop more userspace-friendly (e.g., fewer root privileges). As part of that, we're migrating the sockets to user-owned directories rather than root-owned directories (like /var/run). Desktop 4.13 no longer installs a socket under /var/run.

For now you can work around this by creating the symlink manually:

sudo ln -s "$HOME/.docker/run/docker.sock" /var/run/docker.sock

We're currently working on rolling this change back and figuring out how to roll it out in a more orderly way, including updating the docker-py library to detect this case. See docker/for-mac#6529 for more info.

[milas]

Please upgrade to Docker Desktop 4.13.1 if you were encountering this issue!

To echo what @nicks said, the change in Docker Desktop was reverted, and if/when it's re-introduced, we'll make sure there's a clear path to better support it.

[pdoorr]

Upgraded, now it starts but after a while stop unexpectedly... Il giorno mer 2 nov 2022 alle ore 20:34 Milas Bowman < ***@***.***> ha scritto:

…

Please upgrade to Docker Desktop 4.13.1 if you were encountering this issue! To echo what @nicks <https://github.com/nicks> said, the change in Docker Desktop was reverted, and if/when it's re-introduced, we'll make sure there's a clear path to better support it. — Reply to this email directly, view it on GitHub <#3059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3VFM6FKPMEPQGXMCXI2GTWGK63HANCNFSM6AAAAAARMU77NE> . You are receiving this because you commented.Message ID: ***@***.***>

[milas]

@pdoorr Please open a new issue for that with your OS/environment info and any other context. This issue was about the /var/run/docker.sock symlink being removed in Docker Desktop 4.13.0, which was re-added in 4.13.1, which would result in FileNotFoundError immediately. Stopping unexpectedly after starting is a different issue.

[fabiosanger]

Docker desktop version is 4.13.1 but i still get the same error

[peteristhegreat]

If you are running Rancher Desktop you need this line:

sudo ln -s ~$USER/.rd/docker.sock /var/run/docker.sock

rancher-sandbox/rancher-desktop#2534