Docker module not compatible with private registry and internal/self-signed SSL certificate

[mattupstate]

Issue Type:

Bug Report

Ansible Version:

1.7.2

Environment:

Ubuntu 14.04

Summary:

I've deployed a private Docker registry, configured such that non-volatile requests (GET, OPTIONS) do not require authentication. However, pushing images does require authentication and the Docker CLI requires that the connection be HTTPS when it has to send credentials. So I've configured the registry as such and I've added my internal CA to the system certificates. Now I can run normal Docker CLI commands without any SSL problems.

However, I believe therein lies a problem with the usage of the docker-py library. The docker-py library uses the requests library to interact with the registry. Additionally, requests vendors its own CA bundle effectively making my change to the system certificates mute. requests does support some environment variables, such as REQUESTS_CA_BUNDLE to point to a different bundle. In a naive attempt to fix this I added REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt to /etc/environment but that didn't work.

So I'm not sure if this can be solved here, perhaps it can and makes sense, but perhaps its also partly a problem with docker-py. There is an issue that might be related on that project as well: docker/docker-py#317

Steps To Reproduce:

1. Deploy a private Docker regsitry with HTTPS only
2. Configure HTTPS to use an internal CA/self-signed certificate
3. Use the docker module to run a container from an internal registry image:

  - name: start app service
    docker:
      image: "docker.<internal-registry-hostname>/app"

Expected Results:

A running Docker container

Actual Results:

TASK: [start app service] *****************************************************
failed to parse json:     repository, insecure=insecure_registry
  File "/usr/local/lib/python2.7/dist-packages/docker/auth/auth.py", line 69, in resolve_repository_name
    return expand_registry_url(parts[0], insecure), parts[1]
  File "/usr/local/lib/python2.7/dist-packages/docker/auth/auth.py", line 48, in expand_registry_url
    "HTTPS endpoint unresponsive and insecure mode isn't enabled."
docker.errors.DockerException: HTTPS endpoint unresponsive and insecure mode isn't enabled.

fatal: [10.80.155.196] => Traceback (most recent call last):
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/runner/__init__.py", line 561, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/runner/__init__.py", line 666, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/runner/__init__.py", line 884, in _executor_internal_inner
    result = handler.run(conn, tmp, module_name, module_args, inject, complex_args)
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/runner/action_plugins/normal.py", line 57, in run
    return self.runner._execute_module(conn, tmp, module_name, module_args, inject=inject, complex_args=complex_args)
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/runner/__init__.py", line 535, in _execute_module
    data = utils.parse_json(res['stdout'], from_remote=True)
  File "/Users/matt/.virtualenvs/infra/lib/python2.7/site-packages/ansible/utils/__init__.py", line 444, in parse_json
    tokens = shlex.split(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shlex.py", line 279, in split
    return list(lex)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

[mattupstate]

After some research I've discovered that kennethreitz/requests#1347 fixes this issue for me because I was hosting two SSL sites on one machine.

[lenfree]

hey @mattupstate I am having this same issue and I don't know if I have configured my Nginx incorrectly. Could you tell me how you fixed it? :)

[mattupstate]

@lenfree The issue does not reside with Nginx. It resides with Python not supporting SNI out of the box. I've forgone deploying private registry images with Ansible because of this.

[lenfree]

Hi @mattupstate, I installed the below packages for the SNI support, but I am still having the same error. I have port 80 redirected to port 443, disabled authentication as well. hmm..

pyopenssl
ndg-httpsclient
pyasn1

[lenfree]

tried to debug the code, and the issue is when it is doing a check

if utils.ping('https://' + hostname + '/v1/_ping'):

and this function is making an https requests but it can't find the cert.

requests.get('my.registry', timeout=3)

Furthermore, I did an strace on the request, and it reads this cacert file:

/usr/lib/python2.6/site-packages/requests/cacert.pem

Is there a config somewhere that we can specify where the cert path is?

[mattupstate]

@lenfree Thats exactly right, thats why I stopped/can't use Ansible to deploy private Docker images right now. Unless there is some way to set one of two environment variables during the context of an Ansible task, you're out of luck. I haven't figured out how to do that.

What I thought would work was adding our private CA cert to the system bundle, but it appears that the requests library doesn't not currently consider that bundle out of the box. That said, the two environment variables you can set are: CURL_CA_BUNDLE or REQUESTS_CA_BUNDLE that point to a, you guessed it, a CA bundle.

The official Docker client/cli, behaves differently. If you don't already know, you must place your private CA certificate in a predictable folder. That folder is:

/etc/docker/certs.d/<registry-host-name>/<registry-host-name>.crt

[bcoca]

the environment directive allows you to set CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE