Registry freezes regularly whilst fetching image from S3

[pwaller]

Regularly docker pull fails by either hanging or crashing out.

I've narrowed the problem down, I can reproduce it reliably with a curl to http://.../v1/images/.../layer (which is ~100MiB) with no proxies in the way. The frequency seems to vary. Sometimes it is 1 in 15, sometimes it is 1 in 2.

The problem has a similar feel to #815.

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
  result = self._run(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/boto.py", line 75, in _fetch_part
  boto_key.get_contents_to_file(f, headers={'Range': brange})
File "/usr/local/lib/pyth  on2.7/dist-packages/boto/s3/key.py", line 1648, in get_contents_to_file
  response_headers=response_headers)  
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1480, in get_file
  query_args=None)  
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1533, in _get_file_internal
  for bytes in self:  
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 386, in next
  data = self.resp.read(self.BufferSize)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 413, in read
  return http_client.HTTPResponse.read(self, amt)
File "/usr/lib/python2.7/httplib.py", line 567, in read
  s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
  data = self._sock.recv(left)
File "/usr/local/lib/python2.7/dist-packages/gevent/ssl.py", line 208, in recv
  return self.read(buflen)
File "/usr/local/lib/python2.7/dist-packages/gevent/ssl.py", line 110, in read
  return self._sslobj.read(len)
error: [Errno 104] Connection reset by peer
<Greenlet at 0x7f9a8a95f410: <bound method ParallelKey._fetch_part of <docker_registry.core.boto.ParallelKey object at 0x7f9a8a8a8250>>('/tmp')

Docker client output looks like this:

Error pulling image (latest) from ***/***, endpoint: https://***/v1/, ApplyLayer exit status 1 flate: corrupt input before offset 13494754
Error pulling image (latest) from ***/***, ApplyLayer exit status 1 flate: corrupt input before offset 13494754
Container run failed: Error pulling image (latest) from ***/***, ApplyLayer exit status 1 flate: corrupt input before offset 13494754

[pwaller]

By the way, I wanted to report the docker registry version, but I cannot tell which version it is. I'm running image ID 985e98f8266b from the official docker registry which was pulled about 3 days ago.

[dmp42]

Here is a description of what and how to obtain a bunch of useful information:

https://github.com/docker/docker-registry/blob/master/DEBUGGING.md#basics

(the "basics" and "Your private registry" sections)

These infos will help a lot figuring out what's happening.

From a quick glance at your stacktrace, it looks like your communication with your S3 bucket is not working well.

[pwaller]

Another thing to mention is that I've reproduced the issue on two separate EC2 instances.

[pwaller]

I don't think "the basics" section matter since the issue is reproducible just with cURL and without docker in the way.

Result from the _ping endpoint with DEBUG:

{
  "versions": {
    "zlib": "1.0",
    "yaml": "3.11",
    "werkzeug": "0.9.6",
    "urllib2": "2.7",
    "urllib": "1.17",
    "tarfile": "$Revision: 85213 $",
    "simplejson": "3.6.2",
    "rsa": "3.1.4",
    "requests.utils": "2.3.0",
    "requests.packages.urllib3.packages.six": "1.2.0",
    "requests.packages.urllib3": "dev",
    "requests.packages.chardet": "2.2.1",
    "flask": "0.10.1",
    "email": "4.0.3",
    "docker_registry.server": "0.9.0",
    "docker_registry.core": "2.0.3",
    "docker_registry.app": "0.9.0",
    "distutils": "2.7.6",
    "decimal": "1.70",
    "ctypes": "1.1.0",
    "SocketServer": "0.4",
    "argparse": "1.1",
    "backports.lzma": "0.0.3",
    "blinker": "1.3",
    "boto": "2.34.0",
    "boto.vendored.six": "1.7.2",
    "cPickle": "1.71",
    "cgi": "2.6",
    "gevent": "1.0.1",
    "greenlet": "0.4.5",
    "gunicorn": "19.1.0",
    "gunicorn.arbiter": "19.1.0",
    "gunicorn.config": "19.1.0",
    "gunicorn.six": "1.2.0",
    "jinja2": "2.7.3",
    "json": "2.0.9",
    "logging": "0.5.1.2",
    "parser": "0.5",
    "pickle": "$Revision: 72223 $",
    "platform": "1.0.7",
    "python": "2.7.6 (default, Mar 22 2014, 22:59:56) \n[GCC 4.8.2]",
    "re": "2.2.1",
    "redis": "2.10.3",
    "requests": "2.3.0"
  },
  "launch": [
    "/usr/local/bin/gunicorn",
    "--access-logfile",
    "-",
    "--error-logfile",
    "-",
    "--max-requests",
    "100",
    "-k",
    "gevent",
    "--graceful-timeout",
    "3600",
    "-t",
    "3600",
    "-w",
    "4",
    "-b",
    "0.0.0.0:5000",
    "--reload",
    "docker_registry.wsgi:application"
  ],
  "host": [
    "Linux",
    "61d7f9799171",
    "3.17.2",
    "#2 SMP Thu Dec 11 02:25:47 EST 2014",
    "x86_64",
    "x86_64"
  ]
}

Starting with:

docker run \
    --rm \
    --env SETTINGS_FLAVOR=s3 \
    --env DEBUG=true \
    --env AWS_REGION=eu-west-1 \
    --env AWS_BUCKET=foo \
    --env AWS_SECURE=true \
    --env AWS_ENCRYPT=true \
    --name docker-registry \
    --publish 80:5000 \
    registry:latest

[pwaller]

Authentication is done via IAM instance role, hence why AWS_SECRET etc aren't supplied, it "just works" with boto.

(Side note: I'm concerned this might be broken with the NG registry, is it worth bringing this up in its own issue now, or should I wait?).

[dmp42]

@pwaller about this not being supported by next-generation, yes by all means go ahead and open a new issue (cc @BrianBland @AndreyKostov )

About your v1 issue, can you try without AWS_ENCRYPT and AWS_SECURE?

[AndreyKostov]

I've added support for IAM instance role authentication in next-generation in the upcoming s3 driver refactor pr

[dmp42]

@pwaller any news on trying without AWS_ENCRYPT and AWS_SECURE?

[pwaller]

We haven't done much over the holiday period. Glad to hear IAM authentication is in the rewrite.

Please re-ping if you come across this issue in greater than 10 days from now, since I should continue to experience it and be frustrated enough to do the AWS_ENCRYPT experiment if it is still a problem.

[pwaller]

Tidying up my personal issues list, so closing this. Please create a new issue if you're still interested in tracking it.

[dmp42]

I believe this was fixed by #961