Problem with Docker 1.2.0 and agent 5.0.0

[andyshinn]

I'm having a problem with docker ps hanging after starting the Docker agent on Ubuntu 14.04 and Docker 1.2.0. I'm not sure if this is a Docker 1.2.0 problem or a Datadog problem. But I'm only able to produce it with the Datadog agent.

After starting the agent with the Docker plugin enabled, the /containers/json endpoint stops responding:

$ echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock

If I stop the Datadog agent and restart Docker, the endpoint starts working again:

$ echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock
HTTP/1.0 200 OK
Content-Type: application/json
Date: Sun, 31 Aug 2014 23:34:28 GMT
Content-Length: 2

Any ideas?

[andyshinn]

I was able to confirm it has something to do with Datadog and the Docker 1.2.0 upgrade. Here are some steps to reproduce on Ubuntu 14.04 (reproduced in EC2):

• Install Ubuntu 14.04
• Get the Docker APT repository installed:

if [ ! -e /usr/lib/apt/methods/https ]; then
    sudo apt-get update
    sudo apt-get install -y apt-transport-https
fi

echo deb https://get.docker.io/ubuntu docker main sudo tee -a /etc/apt/sources.list.d/docker.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo apt-get update

• Install Docker 1.1.2 - sudo apt-get install -y lxc-docker-1.1.2
• Start the Datadog container - sudo docker run -d --privileged --name dd-agent -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=<your_key> datadog/docker-dd-agent
• Start some other container - sudo docker run -d --name sleep ubuntu:trusty sleep 99999999
• sudo docker ps works fine

Now we upgrade Docker to 1.2.0 and do the same test.

• Kill the containers - sudo docker rm -f dd-agent sleep
• Upgrade Docker - sudo apt-get -y install lxc-docker-1.2.0
• Restart daemon - sudo service docker restart
• Start the Datadog container - sudo docker run -d --privileged --name dd-agent -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=<your_key> datadog/docker-dd-agent
• Start some other container - sudo docker run -d --name sleep ubuntu:trusty sleep 99999999
• After a few moments sudo docker ps will stop responding

[LotharSee]

Thanks for all your work @andyshinn !

I managed to reproduce this same issue on both CoreOS 427 and Debian 7.
However this happened to me only when the Datadog Agent container was running: the issue I identifies doesn't occur when the Agent run outside of the container. Can you confirm me this?

• Run your Ubuntu 14.04, with Docker 1.2.0 installed, no other container.
• Run some container but the Agent: sudo docker run -d --name sleep ubuntu:trusty sleep 99999999
• Install the Agent on your host (According to https://app.datadoghq.com/account/settings#agent/ubuntu )
• Enable the Docker integration on your host https://app.datadoghq.com/account/settings#integrations/docker
• Restart your Datadog Agent sudo service datadog-agent restart
• Check if it works or not (try a sudo docker ps, check your metrics in Datadog, run the netcat command, ...)

In my case, it works.
And if it's good for you too, I'll be able to release a fix quickly.

[andyshinn]

I just tested this some more. It still hangs even using the agent outside of a container (on the Ubuntu 14.04 host directly). I performed the steps you outlined on a fresh Ubuntu 14.04 EC2 instance. Let me know what other debug information you need.

It looks like this could also be related to bad Docker behavior in moby/moby#7905 and moby/moby#8010. So even if there is a workaround in Datadog, the ultimate fix might be from Docker.