Docker metrics when host is CoreOS

[andyshinn]

I am running the Docker container on a CoreOS host and it only reports on running container count. I am running by passing the cgroup data like:

docker run \
  --privileged \
  -h myddagent1 \
  -e API_KEY=231142123d509321fe38442e318634e5 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /sys/fs/cgroup/:/sys/fs/cgroup:ro \
  -v /home/core/datadog:/etc/dd-agent/conf.d \
  datadog/docker-dd-agent

It appears to be erroring with:

Instance #0[ERROR]: Exception('Cannot find Docker cgroup file. If you are using Docker 0.9 or 0.10, it is a known bug in Docker fixed in Docker 0.11',)

This appears to be due to a cgroup layout difference in CoreOS. Instead of a docker or lxc folder, there is a system.slice and user.slice. These contain IDs to containers in a differen't format:

$ ls -al /sys/fs/cgroup/cpuacct/
total 0
dr-xr-xr-x   4 root root   0 Jun 22 22:58 .
drwxr-xr-x  11 root root 300 Jun 22 22:58 ..
-rw-r--r--   1 root root   0 Jun 25 20:04 cgroup.clone_children
-rw-r--r--   1 root root   0 Jun 25 19:47 cgroup.procs
-r--r--r--   1 root root   0 Jun 25 20:04 cgroup.sane_behavior
-rw-r--r--   1 root root   0 Jun 25 20:04 cpu.cfs_period_us
-rw-r--r--   1 root root   0 Jun 25 16:33 cpu.cfs_quota_us
-rw-r--r--   1 root root   0 Jun 25 20:04 cpu.rt_period_us
-rw-r--r--   1 root root   0 Jun 25 20:04 cpu.rt_runtime_us
-rw-r--r--   1 root root   0 Jun 25 16:33 cpu.shares
-r--r--r--   1 root root   0 Jun 25 16:28 cpu.stat
-r--r--r--   1 root root   0 Jun 25 16:28 cpuacct.stat
-rw-r--r--   1 root root   0 Jun 25 16:28 cpuacct.usage
-r--r--r--   1 root root   0 Jun 25 16:28 cpuacct.usage_percpu
-rw-r--r--   1 root root   0 Jun 25 20:04 notify_on_release
-rw-r--r--   1 root root   0 Jun 25 20:04 release_agent
drwxr-xr-x 148 root root   0 Jun 22 22:58 system.slice
-rw-r--r--   1 root root   0 Jun 25 20:04 tasks
drwxr-xr-x   2 root root   0 Jun 22 22:58 user.slice

$ ls -al /sys/fs/cgroup/cpuacct/system.slice/
total 0
drwxr-xr-x   2 root root 0 Jun 22 22:58 -.mount
drwxr-xr-x 148 root root 0 Jun 22 22:58 .
dr-xr-xr-x   4 root root 0 Jun 22 22:58 ..
-rw-r--r--   1 root root 0 Jun 25 20:07 cgroup.clone_children
-rw-r--r--   1 root root 0 Jun 25 19:47 cgroup.procs
drwxr-xr-x   2 root root 0 Jun 22 22:58 coreos-setup-environment.service
-rw-r--r--   1 root root 0 Jun 25 20:07 cpu.cfs_period_us
-rw-r--r--   1 root root 0 Jun 25 20:07 cpu.cfs_quota_us
-rw-r--r--   1 root root 0 Jun 25 20:07 cpu.rt_period_us
-rw-r--r--   1 root root 0 Jun 25 20:07 cpu.rt_runtime_us
-rw-r--r--   1 root root 0 Jun 25 19:47 cpu.shares
-r--r--r--   1 root root 0 Jun 25 20:07 cpu.stat
-r--r--r--   1 root root 0 Jun 25 20:07 cpuacct.stat
-rw-r--r--   1 root root 0 Jun 25 20:07 cpuacct.usage
-r--r--r--   1 root root 0 Jun 25 20:07 cpuacct.usage_percpu
drwxr-xr-x   2 root root 0 Jun 22 22:58 dbus.service
drwxr-xr-x   2 root root 0 Jun 22 22:58 dev-hugepages.mount
drwxr-xr-x   2 root root 0 Jun 22 22:58 dev-mqueue.mount
drwxr-xr-x   2 root root 0 Jun 24 16:40 docker-00e8b2292809150c645c3d53bb1366cada02ed34b037f2445cb0074adfef96f6.scope
drwxr-xr-x   2 root root 0 Jun 23 13:53 docker-04d542edb3eaf4c401610824a9693fa1ef8dc295395dd4d82daa7a49f238e493.scope
drwxr-xr-x   2 root root 0 Jun 24 07:16 docker-086e65dc7cb12951b52962cd0bf1893f31c54252c2350cb44ef6e5e4902a1fdc.scope
drwxr-xr-x   2 root root 0 Jun 24 11:21 docker-0d47096538e11e20f0bdb38f8aeb3d5b2136ca86b62162ea5a7962512789bd25.scope

...

It would be great if we can support CoreOS in the future.

[remh]

@LotharSee Can you have a look ?

[LotharSee]

@andyshinn good job, it seems to be a pretty solution to make the Docker integration work from inside a container. I'll see what we can do then I'll update this issue.

[LotharSee]

Okay, the problem is that we refer to /proc/mounts to get the list of available cgroups. So even if they are in /sys/fs/cgroup, we won't go get them if they don't appear as mounted.
There is plenty of ways to fix it (hardcoding some paths, making overridable in the config, trying to guess without looking at mounts, ...) but none of them are great.

[remh]

We probably can guess them based on Docker version and the container ids
right? It should be a fall back enough if we can't get them the regular
way.
On Jul 4, 2014 12:15 AM, "Benjamin Fernandes" [email protected]
wrote:

Okay, the problem is that we refer to /proc/mounts to get the list of
available cgroups. So even if they are in /sys/fs/cgroup, we won't go get
them if they don't appear as mounted.
There is plenty of ways to fix it (hardcoding some paths, making
overridable in the config, trying to guess without looking at mounts, ...)
but none of them are great.

—
Reply to this email directly or view it on GitHub
#1001 (comment).

[remh]

@LotharSee what's the status of that ?
could you fix ?

[LotharSee]

There is two things:

• need access to /proc/mounts inside the container. We can mount on read-only mode the one from the host and replace the one on the guest with no specific damage.
• the cgroup hierarchy is totally different on CoreOS. Which means that many things need to be rewritten on the Docker check. To be precise, if you want to look at memory metrics, instead of /sys/fs/cgroup/memory/docker/[container-id]/ (on Debian), the path to look at is /sys/fs/cgroup/memory/system.slice/docker-[container-id].scope/.

We could quick fix it. But I think we'd better seize this opportunity to work on the CoreOS support.

[LotharSee]

Done in this PR:
#1043

The docker-dd-agent image and documentation will be updated once the Agent 5.0 is released.