cAdvisor container names not captured using cadvisor->influxdb driver

[marcellodesales]

Problem

The container names captured by cAdvisor are the container IDs instead of the names...

This is directly reflected in the InfluxDB values of the container_name as shown below:

> SHOW TAG VALUES FROM "cpu_usage_total" WITH KEY = "container_name"
name: container_nameTagValues
-----------------------------
container_name
/
/docker
/docker/01d931a43dcdca1ba71f26ef3eadf946aaec2c6c791785101bb4260b5c21ad40
/docker/02cc515f2cb6234d80ab4f8615936241ab71f4e6a4e58810282a543e1044387f
/docker/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112

Details

The docker-engine lists the proper names that are intended to be submitted...

[root@pprdnpmas300 npmo-server]# docker ps
CONTAINER ID        IMAGE                                            COMMAND                  CREATED             STATUS              PORTS                            NAMES
3f3a458b1863        npmoserver_newww                                 "/etc/npme/start.sh"     2 days ago          Up 2 days           5005/tcp, 0.0.0.0:80->8081/tcp   npmoserver_newww_1
78a032e4a4b4        bcoe/policy-follower:1.0.18                      "/etc/npme/start.sh"     2 days ago          Up 2 days                                            npmoserver_policyfollower_1
e206626d1b51        npmoserver_rrfollower                            "/etc/npme/start.sh"     2 days ago          Up 2 days                                            npmoserver_rrfollower_1
4f903da5431d        bcoe/es-follower:1.0.3                           "/etc/npme/start.sh"     2 days ago          Up 2 days                                            npmoserver_esfollower_1

The expected names are npmoserver_newww_1, npmoserver_policyfollower_1 ….

Question

• What value does cAdvisor use to send to influxdb?
• Do I need to specify any Docker Label?
• What's the suggested way of naming containers, given that those containers may be running on multiple machines? Use the Hostname? IP Address? The problem is that we need to distinguish them from different machines...

cAdvisor Version

Using the v0.20.5 version.

[root@pprdnpmas300 npmo-server]# docker ps | grep cadvisor
7c7e52082f6c        google/cadvisor:v0.20.5                          "/usr/bin/cadvisor -l"   2 days ago          Up 2 days           0.0.0.0:6090->8080/tcp           services_cadvisor_1

[vishh]

cAdvisor will expose the Name of the docker container to InfluxDB. Is
Cadvisor having issues communicating with docker in your setup? Does
/docker endpoint on cAdvisor work for you?

On Thu, Feb 25, 2016 at 10:42 AM, Marcello de Sales <
[email protected]> wrote:

Problem

The container names captured by cAdvisor are the container IDs instead of
the names...

[image: screen shot 2016-02-25 at 10 31 37 am]
https://cloud.githubusercontent.com/assets/131457/13329967/991a6b54-dbab-11e5-904b-330819d8a306.png

This is directly reflected in the InfluxDB values of the container_name
as shown below:

SHOW TAG VALUES FROM "cpu_usage_total" WITH KEY = "container_name"

name: container_nameTagValues

container_name
/
/docker
/docker/01d931a43dcdca1ba71f26ef3eadf946aaec2c6c791785101bb4260b5c21ad40
/docker/02cc515f2cb6234d80ab4f8615936241ab71f4e6a4e58810282a543e1044387f
/docker/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112

Details

The docker-engine lists the proper names that are intended to be
submitted...

[root@pprdnpmas300 npmo-server]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3f3a458b1863 npmoserver_newww "/etc/npme/start.sh" 2 days ago Up 2 days 5005/tcp, 0.0.0.0:80->8081/tcp npmoserver_newww_1
78a032e4a4b4 bcoe/policy-follower:1.0.18 "/etc/npme/start.sh" 2 days ago Up 2 days npmoserver_policyfollower_1
e206626d1b51 npmoserver_rrfollower "/etc/npme/start.sh" 2 days ago Up 2 days npmoserver_rrfollower_1
4f903da5431d bcoe/es-follower:1.0.3 "/etc/npme/start.sh" 2 days ago Up 2 days npmoserver_esfollower_1

The expected names are npmoserver_newww_1, npmoserver_policyfollower_1 ….
Question

• What value does cAdvisor use to send to influxdb?
• Do I need to specify any Docker Label?
• What's the suggested way of naming containers, given that those
  containers may be running on multiple machines? Use the Hostname? IP
  Address? The problem is that we need to distinguish them from different
  machines...

cAdvisor Version

Using the v0.20.5 version.

[root@pprdnpmas300 npmo-server]# docker ps | grep cadvisor
7c7e52082f6c google/cadvisor:v0.20.5 "/usr/bin/cadvisor -l" 2 days ago Up 2 days 0.0.0.0:6090->8080/tcp services_cadvisor_1

—
Reply to this email directly or view it on GitHub
#1127.

[marcellodesales]

@vishh The screenshot above is from the /docker endpoint... That shows the same docker ID... The query above is in the production remote influxDB host, so no communication problem so far...

[vishh]

Can you post the output of http://<cadvisor_ip:port>/api/v2.1/stats/docker/ 01d931a43dcdca1ba71f26ef3eadf946aaec2c6c791785101bb4260b5c21ad40 or some
other docker container that is still running?

On Thu, Feb 25, 2016 at 1:52 PM, Marcello de Sales <[email protected]

wrote:

@vishh https://github.com/vishh The screenshot above is from the /docker
endpoint... That shows the same docker ID... The query above is in the
production remote influxDB host, so no communication problem so far...

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh sure! Let me collect the values and report...

[marcellodesales]

Hi @vishh

Here's the output of pprdnpmas300.corp.ccc.net:6090/api/v2.1/stats/docker/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112

https://gist.github.com/marcellodesales/ca8922b8fa6cf3d42f7c

[vishh]

I still suspect that cadvisor is not able to talk to docker and so is
treating your containers as cgroups-only containers.

On Thu, Feb 25, 2016 at 2:32 PM, Marcello de Sales <[email protected]

wrote:

Hi @vishh https://github.com/vishh

Here's the output of
http://pprdnpmas300.corp.ccc.net:6090/api/v2.1/stats/docker/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112

https://gist.github.com/marcellodesales/ca8922b8fa6cf3d42f7c

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh even running on the same host, what does cadvisor needs to talk to the daemon? I'm starting cadvisor from Docker Compose... Is that a possible problem?

cadvisor:
  image: google/cadvisor:v0.20.5
  command: -storage_driver=influxdb 
    -storage_driver_db=cadvisor -storage_driver_host=xyz.abc.194.188:8086 
    -storage_driver_user=cadvisor -storage_driver_password=xxxxxxx 
    -storage_driver_secure=False
  restart: always
  ports:
    - "6090:8080"
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

[vishh]

Can you post cAdvisor logs? If its having trouble connecting to docker it
would have logged it.

On Thu, Feb 25, 2016 at 2:50 PM, Marcello de Sales <[email protected]

wrote:

@vishh https://github.com/vishh even running on the same host, what
does cadvisor needs to talk to the daemon? I'm starting cadvisor from
Docker Compose... Is that a possible problem?

cadvisor:
image: google/cadvisor:v0.20.5
command: -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=xyz.abc.194.188:8086 -storage_driver_user=cadvisor -storage_driver_password=xxxxxxx -storage_driver_secure=False
restart: always
ports:
- "6090:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh let me collect those...

[root@pprdnpmas300 npmo-server]# DATA_CENTER=qydc NPMO_ROLE=primary ENV=prd LOG_FORMAT=docker docker-compose -f services/ccc-ops.yml logs cadvisor
Attaching to services_cadvisor_1
cadvisor_1 | I0223 10:20:31.083785       1 storagedriver.go:42] Using backend storage type "influxdb"
cadvisor_1 | I0223 10:20:31.084018       1 storagedriver.go:44] Caching stats in memory for 2m0s
cadvisor_1 | I0223 10:20:31.084147       1 manager.go:131] cAdvisor running in container: "/docker/7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672"
cadvisor_1 | I0223 10:20:31.330678       1 fs.go:107] Filesystem partitions: map[/dev/mapper/docker-253:1-134296580-53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62 major:253 minor:7 fsType: blockSize:0} /dev/mapper/vg_root-lv_root:{mountpoint:/rootfs major:253 minor:1 fsType: blockSize:0} /dev/mapper/vg_appdata-lvm_appdata:{mountpoint:/rootfs/app major:253 minor:0 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31 major:253 minor:5 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153 major:253 minor:6 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c major:253 minor:9 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52 major:253 minor:10 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113 major:253 minor:12 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672:{mountpoint:/ major:253 minor:17 fsType: blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f major:253 minor:4 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978 major:253 minor:3 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b major:253 minor:8 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb6103b90fb3e2aea0e62bff73:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb6103b90fb3e2aea0e62bff73 major:253 minor:13 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112 major:253 minor:16 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67 major:253 minor:11 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025 major:253 minor:14 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-1e4476a9a20d2d27a9afed0d7dd924285531488af1baaa5226c5747b71f6b342:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/1e4476a9a20d2d27a9afed0d7dd924285531488af1baaa5226c5747b71f6b342 major:253 minor:15 fsType: blockSize:0}]
cadvisor_1 | I0223 10:20:31.352636       1 manager.go:166] Machine: {NumCores:4 CpuFrequency:2128000 MemoryCapacity:16651481088 MachineID:0855d19bf2cf40d7a70cb45145950327 SystemUUID:42066107-97D5-B986-DDC9-8E3CE26464A0 BootID:42e1e5fe-d80e-4a42-8a20-5018316c8343 Filesystems:[{Device:/dev/mapper/docker-253:1-134296580-03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb6103b90fb3e2aea0e62bff73 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-1e4476a9a20d2d27a9afed0d7dd924285531488af1baaa5226c5747b71f6b342 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112 Capacity:107320705024} {Device:/dev/mapper/vg_root-lv_root Capacity:40488808448} {Device:/dev/mapper/vg_appdata-lvm_appdata Capacity:31568424960} {Device:/dev/mapper/docker-253:1-134296580-b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672 Capacity:107320705024} {Device:/dev/sda1 Capacity:264941568} {Device:/dev/mapper/docker-253:1-134296580-e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153 Capacity:107320705024}] DiskMap:map[253:13:{Name:dm-13 Major:253 Minor:13 Size:107374182400 Scheduler:none} 253:14:{Name:dm-14 Major:253 Minor:14 Size:107374182400 Scheduler:none} 253:15:{Name:dm-15 Major:253 Minor:15 Size:107374182400 Scheduler:none} 253:8:{Name:dm-8 Major:253 Minor:8 Size:107374182400 Scheduler:none} 253:9:{Name:dm-9 Major:253 Minor:9 Size:107374182400 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:42949672960 Scheduler:deadline} 253:10:{Name:dm-10 Major:253 Minor:10 Size:107374182400 Scheduler:none} 253:17:{Name:dm-17 Major:253 Minor:17 Size:107374182400 Scheduler:none} 253:6:{Name:dm-6 Major:253 Minor:6 Size:107374182400 Scheduler:none} 253:7:{Name:dm-7 Major:253 Minor:7 Size:107374182400 Scheduler:none} 253:16:{Name:dm-16 Major:253 Minor:16 Size:107374182400 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:4:{Name:dm-4 Major:253 Minor:4 Size:107374182400 Scheduler:none} 253:5:{Name:dm-5 Major:253 Minor:5 Size:107374182400 Scheduler:none} 8:16:{Name:sdb Major:8 Minor:16 Size:32212254720 Scheduler:deadline} 253:12:{Name:dm-12 Major:253 Minor:12 Size:107374182400 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:40508588032 Scheduler:none} 253:11:{Name:dm-11 Major:253 Minor:11 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:32208060416 Scheduler:none}] NetworkDevices:[{Name:bridge0 MacAddress:12:32:bd:42:78:86 Speed:0 Mtu:1500} {Name:eth0 MacAddress:00:50:56:86:01:9b Speed:10000 Mtu:1500} {Name:eth1 MacAddress:00:50:56:86:25:f5 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:17179336704 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]} {Id:2 Threads:[2] Caches:[]} {Id:3 Threads:[3] Caches:[]}] Caches:[{Size:25165824 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
cadvisor_1 | I0223 10:20:31.354104       1 manager.go:172] Version: {KernelVersion:3.10.0-327.10.1.el7.x86_64 ContainerOsVersion:Alpine Linux v3.2 DockerVersion:1.9.1 CadvisorVersion:0.20.5 CadvisorRevision:9aa348f}
cadvisor_1 | I0223 10:20:31.355327       1 factory.go:210] System is using systemd
cadvisor_1 | I0223 10:20:31.633518       1 factory.go:252] Registering Docker factory
cadvisor_1 | I0223 10:20:31.644199       1 factory.go:94] Registering Raw factory
cadvisor_1 | I0223 10:20:32.006032       1 manager.go:1000] Started watching for new ooms in manager
cadvisor_1 | W0223 10:20:32.006974       1 manager.go:239] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
cadvisor_1 | I0223 10:20:32.007723       1 manager.go:252] Starting recovery of all containers
cadvisor_1 | I0223 10:20:32.031139       1 manager.go:257] Recovery completed
cadvisor_1 | I0223 10:20:32.060334       1 cadvisor.go:106] Starting cAdvisor version: 0.20.5-9aa348f on port 8080

From here, I see the first error... cadvisor_1 | E0224 23:10:26.798628 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused

cadvisor_1 | W0223 10:33:32.173015       1 container.go:354] Failed to get RecentStats("/docker/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62") while determining the next housekeeping: unable to find data for container /docker/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62
cadvisor_1 | W0223 10:37:12.719889       1 container.go:341] Failed to create summary reader for "/docker/69145724a3bc855385119e2552c70064bb1c42be7857ad6413dff4cfcb41793b": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:15.991376       1 container.go:341] Failed to create summary reader for "/docker/64cba7065673a8025785ead336e5c405dc97ec1a2ca6d66fe5b758f59bcc765b": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:17.376885       1 container.go:341] Failed to create summary reader for "/docker/978117c3fbf2e814dd1907f8b109cf1b1b683325b4f250799fa462eec9ed820f": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:29.095995       1 container.go:341] Failed to create summary reader for "/docker/78a032e4a4b4332238427ea20b4cdb44ee8ed44ac483131e5cba18b746513c0c": none of the resources are being tracked.
cadvisor_1 | E0224 09:39:55.290111       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: i/o timeout
cadvisor_1 | E0224 09:40:55.385761       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: i/o timeout
cadvisor_1 | E0224 09:41:25.464999       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 09:42:24.102757       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: read tcp 192.168.4.15:45426->10.180.194.188:8086: read: connection reset by peer
cadvisor_1 | E0224 21:50:30.219446       1 memory.go:91] failed to write stats to influxDb - {"error":"timeout"}
cadvisor_1 | E0224 23:07:26.649126       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:08:26.655138       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:09:26.696369       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:10:26.798628       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused

Then, I think the server went up again, and lost the database... {"error":"database not found: \"cadvisor\""}

cadvisor_1 | E0225 01:42:39.394727       1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0225 01:43:40.328899       1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: \"cadvisor\""}
cadvisor_1 | E0225 01:44:39.815301       1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: \"cadvisor\""}
cadvisor_1 | E0225 01:45:39.809165       1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: \"cadvisor\""}

Question

Could this be the source of the problems? We used to see the full container names before... And now only the IDs...

[vishh]

No. Can you try running with --nosystemd flag? We fixed a systemd
integration issue on HEAD. I suspect that might be the cause of the issue.

On Thu, Feb 25, 2016 at 3:16 PM, Marcello de Sales <[email protected]

wrote:

@vishh https://github.com/vishh let me collect those...

[root@pprdnpmas300 npmo-server]# DATA_CENTER=qydc NPMO_ROLE=primary ENV=prd LOG_FORMAT=docker docker-compose -f services/ccc-ops.yml logs cadvisor
Attaching to services_cadvisor_1
cadvisor_1 | I0223 10:20:31.083785 1 storagedriver.go:42] Using backend storage type "influxdb"
cadvisor_1 | I0223 10:20:31.084018 1 storagedriver.go:44] Caching stats in memory for 2m0s
cadvisor_1 | I0223 10:20:31.084147 1 manager.go:131] cAdvisor running in container: "/docker/7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672"
cadvisor_1 | I0223 10:20:31.330678 1 fs.go:107] Filesystem partitions: map[/dev/mapper/docker-253:1-134296580-53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62 major:253 minor:7 fsType: blockSize:0} /dev/mapper/vg_root-lv_root:{mountpoint:/rootfs major:253 minor:1 fsType: blockSize:0} /dev/mapper/vg_appdata-lvm_appdata:{mountpoint:/rootfs/app major:253 minor:0 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31 major:253 minor:5 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153 maj
or:253 minor:6 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c major:253 minor:9 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52 major:253 minor:10 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113 major:253 minor:12 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672:{mountpoint:/ major:253 minor:17 fsType: blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot ma
jor:8 minor:1 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f major:253 minor:4 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978 major:253 minor:3 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b major:253 minor:8 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb6103b90fb3e2aea0e62bff73:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb61
03b90fb3e2aea0e62bff73 major:253 minor:13 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112 major:253 minor:16 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67 major:253 minor:11 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025 major:253 minor:14 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-1e4476a9a20d2d27a9afed0d7dd924285531488af1baaa5226c5747b71f6b342:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/1e4476a9a20
d2d27a9afed0d7dd924285531488af1baaa5226c5747b71f6b342 major:253 minor:15 fsType: blockSize:0}]
cadvisor_1 | I0223 10:20:31.352636 1 manager.go:166] Machine: {NumCores:4 CpuFrequency:2128000 MemoryCapacity:16651481088 MachineID:0855d19bf2cf40d7a70cb45145950327 SystemUUID:42066107-97D5-B986-DDC9-8E3CE26464A0 BootID:42e1e5fe-d80e-4a42-8a20-5018316c8343 Filesystems:[{Device:/dev/mapper/docker-253:1-134296580-03800a7b7a98d17a31fc6fe041fe0871edfa89086e606f4d02baf6338cbc6978 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-72d562779765fd40e4ea4f483de33a52cb08809480a300091d3ba4eb9390ea6b Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-7efd1ec3fcb5f528204acf8c46a4535fd4dc77fb6103b90fb3e2aea0e62bff73 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-0b140c1ae4d2aa4f40b2cfe5aa9e74277945bc1abfd090321bf79de0c8a47e67 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-27564bd828e2ac540ce14cb8d4fbf63418afe60a76fc847be0fcc9e5a7a40025 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-1e4476a9a20d2d27a9afed0d
7dd924285531488af1baaa5226c5747b71f6b342 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-0e1e84b6291f9e6b8b328b43eab7901bb2b82fff78bbf41248299bfc1a9fc112 Capacity:107320705024} {Device:/dev/mapper/vg_root-lv_root Capacity:40488808448} {Device:/dev/mapper/vg_appdata-lvm_appdata Capacity:31568424960} {Device:/dev/mapper/docker-253:1-134296580-b54edf37a7493b3aa7ad081c44f3153c1137c0dc75935061f65784a2de4a5c31 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-2bff51cd0def977b86089b215b5ea60f42fb2a9f1b16f811621218fceee6690c Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-dd34e8042e298836e9d1557f74428f69a1d20b6b1dc9560c2d799cfad0ce9c52 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-76a8bf6d879db577f40cba5537e4af0dd6e9d262cdc8062ca9735e53da764113 Capacity:107320705024} {Device:/dev/mapper/dock
er-253:1-134296580-7c7e52082f6c4f0d8b96f3a55f0f423a291df0f70a53725dd8802af22904c672 Capacity:107320705024} {Device:/dev/sda1 Capacity:264941568} {Device:/dev/mapper/docker-253:1-134296580-e15e615b909cea26ed32345363d86a13adde5599bea79f60f282035f9aa73e5f Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-7761918b90b9d879aaa4dbbaa8fcc237b4a8c398ce8005102f6fbeaaa9364153 Capacity:107320705024}] DiskMap:map[253:13:{Name:dm-13 Major:253 Minor:13 Size:107374182400 Scheduler:none} 253:14:{Name:dm-14 Major:253 Minor:14 Size:107374182400 Scheduler:none} 253:15:{Name:dm-15 Major:253 Minor:15 Size:107374182400 Scheduler:none} 253:8:{Name:dm-8 Major:253 Minor:8 Size:107374182400 Scheduler:none} 253:9:{Name:dm-9 Major:253 Minor:9 Size:107374182400 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:42949672960 Scheduler:deadline} 253:10:{Name:dm-10 Major:253 Minor:10 Size:107374182400 Scheduler:none} 253:17:{Name:dm-17 Major:253 Minor:17 Size:107374182400 Scheduler:none} 253:6:{Name:
dm-6 Major:253 Minor:6 Size:107374182400 Scheduler:none} 253:7:{Name:dm-7 Major:253 Minor:7 Size:107374182400 Scheduler:none} 253:16:{Name:dm-16 Major:253 Minor:16 Size:107374182400 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:4:{Name:dm-4 Major:253 Minor:4 Size:107374182400 Scheduler:none} 253:5:{Name:dm-5 Major:253 Minor:5 Size:107374182400 Scheduler:none} 8:16:{Name:sdb Major:8 Minor:16 Size:32212254720 Scheduler:deadline} 253:12:{Name:dm-12 Major:253 Minor:12 Size:107374182400 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:40508588032 Scheduler:none} 253:11:{Name:dm-11 Major:253 Minor:11 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:32208060416 Scheduler:none}] NetworkDevices:[{Name:bridge0 MacAddress:12:32:bd:42:78:86 Speed:0 Mtu:1500} {Name:eth0 MacAddress:00:50:56:86:01
:9b Speed:10000 Mtu:1500} {Name:eth1 MacAddress:00:50:56:86:25:f5 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:17179336704 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]} {Id:2 Threads:[2] Caches:[]} {Id:3 Threads:[3] Caches:[]}] Caches:[{Size:25165824 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
cadvisor_1 | I0223 10:20:31.354104 1 manager.go:172] Version: {KernelVersion:3.10.0-327.10.1.el7.x86_64 ContainerOsVersion:Alpine Linux v3.2 DockerVersion:1.9.1 CadvisorVersion:0.20.5 CadvisorRevision:9aa348f}
cadvisor_1 | I0223 10:20:31.355327 1 factory.go:210] System is using systemd
cadvisor_1 | I0223 10:20:31.633518 1 factory.go:252] Registering Docker factory
cadvisor_1 | I0223 10:20:31.644199 1 factory.go:94] Registering Raw factory
cadvisor_1 | I0223 10:20:32.006032 1 manager.go:1000] Started watching for new ooms in manager
cadvisor_1 | W0223 10:20:32.006974 1 manager.go:239] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
cadvisor_1 | I0223 10:20:32.007723 1 manager.go:252] Starting recovery of all containers
cadvisor_1 | I0223 10:20:32.031139 1 manager.go:257] Recovery completed
cadvisor_1 | I0223 10:20:32.060334 1 cadvisor.go:106] Starting cAdvisor version: 0.20.5-9aa348f on port 8080

From here, I see the first error... cadvisor_1 | E0224 23:10:26.798628 1
memory.go:91] failed to write stats to influxDb - Post
http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=:
dial tcp
10.180.194.188:8086: getsockopt: connection refused

cadvisor_1 | W0223 10:33:32.173015 1 container.go:354] Failed to get RecentStats("/docker/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62") while determining the next housekeeping: unable to find data for container /docker/53464f81ac6740416c0e9ab0d66d551a102823e1dd3761308522004419ca3e62
cadvisor_1 | W0223 10:37:12.719889 1 container.go:341] Failed to create summary reader for "/docker/69145724a3bc855385119e2552c70064bb1c42be7857ad6413dff4cfcb41793b": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:15.991376 1 container.go:341] Failed to create summary reader for "/docker/64cba7065673a8025785ead336e5c405dc97ec1a2ca6d66fe5b758f59bcc765b": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:17.376885 1 container.go:341] Failed to create summary reader for "/docker/978117c3fbf2e814dd1907f8b109cf1b1b683325b4f250799fa462eec9ed820f": none of the resources are being tracked.
cadvisor_1 | W0223 10:37:29.095995 1 container.go:341] Failed to create summary reader for "/docker/78a032e4a4b4332238427ea20b4cdb44ee8ed44ac483131e5cba18b746513c0c": none of the resources are being tracked.
cadvisor_1 | E0224 09:39:55.290111 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: i/o timeout
cadvisor_1 | E0224 09:40:55.385761 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: i/o timeout
cadvisor_1 | E0224 09:41:25.464999 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 09:42:24.102757 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: read tcp 192.168.4.15:45426->10.180.194.188:8086: read: connection reset by peer
cadvisor_1 | E0224 21:50:30.219446 1 memory.go:91] failed to write stats to influxDb - {"error":"timeout"}
cadvisor_1 | E0224 23:07:26.649126 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:08:26.655138 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:09:26.696369 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0224 23:10:26.798628 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused

Then, I think the server went up again, and lost the database... {"error":"database
not found: "cadvisor""}

cadvisor_1 | E0225 01:42:39.394727 1 memory.go:91] failed to write stats to influxDb - Post http://10.180.194.188:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 10.180.194.188:8086: getsockopt: connection refused
cadvisor_1 | E0225 01:43:40.328899 1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: "cadvisor""}
cadvisor_1 | E0225 01:44:39.815301 1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: "cadvisor""}
cadvisor_1 | E0225 01:45:39.809165 1 memory.go:91] failed to write stats to influxDb - {"error":"database not found: "cadvisor""}

Question

Could this be the source of the problems? We used to see the full
container names before... And now only the IDs...

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh Let me try...

[marcellodesales]

@vishh I just restarted it...

• I don't see any information about connecting to influxDB in the beginning... Shouldn't we add such a log entry during the bootstrap?
• I don't see the systemd entry in the log anymore...
• How do I verify

[root@pprdnpmas300 npmo-server]# DATA_CENTER=qydc NPMO_ROLE=primary ENV=prd LOG_FORMAT=docker docker-compose -f services/ccc-ops.yml logs cadvisor
Attaching to services_cadvisor_1
cadvisor_1 | I0225 23:37:37.344136       1 storagedriver.go:42] Using backend storage type "influxdb"
cadvisor_1 | I0225 23:37:37.344471       1 storagedriver.go:44] Caching stats in memory for 2m0s
cadvisor_1 | I0225 23:37:37.344614       1 manager.go:131] cAdvisor running in container: "/docker/99c215438f9a5c4e1f1e7575ecbf851d94770d61ed2a00123355feda6c2cd91a"
cadvisor_1 | I0225 23:37:37.669296       1 fs.go:107] Filesystem partitions: map[/dev/mapper/docker-253:1-134296580-25aa982db0a4752ad913be7340abc85dfa8349d91618cd42f09f7a5751dfbc46:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/25aa982db0a4752ad913be7340abc85dfa8349d91618cd42f09f7a5751dfbc46 major:253 minor:23 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-978117c3fbf2e814dd1907f8b109cf1b1b683325b4f250799fa462eec9ed820f:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/978117c3fbf2e814dd1907f8b109cf1b1b683325b4f250799fa462eec9ed820f major:253 minor:25 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-2fc22b33b5c67a1db9a814c5fed1aa3e8799968763bdf905e1df8b29cfac7b59:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/2fc22b33b5c67a1db9a814c5fed1aa3e8799968763bdf905e1df8b29cfac7b59 major:253 minor:28 fsType: blockSize:0} /dev/mapper/vg_root-lv_root:{mountpoint:/rootfs major:253 minor:1 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-69145724a3bc855385119e2552c70064bb1c42be7857ad6413dff4cfcb41793b:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/69145724a3bc855385119e2552c70064bb1c42be7857ad6413dff4cfcb41793b major:253 minor:20 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-e206626d1b51575662fc2c014c15fe31e6d10c0f47897a732398ca5562cd7250:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/e206626d1b51575662fc2c014c15fe31e6d10c0f47897a732398ca5562cd7250 major:253 minor:30 fsType: blockSize:0} /dev/mapper/vg_appdata-lvm_appdata:{mountpoint:/rootfs/app major:253 minor:0 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-9db0ea44969ebd620684e4356cec163e6e077474191ad30ed3f78c8234a9b847:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/9db0ea44969ebd620684e4356cec163e6e077474191ad30ed3f78c8234a9b847 major:253 minor:19 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-6d56b1ff2c8e121aefd0409b1b4425554239cea15555b43fcb2eb09637b06327:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/6d56b1ff2c8e121aefd0409b1b4425554239cea15555b43fcb2eb09637b06327 major:253 minor:27 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-b5fc4d10a46d8b4d804ae1c436a33e0596afa4924342f95c6d8f5ab4b9b44101:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/b5fc4d10a46d8b4d804ae1c436a33e0596afa4924342f95c6d8f5ab4b9b44101 major:253 minor:22 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-4f903da5431d21781d920fbd4fd4171caf1406f1bc339961245eefb09cd17053:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/4f903da5431d21781d920fbd4fd4171caf1406f1bc339961245eefb09cd17053 major:253 minor:29 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-3f3a458b1863d150968f00fbd33b301724b37918a83392747c7828d5a10ac422:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/3f3a458b1863d150968f00fbd33b301724b37918a83392747c7828d5a10ac422 major:253 minor:31 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-78a032e4a4b4332238427ea20b4cdb44ee8ed44ac483131e5cba18b746513c0c:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/78a032e4a4b4332238427ea20b4cdb44ee8ed44ac483131e5cba18b746513c0c major:253 minor:32 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-99c215438f9a5c4e1f1e7575ecbf851d94770d61ed2a00123355feda6c2cd91a:{mountpoint:/ major:253 minor:34 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-64cba7065673a8025785ead336e5c405dc97ec1a2ca6d66fe5b758f59bcc765b:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/64cba7065673a8025785ead336e5c405dc97ec1a2ca6d66fe5b758f59bcc765b major:253 minor:24 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-f2b8d149a4017325daa7c6f4db48703bcafc2d3e10dd5240465907c797b97628:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/f2b8d149a4017325daa7c6f4db48703bcafc2d3e10dd5240465907c797b97628 major:253 minor:26 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-93e89198343ea01fa98a53a7131d7d4f8c43e75da1fd80023c85e764560d5c37:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/93e89198343ea01fa98a53a7131d7d4f8c43e75da1fd80023c85e764560d5c37 major:253 minor:33 fsType: blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType: blockSize:0} /dev/mapper/docker-253:1-134296580-486efb1a8a5a2740258fc3641a9c580d3e3c04acba53f6546138cb196aeb102f:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/486efb1a8a5a2740258fc3641a9c580d3e3c04acba53f6546138cb196aeb102f major:253 minor:21 fsType: blockSize:0}]
cadvisor_1 | I0225 23:37:37.692723       1 manager.go:166] Machine: {NumCores:4 CpuFrequency:2128000 MemoryCapacity:16651481088 MachineID:0855d19bf2cf40d7a70cb45145950327 SystemUUID:42066107-97D5-B986-DDC9-8E3CE26464A0 BootID:42e1e5fe-d80e-4a42-8a20-5018316c8343 Filesystems:[{Device:/dev/mapper/docker-253:1-134296580-64cba7065673a8025785ead336e5c405dc97ec1a2ca6d66fe5b758f59bcc765b Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-6d56b1ff2c8e121aefd0409b1b4425554239cea15555b43fcb2eb09637b06327 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-b5fc4d10a46d8b4d804ae1c436a33e0596afa4924342f95c6d8f5ab4b9b44101 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-4f903da5431d21781d920fbd4fd4171caf1406f1bc339961245eefb09cd17053 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-3f3a458b1863d150968f00fbd33b301724b37918a83392747c7828d5a10ac422 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-78a032e4a4b4332238427ea20b4cdb44ee8ed44ac483131e5cba18b746513c0c Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-99c215438f9a5c4e1f1e7575ecbf851d94770d61ed2a00123355feda6c2cd91a Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-486efb1a8a5a2740258fc3641a9c580d3e3c04acba53f6546138cb196aeb102f Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-f2b8d149a4017325daa7c6f4db48703bcafc2d3e10dd5240465907c797b97628 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-93e89198343ea01fa98a53a7131d7d4f8c43e75da1fd80023c85e764560d5c37 Capacity:107320705024} {Device:/dev/sda1 Capacity:264941568} {Device:/dev/mapper/docker-253:1-134296580-69145724a3bc855385119e2552c70064bb1c42be7857ad6413dff4cfcb41793b Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-25aa982db0a4752ad913be7340abc85dfa8349d91618cd42f09f7a5751dfbc46 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-978117c3fbf2e814dd1907f8b109cf1b1b683325b4f250799fa462eec9ed820f Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-2fc22b33b5c67a1db9a814c5fed1aa3e8799968763bdf905e1df8b29cfac7b59 Capacity:107320705024} {Device:/dev/mapper/vg_root-lv_root Capacity:40488808448} {Device:/dev/mapper/docker-253:1-134296580-9db0ea44969ebd620684e4356cec163e6e077474191ad30ed3f78c8234a9b847 Capacity:107320705024} {Device:/dev/mapper/docker-253:1-134296580-e206626d1b51575662fc2c014c15fe31e6d10c0f47897a732398ca5562cd7250 Capacity:107320705024} {Device:/dev/mapper/vg_appdata-lvm_appdata Capacity:31568424960}] DiskMap:map[253:12:{Name:dm-12 Major:253 Minor:12 Size:107374182400 Scheduler:none} 253:16:{Name:dm-16 Major:253 Minor:16 Size:107374182400 Scheduler:none} 253:18:{Name:dm-18 Major:253 Minor:18 Size:107374182400 Scheduler:none} 253:21:{Name:dm-21 Major:253 Minor:21 Size:107374182400 Scheduler:none} 253:10:{Name:dm-10 Major:253 Minor:10 Size:107374182400 Scheduler:none} 253:17:{Name:dm-17 Major:253 Minor:17 Size:107374182400 Scheduler:none} 253:20:{Name:dm-20 Major:253 Minor:20 Size:107374182400 Scheduler:none} 253:28:{Name:dm-28 Major:253 Minor:28 Size:107374182400 Scheduler:none} 253:11:{Name:dm-11 Major:253 Minor:11 Size:107374182400 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:24:{Name:dm-24 Major:253 Minor:24 Size:107374182400 Scheduler:none} 253:5:{Name:dm-5 Major:253 Minor:5 Size:107374182400 Scheduler:none} 253:26:{Name:dm-26 Major:253 Minor:26 Size:107374182400 Scheduler:none} 253:27:{Name:dm-27 Major:253 Minor:27 Size:107374182400 Scheduler:none} 253:8:{Name:dm-8 Major:253 Minor:8 Size:107374182400 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:42949672960 Scheduler:deadline} 8:16:{Name:sdb Major:8 Minor:16 Size:32212254720 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:32208060416 Scheduler:none} 253:13:{Name:dm-13 Major:253 Minor:13 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:107374182400 Scheduler:none} 253:30:{Name:dm-30 Major:253 Minor:30 Size:107374182400 Scheduler:none} 253:14:{Name:dm-14 Major:253 Minor:14 Size:107374182400 Scheduler:none} 253:25:{Name:dm-25 Major:253 Minor:25 Size:107374182400 Scheduler:none} 253:32:{Name:dm-32 Major:253 Minor:32 Size:107374182400 Scheduler:none} 253:4:{Name:dm-4 Major:253 Minor:4 Size:107374182400 Scheduler:none} 253:6:{Name:dm-6 Major:253 Minor:6 Size:107374182400 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:40508588032 Scheduler:none} 253:19:{Name:dm-19 Major:253 Minor:19 Size:107374182400 Scheduler:none} 253:29:{Name:dm-29 Major:253 Minor:29 Size:107374182400 Scheduler:none} 253:31:{Name:dm-31 Major:253 Minor:31 Size:107374182400 Scheduler:none} 253:34:{Name:dm-34 Major:253 Minor:34 Size:107374182400 Scheduler:none} 253:9:{Name:dm-9 Major:253 Minor:9 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 253:15:{Name:dm-15 Major:253 Minor:15 Size:107374182400 Scheduler:none} 253:22:{Name:dm-22 Major:253 Minor:22 Size:107374182400 Scheduler:none} 253:23:{Name:dm-23 Major:253 Minor:23 Size:107374182400 Scheduler:none} 253:33:{Name:dm-33 Major:253 Minor:33 Size:107374182400 Scheduler:none} 253:7:{Name:dm-7 Major:253 Minor:7 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:bridge0 MacAddress:22:27:d2:62:84:d9 Speed:0 Mtu:1500} {Name:eth0 MacAddress:00:50:56:86:01:9b Speed:10000 Mtu:1500} {Name:eth1 MacAddress:00:50:56:86:25:f5 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:17179336704 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]} {Id:2 Threads:[2] Caches:[]} {Id:3 Threads:[3] Caches:[]}] Caches:[{Size:25165824 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
cadvisor_1 | I0225 23:37:37.694066       1 manager.go:172] Version: {KernelVersion:3.10.0-327.10.1.el7.x86_64 ContainerOsVersion:Alpine Linux v3.2 DockerVersion:1.9.1 CadvisorVersion:0.20.5 CadvisorRevision:9aa348f}
cadvisor_1 | I0225 23:37:37.967196       1 factory.go:252] Registering Docker factory
cadvisor_1 | I0225 23:37:37.978313       1 factory.go:94] Registering Raw factory
cadvisor_1 | I0225 23:37:38.643410       1 manager.go:1000] Started watching for new ooms in manager
cadvisor_1 | W0225 23:37:38.644505       1 manager.go:239] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
cadvisor_1 | I0225 23:37:38.678312       1 manager.go:252] Starting recovery of all containers
cadvisor_1 | I0225 23:37:38.778898       1 manager.go:257] Recovery completed
cadvisor_1 | I0225 23:37:38.812340       1 cadvisor.go:106] Starting cAdvisor version: 0.20.5-9aa348f on port 8080

The /docker/ cadvisor endpoint still not showing the name of the containers...

/docker
root docker
Subcontainers
/docker/25aa982db0a4752ad913be7340abc85dfa8349d91618cd42f09f7a5751dfbc46
/docker/2fc22b33b5c67a1db9a814c5fed1aa3e8799968763bdf905e1df8b29cfac7b59
/docker/3f3a458b1863d150968f00fbd33b301724b37918a83392747c7828d5a10ac422
/docker/486efb1a8a5a2740258fc3641a9c580d3e3c04acba53f6546138cb196aeb102f
/docker/4f903da5431d21781d920fbd4fd4171caf1406f1bc339961245eefb09cd17053

[marcellodesales]

Also, I started seeing the following:

Is there a to setup the rate and maybe decrease the throughput?

cadvisor_1 | E0225 23:43:38.044736       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:44:38.171171       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:45:38.365485       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:46:38.780019       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:47:38.909916       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:48:39.045459       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:49:39.002101       1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}

[vishh]

It looks an InfluxDB rate limiter. You can set the syncing period to be
higher. It is not clear from the logs why cAdvisor is not detecting docker
containers. One option is to set --vmodule=*=4 flag to get additional
information.
If everything works, you should see the container pseudo names in the UI
(http://<cadvisor_ip:port>/docker)

On Thu, Feb 25, 2016 at 3:50 PM, Marcello de Sales <[email protected]

wrote:

Also, I started seeing the following:

cadvisor_1 | E0225 23:43:38.044736 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:44:38.171171 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:45:38.365485 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:46:38.780019 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:47:38.909916 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:48:39.045459 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}
cadvisor_1 | E0225 23:49:39.002101 1 memory.go:91] failed to write stats to influxDb - {"error":"write failed: engine: write points: write throughput too high. backoff and retry"}

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh Hummm Yeah our influxdb team might have added something to it as they mentioned that our cadvisor from only 4 hosts was pushing events at the rate of 500/s.

Ok, let me add the flag and see if we can collect anything else...

[vishh]

FYI: You can reduce the write rate from cadvisor side

On Thu, Feb 25, 2016 at 4:30 PM, Marcello de Sales <[email protected]

wrote:

@vishh https://github.com/vishh Hummm Yeah our influxdb team might have
added something to it as they mentioned that our cadvisor from only 4 hosts
was pushing events at the rate of 500/s.

Ok, let me add the flag and see if we can collect anything else...

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh how? :) Any pointers?

[vishh]

--storage_driver_buffer_duration

On Thu, Feb 25, 2016 at 4:32 PM, Marcello de Sales <[email protected]

wrote:

@vishh https://github.com/vishh how? :)

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh awesome

[marcellodesales]

@vishh I added the flag --vmodule=*=4 and I'm getting a failure... failed to collect filesystem stats - du command failed on with output du: : No such file or directory.

The current command is as follows:

-storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=10.180.194.188:8086 -storage_driver_user=cadvisor -storage_driver_passwor d=yH70hxwgSI -storage_driver_secure=False --nosystemd --vmodule=*=4

Here's the log...

cadvisor_1 | I0226 00:34:16.858215       1 manager.go:257] Recovery completed
cadvisor_1 | I0226 00:34:16.858488       1 container.go:386] Start housekeeping for container "/docker/e206626d1b51575662fc2c014c15fe31e6d10c0f47897a732398ca5562cd7250"
cadvisor_1 | I0226 00:34:16.899940       1 cadvisor.go:106] Starting cAdvisor version: 0.20.5-9aa348f on port 8080
cadvisor_1 | I0226 00:35:11.174489       1 container.go:403] [/docker/4f903da5431d21781d920fbd4fd4171caf1406f1bc339961245eefb09cd17053] Housekeeping took 1.090983293s
cadvisor_1 | I0226 00:35:16.783319       1 fsHandler.go:96] failed to collect filesystem stats - du command failed on  with output du: : No such file or directory
cadvisor_1 |  - exit status 1
cadvisor_1 | I0226 00:35:16.784036       1 fsHandler.go:96] failed to collect filesystem stats - du command failed on  with output du: : No such file or directory
cadvisor_1 |  - exit status 1
cadvisor_1 | I0226 00:35:16.789255       1 fsHandler.go:96] failed to collect filesystem stats - du command failed on  with output du: : No such file or directory
cadvisor_1 |  - exit status 1
cadvisor_1 | I0226 00:35:16.793966       1 fsHandler.go:96] failed to collect filesystem stats - du command failed on  with output du: : No such file or directory
cadvisor_1 |  - exit status 1
cadvisor_1 | I0226 00:35:16.802140       1 fsHandler.go:96] failed to

[marcellodesales]

I will be adding the rate as described at #1074 (comment)...

@vishh Any points from the logs above?

[vishh]

Ignore the du logs it will be fixed in the next release. Can you post all
the logs?

On Fri, Feb 26, 2016, 12:02 PM Marcello de Sales [email protected]
wrote:

I will be adding the rate as described at #1074 (comment)
#1074 (comment)...

@vishh https://github.com/vishh Any points from the logs above?

—
Reply to this email directly or view it on GitHub
#1127 (comment).

[marcellodesales]

@vishh Sounds good... I will be busy for the next 1-2hs but I will post another Gist with it... Thanks a lot Vish!

[TeChn4K]

Same problem here. Any update ?

[studna]

Same here. I've upgraded Docker 1.9 -> 1.11 and now I can't see container names in influxdb. Any update? @vishh @marcellodesales @TeChn4K

[mrgleam]

Me too. i've upgraded to 1.11.1

I can't see container name in influxdb and i can't see network interface in cadvisor.

Please help me.

Thank you.

[vishh]

Can you try the latest cadvisor version - google/cadvisor:v0.23.0?

[mrgleam]

I used cadvisor tag :lastest
Not work for me :(

[vishh]

We added support for docker v1.11 in cadvisor v:0.23.0. We will soon
switch latest.

On Sun, May 1, 2016 at 1:50 PM, Santi Lertsumran [email protected]
wrote:

I used cadvisor tag :lastest
Not work for me :(

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1127 (comment)

[mrgleam]

Ok this work. Tahnk you very much.

[biswars]

Having the same issue in kubernetes setup. As cadvisor is embedded in kubelet binary, how can the cadvisor be upgraded in the kubernetes setup ?

[timstclair]

As cadvisor is embedded in kubelet binary, how can the cadvisor be upgraded in the kubernetes setup ?

The only way to upgrade the cAdvisor version used by the Kubelet is to build a new kubelet with the latest release (included at head, but not the latest pre-release). cAdvisor can be run in "standalone" mode (see https://github.com/timstclair/kube-contrib/blob/master/devel/manifests/cadvisor-pod.yaml), but you are likely to run into other issues running Kubernetes with docker v1.11 without running Kubelet binaries compiled in the last week.

[biswars]

Thanks @timstclair - very useful.

[dashpole]

Closing this, as it deals with a previous version, and appears to be resolved.