High CPU consumption in probes

[2opremio]

On the three worker machines of the service the CPU consumption of the 0.15 candidate is over the roof

Also, reports are being dropped. Probably for the same reason:

<probe> WARN: 2016/05/09 14:22:16.781555 Docker reporter took longer than 1s
<probe> ERRO: 2016/05/09 14:22:23.174719 Dropping report to 10.0.26.13:4040
<probe> WARN: 2016/05/09 14:22:25.812799 Docker reporter took longer than 1s
<probe> ERRO: 2016/05/09 14:22:27.388793 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:29.333923 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:32.135056 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:38.283660 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:41.262928 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:44.862768 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:50.425897 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:53.383612 Dropping report to 10.0.26.13:4040
<probe> WARN: 2016/05/09 14:22:55.870504 Docker reporter took longer than 1s
<probe> ERRO: 2016/05/09 14:22:57.934171 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:22:59.411752 Dropping report to 10.0.26.13:4040
<probe> WARN: 2016/05/09 14:23:01.011165 Docker reporter took longer than 1s
<probe> WARN: 2016/05/09 14:23:02.742918 Docker reporter took longer than 1s
<probe> ERRO: 2016/05/09 14:23:03.517608 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:23:06.190370 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:23:08.418448 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:23:11.232937 Dropping report to 10.0.26.13:4040
<probe> ERRO: 2016/05/09 14:23:14.856745 Dropping report to 10.0.26.13:4040

Resulting in a half-assed visualization of the service:

Note how an app-mapper, a frontend and the ui-servers are missing

Profile:
pprof.localhost:4041.samples.cpu.001.pb.gz

[2opremio]

There are ~200 containers per machine from which less than 100 are running.

In order to tweak this, kubelet's arguments need to be adjusted http://kubernetes.io/docs/admin/garbage-collection/ so it seems we should be able to support that number of containers.

[2opremio]

Here's the output of go tool pprof -focus 'GetNode' -png pprof.localhost\:4041.samples.cpu.001.pb.gz

[2opremio]

It seems we are building the nodes for the report in every reporter iteration which, for containers which didn't change, are wasted CPU cycles.

I will try caching the nodes and only regenerating then when they are affected by a docker event.

[tomwilkie]

The "Dropping report" is a new logging line added this release; it will be dropping reports due to the app being slow, not the probe.

[2opremio]

The "Dropping report" is a new logging line added this release; it will be dropping reports due to the app being slow, not the probe.

Good to know, then that's #1457