[WIP] Rate limit reading expensive files from /proc

[tomwilkie]

Fixes #812

[tomwilkie]

This seems to have some unintended side effects on the edges that get detected. Needs more work.

Example without this change:

With this change:

[tomwilkie]

Have rebased, increased limits and made it so results are deleted once used (a bit racey right now, just an experiment). Results are improved but he connections are still flakey and sometime outright odd.

Its worth testing without conntrack (as that has known bugs) so you know any mis-connections are due to this change:

./scope launch --probe.conntrack=false

[2opremio]

I've experimented with grouping the processes by namespace to ensure that all the reads of /proc/PID/fd/* are done immediately after reading /proc/PID/net/tcp{,6}.

However, I still miss a lot of connections.

The cause most likely lays in the inconsistencies resulting from caching /proc/PID/net/tcp{,6} and not /proc/PID/fds. I will start by caching both to see if it solves the problem.

[2opremio]

This is still hacky/work in process, but works well. Now there's a background gorountine walking /proc/PID/fd/* and /proc/PID/net/tcp{,6} in a loop with dynamically adjusted ratelimiting.

On the way I found out that we were not reading the network namespace inodes properly (the inode of the symlink is not the inode of the namespace), resulting in reading /proc/PID/net/tcp{,6} for all processes (see f63b637).

In the average case that should provide over one order of magnitude speedup to the PID walker (e.g. in my machine I have 162 processes and only 7 net namespaces).

So, for now, maybe it's enough to simply cherrypick f63b637 and probably 2c32110 .

[2opremio]

I have made some measurements to see if it's still worth having a background reader after fixing the namespace reading but it doesn't seem to be the case.

With 60 network namespaces, the probe is at ~15% CPU in both cases. BTW, master is over 30%, which gives us a 2x gain :) .

So, I am closing this in favor of #898 and I will cherry-pick f799dc8 into a separate PR.

[tomwilkie]

+1

On Tuesday, 2 February 2016, Alfonso Acosta [email protected]
wrote:

Closed #814 #814.

—
Reply to this email directly or view it on GitHub
#814 (comment).