List of endpoints inside probe can grow over time

[bboreham]

Another picture showing report sizes over time, similar to #3576

It's not clear what the trigger is, but looking inside the reports the number of Endpoints grows by a few every time; up to 36,000 after 8 days. Restarting the probe resets the size down to normal levels.

conntrack -L on the node lists around 8,000 connections, mostly in TIME_WAIT.

[bboreham]

I suspect all the probes that are affected are using conntrack rather than ebpf. E.g. I see this in the logs:

<probe> ERRO: 2019/07/01 13:39:49.650529 tcp tracer received event with timestamp 726896371146595 even though the last timestamp was 726896371186140. Stopping the eBPF tracker.
<probe> WARN: 2019/07/01 13:39:50.818972 ebpf tracker died, restarting it
<probe> ERRO: 2019/07/01 13:42:41.287671 tcp tracer received event with timestamp 727068008148899 even though the last timestamp was 727068008151558. Stopping the eBPF tracker.
<probe> WARN: 2019/07/01 13:42:41.816997 ebpf tracker died again, gently falling back to proc scanning 

[bboreham]

Still happening after #3648.
Right now I am out of ideas how we can be leaking connections.
I wonder if we should do a periodic resync, e.g. once an hour, which would mask whatever is really causing it.

[bboreham]

I had an idea to improve the first problem - ebpf tracker died: iovisor/gobpf#42 (comment)

[bboreham]

#3653 made a big improvement, according to my stats, but still seeing fall-back to conntrack in a few cases and then constant growth over hours.