feat(serf) override serf probe interval and timeout w/ environment variables

[akomelj]

This pull request implements an optional mechanism for overriding serf probe interval (default: 1 s) and timeout (default: 0.5 s) via environment variables AGENT_PROBE_INTERVAL and AGENT_PROBE_TIMEOUT.

A little background: I'm running Docker Swarm workers on a network of Raspberry Pis scattered all around the site. Network connectivity to the Pis is reliable enough for a swarm with longer dispatcher-heartbeat setting to survive regular network slowdowns, but Portainer Agent, in my case, relies too heavily on refuted suspect messages to keep the other agents connected. Default timeout for receiving acknowledgments from peers is simply too low for such network (bad Wi-fi signal reception + underpowered hardware). Moreover, sometimes even refutes get lost and fall-back TCP ping mechanism does not wait for successful TCP retransmits before deciding the peer is dead. As a consequence, I'm experiencing all the symptoms described in portainer/portainer#2535.

I tracked the instability issues to default (sane) LAN configuration of serf probe in Agent, namely the probe interval and timeout. By running a custom build of Agent where I replaced the Agent's default settings with serf's WAN presets, I got rid of endpoint instability issues completely and my agent logs are much cleaner now. I'm therefore submitting this pull request where probe interval and timeout can be overridden by two environment variables - it may come in handy to people running swarm on poor networks.

[deviantony]

Thanks for this contribution @akomelj ! I'll do a review of this real soon.

[deviantony]

Closing in favor of #241

Super sorry for giving you no update on this topic @akomelj, this is quite a good contribution and should have been merged earlier.

Sorry and thanks again !