Implement backoff for Kafka output

[ycombinator]

Describe the enhancement:

Currently the Kafka output does not support any sort of backoff for publishing events in the situation where the Kafka broker might've temporarily gone away. We should add support for this, similar to what the Redis and Elasticsearch outputs.

Describe a specific use case for the enhancement or feature:

To prevent the Kafka output aggressively retrying to publish events to a Kafka broker that might have temporarily gone away.

[ycombinator]

Potentially useful for implementation:

• beats/libbeat/outputs/kafka/config.go

  Line 256 in 95626b8

  // TODO: k.Producer.Retry.Backoff = ?

• https://github.com/Shopify/sarama/blob/58123455d1a70c7f438871597d9a1715462e5d1c/config.go#L218-L229

[ycombinator]

@faec @urso See the link in the previous comment for backoff/retry options for the producer in Sarama.

[urso]

BackoffFunc is "new". We should give it a try. From experience the 'Backoff' setting did not always work correctly, which did lead to us using 100% CPU (depending where an error occured and if the internal write buffer is full). Not sure if this has been improved.

We try to move most 'retry' handling to libbeat, because sarama does not support infinite retry. This might impact the BackoffFunc I presume. Plus, with exponential backoff we will need a way to reset the wait state upon success.

[ZHumphries]

Just experienced this issue. Someone changed kafka to use authentication and all winlogbeat instances on every server spiked to 100% cpu once the buffer was full. Fortunately the winlogbeat rollout wasnt complete so vmware didnt grid completely to a halt.

[toby-sutor]

It's worth mentioning that a temporary workaround is to add the Kafka IPs to the local /etc/hosts file of the Beats nodes to relieve the DNS servers until this has been implemented.