Output kafka confluent

[otherpirate]

Hi,

Overview

I created these plugins to handling with large number of metrics.
I'm using a library from Kafka Confluent called librdkafka
It was write in C/C++, so to create telegraf docker image, I had to build with static files on debian environment.
All files are in this pull request, but the image are here

Build

To build I'm follwing this steps (I'm not sure it's the best way, maybe Telegraf-team could help me improve it?)
1 - Go to Telegraf folder
2 - docker run --name telegraf-librdkafka-deb -it -v $PWD:/go/src/app --rm otherpirate/golang1.10.1-stretch-librdkafka
3 - ARGS=--already-builded make docker-image

Benchmark

I tried the kafka output, but it is too slow to large number of metrics per second.
All benchmark are running in the cloud.

# Test with 150 messages
2018-05-16T13:07:00Z D! Output [kafka] buffer fullness: 150 / 1000000 metrics.
2018-05-16T13:07:00Z D! Output [kafka_confluent] buffer fullness: 150 / 1000000 metrics.
2018-05-16T13:07:00Z D! Output [kafka_confluent] wrote batch of 150 metrics in 46.930293ms
2018-05-16T13:07:00Z D! Output [kafka] wrote batch of 150 metrics in 318.260897ms

# Test with 3000 messages
Output [kafka] buffer fullness: 2941 / 1000000 metrics.
Output [kafka_confluent] buffer fullness: 2941 / 1000000 metrics.
Output [kafka_confluent] wrote batch of 2941 metrics in 650.70158ms
Output [kafka] wrote batch of 2941 metrics in 6.185083317s

# Test with 15000 messages
Output [kafka] buffer fullness: 14627 / 1000000 metrics.
Output [kafka_confluent] buffer fullness: 14627 / 1000000 metrics.
Output [kafka_confluent] wrote batch of 14627 metrics in 2.411144316s
Output [kafka] wrote batch of 14627 metrics in 28.912976898s

Obs: Working on unit tests

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

[danielnelson]

@otherpirate Thanks for the pull request. I've heard that the kafka output is too slow, and I hope we can take a pass at optimizing the sarama based output soon. I think there are some changes we can make that will be helpful, but I expect it can not be as fast as the C library.

Before we can accept plugins with C dependencies into the main repo we need to ensure that having the library/headers is optional. Since we support so many platforms we can't want any hard dependencies. For the official builds, currently all platforms except linux/amd64 are cross-compiled without CGO from the .circle/config.yml. This is probably non-trivial to setup because the go tools aren't really sufficient.

[danielnelson]

I believe we can close this with the changes from #4491.