Core Kinesis Firehose and Kinesis Streams plugins

[PettitWesley]

Currently we have go plugins:

• https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit
• https://github.com/aws/amazon-kinesis-streams-for-fluent-bit

Ideally, these plugins should be re-written in C and contributed to the core of fluent bit. This would fix known throughput issues with the go plugins: #2159

I will do this eventually, though I am not sure the core versions will necessarily fully replace the go plugins. The Kinesis streams go plugin will soon have KPL aggregation support- that feature might be harder to port to C- the core versions of these plugins might be simpler.

[PettitWesley]

I've started prototyping a core Kinesis Firehose plugin, though its not clear when I will have time to finish it.

[zihengCat]

Fluent Bit AWS Kinesis output plugins rely on AWS Go SDK, it's not so easy to be re-written in C.
Maybe ports AWS Go SDK Kinesis to C ?
https://github.com/aws/amazon-kinesis-streams-for-fluent-bit/blob/dc9767ccd86f2b0484fa4bb614d5ac24d0908690/kinesis/kinesis.go#L432

[PettitWesley]

I already did that; wrote an AWS SDK in C for Fluent Bit earlier this year: https://github.com/fluent/fluent-bit/tree/master/src/aws

A Kinesis Firehose C plugin will be released in 1.6: https://github.com/fluent/fluent-bit/tree/kinesis-firehose-high-throughput

The firehose plugin and kinesis plugin code are very similar. Long term I want to rewrite that firehose code into a more generic library that can support both kinesis streams and firehose.

[PettitWesley]

New output kinesis_firehose was launched a while ago, and in 1.7.0 today kinesis_streams was added!

We now have core versions of all of the official AWS go plugins!