Add support for dynamic log groups using templating variables.

[davidnewhall]

Closes #46, closes #6, closes #24, closes #49.

Description of changes:

This builds on #78 (more commits to that branch), and can be merged after (making this diff smaller), or in leu of #78.

I create a new struct type called Event that carries the Event payload and metadata through the entire request process. Previously the code was passing around the log stream name, and now it passes around the Event which contains the stream name, group name, tag and log record. This makes the data easier to reason about.

• I adapted the same code that created a template from the log stream name to the log group.
• Added a new groups map-tracker to the OutputPlugin struct. Gets added to when new groups are created.
• Added the log group name to the logStream struct, so each log stream has a pinned unique group.
• Auto create group was removed; groups are now auto-created if they don't exist.
• The group name now supports template variables, explained in the README.

This config:

    log_group_name      /ecs/twitch-fluentbit/$(tag)/$(host)
    log_stream_name     $(tag[0]).$(ident)

Turns into this (groups):

Streams Created:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[cwyl02]

hello when will this get review/merged? This is a feature we are looking for!

[hossain-rayhan]

Hey @cwyl02 , sorry its taking longer but we are on it.

[hossain-rayhan]

Shouldn't it be $(variable)? From #78 conversation, we planned to replace ${ } with $( ). Am I missing something?

[davidnewhall]

I was waiting for your team to provide feedback on the chosen variable identifiers. If y'all are good with #() I'll go ahead and fix the doc. Is there any additional feedback before making a final commit?

[PettitWesley]

We voted and we prefer $()

[davidnewhall]

Sounds good, I'll switch it up and test before committing.

[davidnewhall]

Added a commit changes the identifiers. Updated the readme. Made some comments better. Fixed a few log lines and added a couple debug lines. If you guys don't mind I'll follow-up with a clean-up PR (no features).