-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement tag-rewrite feature #293
Comments
@dawidmalina the tag rewrite plugin was just sent in a PR but have not been reviewed yet. The type of string composing in a configuration is not supported yet, likely to be implemented for 0.13 version (we are releasing 0.11.9 today and 0.12 ~next week) |
Renaming this topic to set it as an enhancement request |
Fluentd For example nginx container needs to separate stderr and stdout for matching parsers for different formats:
|
@edsiper any news (plans) for this improvement? |
I am looking for a possibility to rewrite tag while using another field. for example:
|
No ETA yet for this feature. |
@edsiper any news about this feature? |
@edsiper do you think this can be implemented for 0.15 ? I think there are a lot of users looking for this change. Maybe you could put here some guidelines how this can be implemented and someone could contribute. |
looks like it will complicate #291 |
yes, after checking all sources, much easier way it makes some meta plugin who have one input and many outputs with different tags (after you can be filtering each tag). |
This would be a great feature. |
It depends on what exactly do you do. If you have many inputs with different tags it's possible if you have only one input it's not possible. |
But how to use it with kubernetes, @stalkerg |
@stalkerg |
A short update: there is some progress on this, note that this feature is not a straightforward thing since some previous tasks are required. One the primary task recently done is a new buffering mechanism that implements a new way for input plugins to register records+tags. so this is currently "work in process". |
What is the status of this? |
What about adding this ability to Lua filters? We have the opportunity to modify timestamp and records, and we get the tag as an input, why not allow this to be changed there also? |
Sitting in the deep dive talk at KubeCon right now, realising this should be possible with stream processing and creating a new stream with a new tag. |
This worked for me to create a new event with specific tag for re-processing - only issue is that I wasn't able to find a clean way to discard the original event. YMMV.
|
in the [INPUT] set routable off |
Is there any way to do this with Lua filters? Stream processing seems to be to new. |
Feature added to roadmap for next Major Release v1.4. POC will be updated shortly |
@edsiper where can I find the PoC? Can we do something else to push this forward? (testing, contributing?) |
Would be massively useful if we would have a way to rewrite the tag based on the value of another field. I think this is not possible even in stream processing? As far as I know it's only possible to use a fixed string in stream processing and not select a value of a field to be a part of the tag of that newly created stream. Correct me if I'm wrong. |
Hello! just a heads up! tag rewrite is a feature planned for v1.4 release (end of January 2020), you can track the progress of v1.4 release on this milestone: https://github.com/fluent/fluent-bit/milestone/7 -- @juho9000 yes, tag composition is only string-based for now, that will be improved. |
Will it be on track for the coming release at the end of the month ? By the way, awesome feature ! |
Any updates on when this will be released? Anyway I can help test this feature (or others gating the release)? My team is interested in moving from fluentd to fluent-bit, but we would prefer to use this feature over the stream processing feature for routing. |
it's currently in "work in process", v1.4 release will include this new filter. why is taking more time than expected?Originally Fluent Bit design did not consider a way to re-tag records, even at that time no filtering feature was available. But as of today is a totally different story. The future rewrite_tag filter plugin (this is a filter, not an output plugin as in Fluentd), will support subkeys matching and regex capture, so you will be able to create a rule like this:
As you can see this implementation is a bit complex, we are implementing custom features into the core to make it easier from a plugin perspective like the way to access record subkeys, matching groups and tag/strings composing. I really appreciate your patience on this. No v1.4 will be released until this plugin is ready. |
There are certain scenarios where other plugins of Fluent Bit pipeline would like to ingest a record with a custom tag (e.g: rewrite_tag). This patch implements a new 'internal' input plugin called 'emitter' that exposes an API for such purpose. Q: Why a new input plugin instead of a direct core function ? A: Plugins have a notion of metrics and storage buffering, offering this through an input plugin interface allows to inherit all other great features for monitoring, e.g: you will be able to track metrics for custom emitted records. Signed-off-by: Eduardo Silva <[email protected]>
This patch adds a new filter called 'rewrite_tag' that allows to re-tag records based on matching rules using regular expressions. This plugin is built on top of new Fluent Bit features exposed by the record accessor, allowing to create custom tags using records key content, tag content or any placeholder supported. To get started, the filter needs to define rules, can be one or many and they are processed in order until one of them matches the required criteria, a rule have the following format: Rule KEY REGEX NEW_TAG KEEP - KEY : name of a record key prefixed with '$', e.g: $name - REGEX : regular expression defining the matching criteria - NEW_TAG: new tag to be defined, here you can use any placeholder supported. - KEEP : boolean value that defines if the record matches, the original one must be preserved or discarded. == Usage == Consider the following JSON payload (formatted for readability): { "name": "abc-123", "ss": { "s1": { "s2": "flb" } } } Assumming the Tag associated to the record is 'aa.bb' we can use the following configuration to rewrite it Tag and discard the original record: [FILTER] Name rewrite_tag Match aa.bb Rule $name ^([a-z]+)-([0-9]+)$ test.$TAG[0].$1.${HOSTNAME} false [OUTPUT] Name stdout Match test.* == About Placeholders == The tag composition is very flexible and supports the following placeholders: - ${VAR} => translated from an environment variable, e.g: ${HOSTNAME} - $TAG => full Tag associated to the record - $TAG[n] => Tag part, parts are values between dots e.g: aa.bb, where $TAG[0] will retrieve 'aa' and $TAG[1] will get 'bb'. - $key => value from the record key called 'key', e.g: specifying $name will return 'abc-123'. - $key['x'] => if the record 'key' value is a map, using one or multiple square brackets you can specify the sub-key value desired. Considering the above JSON example we could specify the key $ss['s1']['s2] to retrieve the value 'flb'. Have fun ;) Signed-off-by: Eduardo Silva <[email protected]>
So you can start testing now (you need to build from GIT master):
|
Documentation draft: https://docs.fluentbit.io/manual/v/1.4-dev/pipeline/filters/rewrite-tag |
Documentation extended with examples and Monitoring section. Closing this enhancement request as fixed |
* developer-guide: fix broken links * library_api: fix broken links Signed-off-by: Takahiro YAMASHITA <[email protected]>
Hi
I'm evaluating fluentd-bit and it looks very promissing. I know that record_modifier was just merged and i would like to convert my old config to fluentd-bit way. Is it possible to convert this fluentd config part to fluentd-bit?
The text was updated successfully, but these errors were encountered: