Split INPUT into different OUTPUTs

[scrwr]

I might be missing something here, as I think, the following is a pretty basic task. But we could not get it to work.

Is your feature request related to a problem? Please describe.
I have an input stream, lets say from Syslog INPUT plugin. Based on the logged information, I want to send the messages containing a specific text (with regex pattern) to OUTPUT A and other messages to OUTPUT B.

Describe the solution you'd like
I would have assumed, there is a kind of Re-Tag Mechanism. E.g. via FILTER.
[FILTER]
name retag
match inputtag
regex message ^line-starts-with
Tag newtag

Describe alternatives you've considered
We tried to duplicate the INPUT to the desired OUTPUT + a second FORWARD OUTPUT to an local FORWARD INPUT, which got another "Tag" attribute. However it seems like the Forward INPUT plugin ignores the Tag attribute and uses the forwarded tag instead. But even if it would have been worked, it is of course an ugly hack.

Or in other words, I want pretty much this: https://docs.fluentd.org/v1.0/articles/out_rewrite_tag_filter

[edsiper]

HI, there is no rewrite tag support at the moment but it will be implemented during Q3/Q4 this year,

[qingling128]

Hi, is this feature currently supported now? Sounds like it might have been implemented according to the estimated timeline?

[mitchellmaler]

@edsiper What is the status of this feature? Based on the reply above it seemed like it would have been implemented already but I have not seen it.

[edsiper]

No direct filter to accomplish that, but you can use the stream processor to create a new tagged stream

[kkniffin]

Are there any good examples of using the stream processor with incoming syslog data to tag?

[recipedude]

@kkniffin not re-tagging syslog data but here's an example of tagging via the stream processor that seems to be working.

  stream-processor.conf: |
    [STREAM_TASK]
        Name   rails
        Exec   CREATE STREAM results WITH (tag='rails') AS SELECT * FROM TAG:'kube.*' WHERE kubernetes['labels']['fluentbit']='rails';

[bgola-signalfx]

Any update? Splitting single input into separate outputs using filters + retagging would be really useful.

[edsiper]

@bgola-signalfx to send one record to multiple places you use a match rule, note that the concept of re-tagging overrides the original tag.

[rajulm]

I am not sure if I quite understood the last comment. We have a similar use case where we want to send raw input messages to output A and messages containing a specific text (with regex pattern) to OUTPUT B.

Is it supported now?

[gabegorelick]

The rewrite_tag filter was added in #293.

See https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag.