Multiline Filter example does not work with Forward input #4173
Comments
|
I tried doing things like running the tail example with the output of my python script; that worked, the multiline filter worked, so the python script seems to work fine. And then I tried running the tail example without the multiline filter, just to see what its output would be to stdout. I compared this with the output from the forward example... and its the same. So I can't figure out why the filter isn't working. It's getting the same logs in each case. My next course of investigation might be to log each incoming record to debug in the filter, to double check that in each case it really is getting the same logs. |
|
I found differences, but I don't know if they cause this issue.
I tested my out_gdetail plugin. https://github.com/nokute78/fluentbit-plugin-out-detail forward.log: {"format":"uint 32", "header":"0xce", "raw":"0xce61613776", "value":1633761142},
{"format":"fixmap", "header":"0x81", "length":1, "raw":"0x81a36c6f67af616e6f74686572206c696e652e2e2e", "value":
[
{"key":
{"format":"fixstr", "header":"0xa3", "raw":"0xa36c6f67", "value":"log"},
"value":
{"format":"fixstr", "header":"0xaf", "raw":"0xaf616e6f74686572206c696e652e2e2e", "value":"another line..."}
}
]
}tail.log: {"format":"event time", "header":"0xd7", "type":0, "raw":"0xd7006161378f0536d20c", "value":"2021-10-09 15:32:47.087478796 +0900 JST"},
{"format":"fixmap", "header":"0x81", "length":1, "raw":"0x81a36c6f67af616e6f74686572206c696e652e2e2e", "value":
[
{"key":
{"format":"fixstr", "header":"0xa3", "raw":"0xa36c6f67", "value":"log"},
"value":
{"format":"fixstr", "header":"0xaf", "raw":"0xaf616e6f74686572206c696e652e2e2e", "value":"another line..."}
}
]
} |
|
Oops, the type of timestamp doesn't cause this issue. I added below diff. diff --git a/plugins/filter_multiline/ml.c b/plugins/filter_multiline/ml.c
index 1ce9bc41..d7a118e4 100644
--- a/plugins/filter_multiline/ml.c
+++ b/plugins/filter_multiline/ml.c
@@ -171,7 +171,7 @@ static int cb_ml_filter(const void *data, size_t bytes,
size_t tmp_size;
struct ml_ctx *ctx = filter_context;
struct flb_time tm;
-
+ flb_error("size=%d",bytes);
/* reset mspgack size content */
ctx->mp_sbuf.size = 0;The output is different. in_tail: in_forward: |
|
I'm facing same issue when using multiline parser with forward input. When enabling multiline filter, logs are followings. When disabling multiline parser by comment out FILTER in fluent-bit.conf, logs are followings. I used fluentd log driver with https://github.com/konoui/multiline-with-forward-input. |
|
@nokute78 I found the same. I added some debug logs here: https://github.com/PettitWesley/fluent-bit/tree/ecs-multiline-debug I found that:
I am not sure what to do to fix this. @edsiper It appears that filter multiline only works with tail or other plugins that will send all logs at in one nice big chunk. Any suggestions? Attached are my logs which show the filter returning each time. |
|
I tried increasing |
|
@nokute78 Do you think this is a bug in the forward input or its interaction with the core? It should be ingesting logs in chunks same as tail? Is it because the coroutine callbacks work differently for each plugin? Because forward uses connections, but tail is just reading from a file. |
|
It's possibly worth highlighting that the forward input seems to work in combination with the multiline parser if the source of the data is another Fluent Bit instance configured with a forward output. However, I was not able to get multiline parsing working with the head input, regardless of the buffer size. |
|
@PettitWesley I think it is necessary to add some kind of buffering mechanism for filter_multiline. By the way, in the document, it is recommended to use in_tail to concatenate CRI logs. |
|
I figured out how to make the filter work as expected, by re-writing it to use in_emitter: aws/aws-for-fluent-bit#100 (comment) I will post a PR and design next week. |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
Implementation of the fix mostly done but still in progress: #4309 |
Yes, it should be highlighted. I didn't dive much into the code. Now that one need to concatenate logs using multiline FILTER coming from docker logs source, put an dedicated peer forward INPUT for the docker instance then forward logs to the next central peer collector. Use the multiline FILTER on the central peer side. This design also compliant with production workflow. But i still wonder how the forward INPUT buffering works differently with the multiline FILTER.
|
|
@prettykingking Once this #4383 is merged then the multiline filter will work as expected :) |
Thank you for the redesign and it's implementation of the improved multiline FILTER. I see that it's been merged. Prior to that feature or major release. For those who face the same issue, this is a workaround for version |
|
Hi, I read the doc from this PR https://github.com/fluent/fluent-bit-docs/pull/675/files, and added What happens is that the "last" log never reaches output unless a new log comes in. I also have set a Am I missing something obvious ? (using Fluent Bit v1.8.12) |
|
@squalou to be clear, in your case, is the log lost because Fluent Bit shut down? If you are using the latest release, any buffered multiline logs should be purged from the buffer once |
|
I'm afraid not : it's running, it is a docker container running side by side with an application, if one of them is shut down the other would be killed too. (that's on purpose, not to give too much details,its "one task" in ECS sense, with two running containers) The application is logging absolutely nothing for a few minutes after it has started, (on purpose). |
@squalou Hmmm I am confused... this was the old behavior before I added the
I understand; I work for ECS and I created FireLens (which I assume is what you are using). You may want to open an issue in the AWS repo or contact AWS Support for further investigation: https://github.com/aws/aws-for-fluent-bit |
|
Thank you for the details, I'll get in touch with them. I use I should try another one when ready and stable I guess. |
|
@squalou Hmmm 2.22.0 should work. Again, please contact AWS Support but also if you can come up with any sort of a simplified case that repros this and send it to me, then that'd help. If you need to send any details that are confidential and can't be shared on github, then AWS Support can facilitate that. |
|
I am trying to read whole message from systemd plugin. Jan 06 13:03:00 edge-Virtual-Machine 9e46f1b5f888[13342]: trce: IoTEdge.Services.IoTHubDeviceApiV2Service[0] In current configuration the is trimmed after 1 line. Jan 06 14:41:40 edge-Virtual-Machine fae510e87e43[13342]: [26] proxy_log: [1641460295.254838000, {"PRIORITY"=>"6", "_TRANSPORT"=>"journal", "_SELINUX_CONTEXT"=>"unconfined The MESSAGE field should be able to read whole message. The regex patterns that I am using seems to be working with tail input plugin but not with systemd. parser.conf fluent-bit.conf Please help me with the issue. What I am doing wrong here. |
|
@mahima145 Use https://rubular.com/ to test your regex against your logs, AFAICT, the start state doesn't match your first log. |
|
@PettitWesley it's a mistake while copying. This is the start state which is matching the log while using Tail |
|
Is this fix merged into which Fluent Bit release - 1.8.x or 1.9.x? |
|
This was released in 1.8.12: https://fluentbit.io/announcements/v1.8.12/ Do to a mistake, we accidentally didn't include it in the 1.9 series at first, but 1.9.3 adds it back in: https://fluentbit.io/announcements/v1.9.3/ |
PettitWesley
changed the title
Bug Report
Describe the bug
I took the multiline example here and turned it into an example that works with Forward input: https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace
However, that didn't work. I am not able to get the filter working and actually concatenating multilines with the forward input. Which is strange, since there shouldn't be a difference between my forward example and the tail example from the POV of the filter.
To Reproduce
First, create a container image that will output the log file from the example to stdout:
You could just cat the file... but I went for a python script that prints it line by line:
test.logis the file from the example in the docs.You can run this container with the fluentd docker log driver:
And then capture the logs with this configuration:
The parsers file is the same as the one from the example.
Expected behavior
Multiline example should work with forward input.
Screenshots
Your Environment
Additional context
Amazon ECS FireLens uses the Fluentd Docker Log Driver and forward input for logs, and so this issue has impacted many AWS customers: aws/aws-for-fluent-bit#100
The text was updated successfully, but these errors were encountered: