multiline parsing not working properly - general guidance

[daniloasfigueiredo]

Hello, we have some applications hosted on EKS and we are using fluentbit to foward logs to elasticsearch and it works well, our application send logs to stdout in json/multiline format and we are using fluentbit to capture and foward logs with custom parser (regex) to support multiline logs.

For archiving purposes we tried to configure another fluentbit container with same configurations and changed output to s3, but the logs stored on s3 it's splited in two or more lines, we tried some fixes, but did not work.

fluentbit image
image: amazon/aws-for-fluent-bit:latest

fluentbit config file
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level debug
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020

@INCLUDE input-*.conf
@INCLUDE output-*.conf
@INCLUDE filters.conf

input-s3logs.conf: |
[INPUT]
Name tail
Tag example.*
Path /var/log/containers/example-*.log
Path_Key filePath
Key log
DB /var/log/flb_api.db
Parser docker
Docker_Mode On
Docker_Mode_Parser multi_line

output-s3.conf: |
[OUTPUT]
Name s3
Match *
bucket /mybucket-logs/
region us-east-1
use_put_object On
s3_key_format /$TAG[1]/$TAG[3]/%Y/%m/%d/
s3_key_format_tag_delimiters .
total_file_size 5M
upload_timeout 1m

filters.conf: |
[FILTER]
Name parser
Parser docker
Match *
Key_Name log
Reserve_Data On
Preserve_Key On

parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On

**[PARSER]
    Name        multi_line
    Format      regex
    Regex      (?<log>^{"log":"{\\"datetime\\":\\"\d{4}-\d{2}-\d{2}.*)**

application log example
{"datetime":"2021-05-02 15:18:00 UTC","level":"warning","system":"example","module":"v3","uri":"default\index","uuid":"b5e3628a-fc86-415c-a0db-86fa8a919581","ip":"example","message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test con
text com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","context":{"warning aqui":"warning","exception":{"message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test c
ontext com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","trace":["#0 [internal function]: api\modules\v3\controllers\DefaultController->actionIndex()","#1 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()","#2 /app/vendor/yiisoft/yii2/base/Contro
ller.php(181): yii\base\InlineAction->runWithParams()","#3 /app/vendor/yiisoft/yii2/base/Module.php(534): yii\base\Controller->runAction()","#4 /app/vendor/yiisoft/yii2/web/Application.php(104): yii\base\Module->runAction()","#5 /app/vendor/yiisoft/yii2/base/Application.php(392): yii\web\Application->handleRequest()","#6 /app/api/web/index.php(22): yii\base\Application->run()","#7 {main}"],"file":"/app/api/modules/v3/controllers/DefaultController.php","line":33}}}

example output logs stored on s3 (log is splited in 4 lines)
{"date":"2021-05-02T15:16:50.743019Z","log":"{"datetime":"2021-05-02 15:16:50 UTC","level":"error","system":"example","module":"v3","uri":"default\\index","uuid":"db952e9a-e0d5-4aee-a670-95ed5830739c","ip":"191.238.221.160","message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test conte\n","stream":"stderr","time":"2021-05-02T15:16:50.743019893Z"}
{"date":"2021-05-02T15:16:50.743101Z","log":"xt com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","context":{"exception":{"message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, \n","stream":"stderr","time":"2021-05-02T15:16:50.743101041Z"}
{"date":"2021-05-02T15:16:50.743113Z","log":"test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","trace":["#0 [internal function]: api\\modules\\v3\\controllers\\DefaultController->actionIndex()","#1 \/app\/vendor\/yiisoft\/yii2\/base\/InlineAction.php(57): call_user_func_array()","#2 \/app\/vendor\/yiisoft\/yii2\/base\/Controller.php(181): yii\\base\\I\n","stream":"stderr","time":"2021-05-02T15:16:50.743113495Z"}
{"date":"2021-05-02T15:16:50.743121Z","log":"nlineAction->runWithParams()","#3 \/app\/vendor\/yiisoft\/yii2\/base\/Module.php(534): yii\\base\\Controller->runAction()","#4 \/app\/vendor\/yiisoft\/yii2\/web\/Application.php(104): yii\\base\\Module->runAction()","#5 \/app\/vendor\/yiisoft\/yii2\/base\/Application.php(392): yii\\web\\Application->handleRequest()","#6 \/app\/api\/web\/index.php(22): yii\\base\\Application->run()","#7 {main}"],"file":"\/app\/api\/modules\/v3\/controllers\/DefaultController.php","line":33}}}\n","stream":"stderr","time":"2021-05-02T15:16:50.743121884Z"}

example output logs stored on elastisearch (working fine)

{
"_index": "example-2021.05.02",
"_type": "docker",
"_id": "_DzILXkB6KeZcSic4r0z",
"_version": 1,
"_score": null,
"fields": {
"stream.keyword": [
"stderr"
],
"context.exception.file.keyword": [
"/app/api/modules/v3/controllers/DefaultController.php"
],
"log": [
"{"datetime":"2021-05-02 15:53:29 UTC","level":"error","system":"example","module":"v3","uri":"default\\index","uuid":"af4c9121-2de0-40e8-b047-616e19e804c3","ip":"191.238.221.160","message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test conte\nxt com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","context":{"log de nivel":"error","exception":{"message":"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test conte\nxt com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL","trace":["#0 [internal function]: api\\modules\\v3\\controllers\\DefaultController->actionIndex()","#1 \/app\/vendor\/yiisoft\/yii2\/base\/InlineAction.php(57): call_user_func_array()","#2 \/app\/vendor\/yiisoft\/yii2\/base\/Controller\n.php(181): yii\\base\\InlineAction->runWithParams()","#3 \/app\/vendor\/yiisoft\/yii2\/base\/Module.php(534): yii\\base\\Controller->runAction()","#4 \/app\/vendor\/yiisoft\/yii2\/web\/Application.php(104): yii\\base\\Module->runAction()","#5 \/app\/vendor\/yiisoft\/yii2\/base\/Application.php(392): yii\\web\\Application->handleRequest()","#6 \/app\/api\/web\/index.php(22): yii\\base\\Application->run()","#7 {main}"],"file":"\/app\/api\/modules\/v3\/controllers\/DefaultController.php","line":33}}}\n"
],
"context.exception.line": [
33
],
"uuid.keyword": [
"af4c9121-2de0-40e8-b047-616e19e804c3"
],
"context.exception.message": [
"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test conte\nxt com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL"
],
"uuid": [
"af4c9121-2de0-40e8-b047-616e19e804c3"
],
"context.exception.file": [
"/app/api/modules/v3/controllers/DefaultController.php"
],
"uri.keyword": [
"default\index"
],
"datetime": [
"2021-05-02 15:53:29 UTC"
],
"stream": [
"stderr"
],
"ip.keyword": [
"example"
],
"tag": [
"example.log"
],
"level": [
"error"
],
"ip": [
"191.238.221.160"
],
"module": [
"v3"
],
"filePath": [
"example.log"
],
"module.keyword": [
"v3"
],
"system.keyword": [
"example"
],
"message": [
"test context com multiplas keys, test context com multiplas keys, , test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test conte\nxt com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys, test context com multiplas keys FINAL"
],
"uri": [
"default\index"
],
"system": [
"example"
],
"@timestamp": [
"2021-05-02T15:53:29.802Z"
],
"context.log de nivel": [
"error"
],
"level.keyword": [
"error"
],
"filePath.keyword": [
"/var/log/containers/example.log"
],
"context.log de nivel.keyword": [
"error"
],
"datetime.keyword": [
"2021-05-02 15:53:29 UTC"
],
"tag.keyword": [
"api.var.log.containers.api-6548469f95-l8t29_example_api-db5dc24f36d97d8aed3708adfdf482117f258a67de00d05f334f9e0400c733c7.log"
],
"context.exception.trace.keyword": [
"#0 [internal function]: api\modules\v3\controllers\DefaultController->actionIndex()",
"#1 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()",
"#2 /app/vendor/yiisoft/yii2/base/Controller\n.php(181): yii\base\InlineAction->runWithParams()",
"#3 /app/vendor/yiisoft/yii2/base/Module.php(534): yii\base\Controller->runAction()",
"#4 /app/vendor/yiisoft/yii2/web/Application.php(104): yii\base\Module->runAction()",
"#5 /app/vendor/yiisoft/yii2/base/Application.php(392): yii\web\Application->handleRequest()",
"#6 /app/api/web/index.php(22): yii\base\Application->run()",
"#7 {main}"
],
"context.exception.trace": [
"#0 [internal function]: api\modules\v3\controllers\DefaultController->actionIndex()",
"#1 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()",
"#2 /app/vendor/yiisoft/yii2/base/Controller\n.php(181): yii\base\InlineAction->runWithParams()",
"#3 /app/vendor/yiisoft/yii2/base/Module.php(534): yii\base\Controller->runAction()",
"#4 /app/vendor/yiisoft/yii2/web/Application.php(104): yii\base\Module->runAction()",
"#5 /app/vendor/yiisoft/yii2/base/Application.php(392): yii\web\Application->handleRequest()",
"#6 /app/api/web/index.php(22): yii\base\Application->run()",
"#7 {main}"
]
},
"sort": [
1619970809802
]
}

[daniloasfigueiredo]

Solved after config review.