[ECS] [Fargate]: Partial json logs concatenation with Fluentbit

[kgruszka]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I want to have multi-line json logging supported with the built-in Fluent Bit logging on ECS/Fargate.
One option would be to make proposed config available by simply referencing the path like

"config-file-value": "/container-json.conf" 

instead of building custom image for that. It's not ideal we need to maintain additional image to make it work.

Which service(s) is this request for?
ECS Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Container logs are being splitted into 16kb chunks with addition of partial attributes. This prevents centralized logging system to display the logs properly.

Are you currently working around this issue?
I have built a custom image based on official AWS fluentbit image and added custom config for parsing.

Dockerfile

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

ADD extra.conf /extra.conf

extra.conf based on https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace#docker-partial-message-use-case

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Flush 1
    Grace 30

[FILTER]
    name multiline
    match *
    multiline.key_content log
    mode  partial_message

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json
    Reserve_Data True    

Then in task definiton we have container defined:

    {
      "essential": true,
      "image": "***.dkr.ecr.us-east-1.amazonaws.com/***/fluentbit:stable",
      "name": "log-router",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true",
          "config-file-type": "file",
          "config-file-value": "/extra.conf"
        }
      }
    },

Additional context
#1326
aws/aws-for-fluent-bit#100
fluent/fluent-bit#821

Attachments
Splitted logs

Merged logs