Input pausing before processing all message after SIGTERM

[LucasHantz]

Hello,

I'm seeking guidance on where to handle my case where I'm losing logs due fact input are pausing and not accepting new input after SIGTERM signal has been sent.

I'm using firelens as the sidecar container of a long-running process CLI process on FARGATE that once is done exits.
As the container is declared as ESSENTIAL, ECS sends a SIGTERM signal to firelens which is taking the 30 of Grace configured but is pausing the input immediately and not allowing the latest record coming from the other container to be processed.
Unfortunately, this occurs more than half of the time and I would like to find a cleaner solution that to add a sleep at the end of all commands executed in the other container. What do you suggest I do?

[2021/07/01 14:40:23] [engine] caught signal (SIGTERM)
[2021/07/01 14:40:23] [ info] [input] pausing forward.0
[2021/07/01 14:40:23] [ info] [input] pausing forward.1
[2021/07/01 14:40:23] [ info] [input] pausing logs
[2021/07/01 14:40:23] [ info] [input] pausing metrics
[2021/07/01 14:40:23] [ info] [input] pausing io
[2021/07/01 14:40:23] [debug] [task] created task=0x7f7d7b437820 id=0 without routes, dropping.
[2021/07/01 14:40:23] [debug] [task] destroy task=0x7f7d7b437820 (task_id=0)
[2021/07/01 14:40:23] [ warn] [engine] service will stop in 30 seconds
[2021/07/01 14:40:37] [debug] [input chunk] tcp.2 is paused, cannot append records
[2021/07/01 14:40:53] [ info] [engine] service stopped
[2021/07/01 14:40:53] [debug] [input chunk] stream_processor.4 is paused, cannot append records
[2021/07/01 14:40:53] [debug] [input chunk] stream_processor.4 is paused, cannot append records
[2021/07/01 14:40:53] [debug] [input chunk] stream_processor.5 is paused, cannot append records
[2021/07/01 14:40:53] [debug] [input chunk] stream_processor.5 is paused, cannot append records
[2021/07/01 14:40:53] [debug] [input chunk] stream_processor.5 is paused, cannot append records

Thank you for your help.
Lucas

[zhonghui12]

Hello,

This issue might be related to an upstream issue: fluent/fluent-bit#2610. We will let the maintainer notice this issue and get it fixed.

Thanks.

[sparrc]

According to the release tags on fluent/fluent-bit@31d26a2, this should be fixed in fluentbit 1.9