Multiline Filter can lose logs on shutdown

[PettitWesley]

Bug Report

For context on the multiline filter see: #4309

On shutdown if there is a pending multiline log in the buffer, it can be lost. I still need to spend some more time double checking exactly in what cases this happens, but I've seen it. Run Fluent Bit, send it multiline logs and use the filter and then send it a SIGTERM, and the last multiline is not always delivered.

I think this is because even if the multiline library flushes the data before shutdown, the in_emitter instance is already paused and so the records are never emitted. I need to investigate this more and check, but that's my hypothesis.

[agup006]

Needs design approval, @PettitWesley looking to get work done. Not critical for 1.9, however as soon as possible.

[PettitWesley]

Discussed with @edsiper I will implement this using the pause callback on the input. Right now a pause call back is not registered by in_emitter, but optionally it could be, and then this can be used to flush the multiline buffer on shutdown. I will work on implementing this.