Consistent crashes on 2.21.0

[PettitWesley]

This issue is to flag for everyone that 2.21.0 seems to be unstable (seems to be networking related): https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.21.0

I'm not able to find an upstream issue corresponding to this problem, and I can't get a consistent stack trace for it.

But the key thing to note is that 2.21.1 fixes the issue and appears to be stable. However, we'd recommend using 2.21.2 which is the same underlying Fluent Bit version with an additional fix for IMDS: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.21.2

This is based on an independent customer report + multiple attempts by me to setup long running workloads on each version and see which ones crash. 2.21.0 consistently crashes after a few days, 2.21.1 seems stable.

Soon, our team will build these sorts of long running workload tests into our release pipeline to ensure we catch these instability issues before we release.

[PettitWesley]

Attaching several valgrind outputs which I got for this. Unfortunately they are not consistent. Since the next upstream version seems to fix these issues and is stable I'm not pursuing deeper investigation for now.
ticket-:log_router:8bc579b6e4f54cc080ec2443e17ba751.txt
ticket-:log_router:d346e6e9c9cf46d5b4b3ab77621b196c.txt
ticket-:log_router:7664995fb51d41afb562600a322c5d0c.txt
ticket-:log_router:50690a6522fd4a8584c4c14fbd4ae592.txt