fluent-bit can sometimes hang indefinitely on read(...)/getaddrinfo(...)

[ShelbyZ]

Bug Report

Describe the bug
fluent-bit can sometimes hang indefinitely on read(...)/getaddrinfo(...)

• Steps to reproduce the problem:
• Using firelens-datajet to simulate input send data to 4 cloudwatch_logs output plugins build a debug fluent-bit version and enable core dumps
• Wait anywhere from 30-180min
• Observe CloudWatch logs are no longer received
• docker exec into container and inspect that at least 1 thread is waiting on read(...) or getaddrinfo(...)
• Re-check CloudWatch logs for no further logs

Expected behavior
Network calls should timeout/closed due to error and all application to proceed/resume.

Screenshots
N/A

Your Environment

• Version used: 2.0.2 (also tested 1.8.15, 1.9.8)
• Configuration:
  fb-hang.conf
  firelens-datajet.json
• Environment name and version (e.g. Kubernetes? What version?): Docker
• Server type and version: EC2
• Operating System and version:
  x86_64 amzn2-ami-kernel-5.10-hvm-2.0.20221004.0-x86_64-gp2
  ami-0d593311db5abb72b
  arm64 amzn2-ami-kernel-5.10-hvm-2.0.20221004.0-arm64-gp2
  MacOSX - Darwin Kernel Version 21.6.0
  ami-0efabcf945ffd8831
• Filters and plugins:
  tcp input plugin
  cloudwatch_logs output plugin

Additional context
arm64

• backtrace1 - Thread4/6/7 blocked on read(...) with SSL_shutdown(...) higher on the stack
• backtrace2 - Thread5/6 blocked on read(...) with SSL_shutdown(...) higher on the stack, Thread4/Thread7 blocked on getaddrinfo(...)

x86_64

• backtrace1 - Thread4 blocked on read(...) with SSL_shutdown(...) higher on the stack

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.