Fluentbit Stops Processing Incoming Events #847
Comments
|
We were able to reproduce this issue by continously posting events, severing the network connection of fluentbit, then restoring the network connection. It seems like a core HTTP client issue. version 2.0+ of fluentbit included some large refactoring of the HTTP facilities via fluent/fluent-bit#5918 If you find yourself in a similar position, you can continue to use the firelens log driver on your app container and use your own fluentbit image to listen to forward port 24224. The fluentd docker log driver will continue to work. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the question/issue
Fluentbit begins consuming a full vCPU endlessly and stops processing inputs. This issue occurs intermittently in production environments. A processing thread is stuck in an epoll_wait loop, causing high CPU utilization and unresponsiveness to new log events.
8-9 hours after the fluentbit container stops processing new events, our main API container starts failing health checks on a simple ready endpoint.
curling one of the http inputs (port 8888, the unused one) below from within the fluentbit container receives a timeout while awaiting a response. (timeout set to 60s).All containers that exhibit this issue have the following log line within a few milliseconds of when they stop processing. Unclear if this is a cause or a symptom, as we see this in healthy container logs as well.
ps -Treveals one thread as the high CPU use.gdbthread dump reveals:Configuration
Fluent Bit Log Output
Last 100 lines (can provide more -- either other containers exhibiting the issue, or further back, if helpful)
Fluent Bit Version Info
amazon/aws-for-fluent-bit:2.32.2
Cluster Details
We're using ECS Fargate with a sidecar deployment.
Relevant portions of ECS task config:
Application Details
The application sends two types of logs over STDOUT through docker's fluentd log-driver (via firelens, not customization on our end here). Most logs are small and fit into a single page, but we have some rogue log statements that contain large blobs of logs. (the fluentd-log-driver splits these into hundreds of pages)
We're also sending telegraf metrics over via the HTTP plugin.
Steps to reproduce issue
Unclear. We have multiple test environments, but are only seeing this in production intermittently.
In our application, the logs that get published to ADX (that get picked up by the rewrite_tag filter) is behind an environment variable. Disabling this greatly reduces the occurrence rate, but does not totally eliminate it.
Similarly, disabling telegraf metrics (but leaving ADX events on) the issue still occurs.
Related Issues
fluent/fluent-bit#1958 Is exhibiting similar symptoms, though resolved many versions ago (1.4.x)
The text was updated successfully, but these errors were encountered: