feat: do not send Fluent Bit application logs to CloudWatch TDE-1016 #374
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paulfouquet
requested review from
l0b0,
blacha,
amfage,
MDavidson17 and
Wentao-Kuang
amfage
approved these changes
Jan 3, 2024
Wentao-Kuang
approved these changes
Jan 3, 2024
3 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 3, 2024
#### Motivation Fluent Bit is experiencing a lot of network errors connecting to `logs.ap-southeast-2.amazonaws.com`. This amount of errors does increase the log storage cost, see #374. This is a known issue for which Fluent Bit team made [some recommendations to reduce it](aws/aws-for-fluent-bit#340). This PR is applying one of these recommendations and has been tested with success on non prod. #### Modification - Remove [the patch](#374) that stops sending Fluent Bit application logs to CloudWatch - Set the Fluent Bit `keepalive idle timeout` to 4s (default is 1.5s) following [the recommendations made here](aws/aws-for-fluent-bit#340). #### Checklist - [ ] Tests updated - N/A - [x] Docs updated - [x] Issue linked in Title --------- Co-authored-by: Victor Engmark <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Due to some managed network issues dealt by Fluent Bit application pods, the amount of logs increased dramatically which has a considerable impact on our cloud storage cost. A solution (that could be temporary until this network errors stop/slow down if we find a way to fix them?) is to stop sending the Fluent Bit application pod logs to CloudWatch.
Modification
Telling Fluent Bit to ignore its own logs using the exclude annotation.
Consequences are that these logs won't be shipped through our system (not being seen in Elasticsearch). We are still able to view them from the EKS Cluster that we have hands on.
Checklist