Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: do not send Fluent Bit application logs to CloudWatch TDE-1016 #374

Merged
merged 1 commit into from
Jan 3, 2024
Merged

feat: do not send Fluent Bit application logs to CloudWatch TDE-1016 #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/infrastructure/components/fluentbit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ The Fluent Bit application version is stored in `appVersion` but this is only he

### `broken connection to logs.ap-southeast-2.amazonaws.com:443`

We can see this error happening from time to time. It is OK as long as the connection retry succeed:
We can see this error happening a lot. It is OK as long as the connection retry succeed:

```console
[2023/12/19 11:31:00] [ warn] [engine] failed to flush chunk [...] retry in 10 seconds: task_id=0, [...]
Expand All @@ -91,3 +91,5 @@ We can see this error happening from time to time. It is OK as long as the conne
However, this issue could potentially cause [a delay for the log](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#log-delay) to come into CloudWatch (the time to retry).

If the retry fails, that could mean logs being lost. In that case it would need investigation. [More information here](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#how-do-i-tell-if-fluent-bit-is-losing-logs).

> **_NOTE:_** One of the consequences of this error is that it increases considerably the amount of the Fluent Bit application pods logs. We had to exclude these logs from being sent to CloudWatch to avoid an increase of our AWS S3 storage cost (as CloudWatch logs are shipped to AWS S3 in our system).
5 changes: 5 additions & 0 deletions infra/charts/fluentbit.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,11 @@ HC_Period 5
{ key: 'karpenter.sh/capacity-type', operator: 'Equal', value: 'spot', effect: 'NoSchedule' },
{ key: 'kubernetes.io/arch', operator: 'Equal', value: 'arm64', effect: 'NoSchedule' },
],
/* To reduce the log volume being sent to CloudWatch (shipped to AWS s3 => storage cost),
tells Fluent Bit to not send the logs from the Fluent Bit application pods.
The Fluent Bit application pods have some (a lot!) network errors that are being logged.
*/
annotations: { 'fluentbit.io/exclude': 'true' },
},
});
}
Expand Down