Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fluentbit loading unwanted index patterns #28

Open
dfishburn opened this issue May 2, 2023 · 0 comments
Open

Fluentbit loading unwanted index patterns #28

dfishburn opened this issue May 2, 2023 · 0 comments

Comments

@dfishburn
Copy link

dfishburn commented May 2, 2023

@nickytd your changes to #27, included some changes from my PR, but did not include these changes:

  _output-opensearch-containers.conf: |-
  {{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}

And the same for systemd.
Did you do it a different way?
Without the change(s), all the data is still indexed and does not fix my original issue.

  • The Kibana fix which you did include was just a bonus

You made a few comments in the issue, but I am left with the same fact.
OpenSearch is eating all my space with data I do not want.

I used all the customization pieces you did mention:

Here is a pattern that demonstrates how to redirect the logs from a specific workload to its own index in fluent-bit configuration

Create a [filter](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/filter-nginx.conf) catching the workload logs and set a new tag. In this example we use kubernetes labels to identify nginx ingress controller logs and tag them with nginx
(Optional) Create a [parser](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/parser-nginx.conf) to transform the logs from lines to structured logs
Create a fluent-bit [output](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/output-nginx.conf) containing the dedicated index prefix name.

If you didn't like way I had turned it off, we just need another mechanism.
Do you have a proposal that I could look into to implement?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant