Fluentbit loading unwanted index patterns

[dfishburn]

2 issues with kubernetes-logging-helm/chart/templates/fluent-bit/fluent-bit-config-sec.yaml.

• I have created a PR for this change, Check if conditions against a true boolean. Only OUTPUT if needed dfishburn/kubernetes-logging-helm-1#1

First issue:

    {{- if .Values.kafka.enabled }}

Only tests if the variable exists, not whether it is true or false.
This was showing up when I ran:

helm upgrade ofd -n os-logging --create-namespace -f ./single-node-setup.yaml ./../chart --install --wait-for-jobs --timeout=30m --debug --dry-run

You will see the @INCLUDE _output-kafka.conf, even though I had defined:

kafka:
  enabled: false

Updating the if to validate the variable is set to true resolves this.

    {{- if eq .Values.kafka.enabled true }}

2nd issue

The default indexPatterns is set to:

opensearch_dashboards:
  indexPatterns:
    - containers
    - systemd
    - nginx

In my case, the containers where chewing up all available space on my system.
I had already created a custom setup so that my logs were indexed under a separate pattern.
Therefore, I was not interested in the containers at all.
Removed - containers, unfortunately, the OUTPUT is still created and logged.

This change will prevent the creation of the [OUTPUT] if the deployment has not requested that index.

It was put into the file using this syntax:

  _output-opensearch-containers.conf: |-
  {{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}
    [OUTPUT]
        Name                opensearch
        Match               kube.*
        Alias               containers
        Host                ${FLUENT_opensearch_HOST}
        Port                ${FLUENT_opensearch_PORT}
        HTTP_User           ${FLUENT_opensearch_USER}
        HTTP_Passwd         ${FLUENT_opensearch_PASSWORD}
        Logstash_Format     On
        {{- if .Values.fluentbit.indexPrefix }}
        Logstash_Prefix     {{ printf "%s-containers" .Values.fluentbit.indexPrefix }}
        {{- else }}
        Logstash_Prefix     containers
        {{- end }}
...
  {{ end }}

I had originally set it up as:

  {{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}
  _output-opensearch-containers.conf: |-
    [OUTPUT]
        Name                opensearch
        Match               kube.*
        Alias               containers
        Host                ${FLUENT_opensearch_HOST}
        Port                ${FLUENT_opensearch_PORT}
        HTTP_User           ${FLUENT_opensearch_USER}
        HTTP_Passwd         ${FLUENT_opensearch_PASSWORD}
        Logstash_Format     On
        {{- if .Values.fluentbit.indexPrefix }}
        Logstash_Prefix     {{ printf "%s-containers" .Values.fluentbit.indexPrefix }}
        {{- else }}
        Logstash_Prefix     containers
        {{- end }}
...
  {{ end }}

But later on in this file we were issuing:

    @INCLUDE _output-opensearch-*.conf

And if we had so happened to turn off BOTH the systemd and containers then this line would fail as there would be no files which would meet the wildcard.

opensearch_dashboards:
  indexPatterns:
    - nginx

Leaving it as I have has worked for me.

[nickytd]

Good catch @dfishburn Can pls you create e PR against the main branch in this repo? I will merge it. Thanks

[dfishburn]

@nickytd the PR is at the top of the issue, did I do it correctly?

[nickytd]

The correct handling of the enabled kafka field is fixed with 0aa4bfd

The opensearch_dashboards.indexPattern is related to creating index patterns in the UI and shall not be used to control what indices shall be created in the opensearch data nodes. The containers index is default and shall not be removed. But that doesn't mean that all workloads outputs shall be targeted there. If the requirement is to create a dedicated indices for particular workloads then nginx configs demonstrates an implementation strategy.

Here is a pattern that demonstrates how to redirect the logs from a specific workload to its own index in fluent-bit configuration

1. Create a filter catching the workload logs and set a new tag. In this example we use kubernetes labels to identify nginx ingress controller logs and tag them with nginx
2. (Optional) Create a parser to transform the logs from lines to structured logs
3. Create a fluent-bit output containing the dedicated index prefix name.

Follow the naming convention for the files using the filter-, parser-, output- prefixes and place those in the fluent-bit-configs folder to be picked up by helm. The secret template for the fluent-bit configuration respects the location and pickups the files with the respective naming convention, adding them to the correct places in the fluent-bit configuration.

[dfishburn]

@nickytd the changes you checked in did not include these changes:

  _output-opensearch-containers.conf: |-
  {{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}

And the same for systemd.
Did you do it a different way?
Without the change(s), all the data is still indexed and does not fix my original issue.