-
Notifications
You must be signed in to change notification settings - Fork 45
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chart,salt,build: Migrate to fluent-bit fluent chart
Migrate from grafana fluent-bit deprecated helm chart to the fluent-bit helm chart from fluent: ``` rm -rf charts/fluent-bit helm repo add fluent https://fluent.github.io/helm-charts helm repo update helm fetch -d charts --untar fluent/fluent-bit ``` Change image from `fluent-bit-loki-plugin` from grafana repository to 1.8.12 `fluent-bit` from fluent repository as Loki output plugin is built-in. Migrate fluent-bit option file to work with the new fluent-bit helm template. Rewrite fluent-bit output from config as it's not exactly the same between `grafana-loki` and `loki`. We also enable HTTP_Server back as it seems to work properly with this version. Render chart to salt state using: ``` ./charts/render.py fluent-bit --namespace metalk8s-logging \ charts/fluent-bit.yaml charts/fluent-bit/ \ > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls ``` Fixes: #3435
- Loading branch information
1 parent
cd5e889
commit d1974fe
Showing
39 changed files
with
2,560 additions
and
745 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,6 +14,7 @@ | |
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,27 @@ | ||
| annotations: | ||
| artifacthub.io/changes: | | ||
| - kind: changed | ||
| description: "Update fluent-bit image to 1.8.12." | ||
| apiVersion: v1 | ||
| appVersion: v2.1.0 | ||
| description: Uses fluent-bit Loki go plugin for gathering logs and sending them to Loki | ||
| home: https://grafana.com/loki | ||
| icon: https://raw.githubusercontent.com/grafana/loki/master/docs/sources/logo.png | ||
| kubeVersion: ^1.10.0-0 | ||
| appVersion: 1.8.12 | ||
| description: Fast and lightweight log processor and forwarder or Linux, OSX and BSD | ||
| family operating systems. | ||
| home: https://fluentbit.io/ | ||
| icon: https://fluentbit.io/assets/img/logo1-default.png | ||
| keywords: | ||
| - logging | ||
| - fluent-bit | ||
| - fluentd | ||
| maintainers: | ||
| - email: [email protected] | ||
| name: Loki Maintainers | ||
| - email: [email protected] | ||
| name: edsiper | ||
| - email: [email protected] | ||
| name: naseemkullah | ||
| - email: [email protected] | ||
| name: Towmeykaw | ||
| - email: [email protected] | ||
| name: stevehipwell | ||
| name: fluent-bit | ||
| sources: | ||
| - https://github.com/grafana/loki | ||
| version: 2.2.0 | ||
| - https://github.com/fluent/fluent-bit/ | ||
| version: 0.19.19 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,124 +1,57 @@ | ||
| # Fluent Bit Loki chart | ||
| # Fluent Bit Helm chart | ||
|
|
||
| This chart install the Fluent Bit application to ship logs to Loki. It defines daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. | ||
| [Fluent Bit](https://fluentbit.io) is a fast and lightweight log processor and forwarder or Linux, OSX and BSD family operating systems. | ||
|
|
||
| ## Get Repo Info | ||
| ## Installation | ||
|
|
||
| ```console | ||
| helm repo add grafana https://grafana.github.io/helm-charts | ||
| helm repo update | ||
| ``` | ||
|
|
||
| _See [helm repo](https://helm.sh/docs/helm/helm_repo/) for command documentation._ | ||
|
|
||
| ## Installing the Chart | ||
|
|
||
| > If you don't have `Helm` installed locally, or `Tiller` installed in your Kubernetes cluster, read the [Using Helm](https://docs.helm.sh/using_helm/) documentation to get started. | ||
| To install the chart with the release name `my-release` using our helm repository: | ||
|
|
||
| ```bash | ||
| helm repo add grafana https://grafana.github.io/helm-charts | ||
| helm upgrade --install my-release grafana/fluent-bit \ | ||
| --set loki.serviceName=loki.default.svc.cluster.local | ||
| ``` | ||
|
|
||
| If you deploy Loki with a custom namespace or service name, you must change the value above for `loki.serviceName` to the appropriate value. | ||
|
|
||
| The command deploys Fluent Bit on the Kubernetes cluster with the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. | ||
| To add the `fluent` helm repo, run: | ||
|
|
||
| To configure the chart to send to [Grafana Cloud](https://grafana.com/products/cloud) use: | ||
|
|
||
| ```bash | ||
| helm upgrade --install my-release grafana/fluent-bit \ | ||
| --set loki.serviceName=logs-us-west1.grafana.net,loki.servicePort=80,loki.serviceScheme=https \ | ||
| --set loki.user=2830,loki.password=1234 | ||
| ```sh | ||
| helm repo add fluent https://fluent.github.io/helm-charts | ||
| ``` | ||
|
|
||
| > **Tip**: List all releases using `helm list` | ||
| To install a custom tag use the following command: | ||
| To install a release named `fluent-bit`, run: | ||
|
|
||
| ```bash | ||
| helm upgrade --install my-release grafana/fluent-bit \ | ||
| --set image.tag=<custom tag> | ||
| ```sh | ||
| helm install fluent-bit fluent/fluent-bit | ||
| ``` | ||
|
|
||
| The full list of available tags on [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki). | ||
| ## Chart values | ||
|
|
||
| Alternatively you can install the full [Loki stack](../loki-stack) (Loki + Fluent Bit) using: | ||
|
|
||
| ```bash | ||
| helm upgrade --install my-release grafana/loki-stack \ | ||
| --set fluent-bit.enabled=true,promtail.enabled=false | ||
| ```sh | ||
| helm show values fluent/fluent-bit | ||
| ``` | ||
|
|
||
| This will automatically configured the `loki.serviceName` configuration field to the newly created Loki instance. | ||
|
|
||
| ## RBAC | ||
|
|
||
| By default, `rbac.create` is set to true. This enable RBAC support in Fluent Bit and must be true if RBAC is enabled in your cluster. | ||
|
|
||
| The chart will take care of creating the required service accounts and roles for Fluent Bit. | ||
|
|
||
| If you have RBAC disabled, or to put it another way, ABAC enabled, you should set this value to `false`. | ||
| ## Using Lua scripts | ||
| Fluent Bit allows us to build filter to modify the incoming records using custom [Lua scripts.](https://docs.fluentbit.io/manual/pipeline/filters/lua) | ||
|
|
||
| ## Uninstalling the Chart | ||
| ### How to use Lua scripts with this Chart | ||
|
|
||
| To uninstall/delete the `my-release` deployment: | ||
| First, you should add your Lua scripts to `luaScripts` in values.yaml, for example: | ||
|
|
||
| ```bash | ||
| helm delete my-release | ||
| ```yaml | ||
| luaScripts: | ||
| filter_example.lua: | | ||
| function filter_name(tag, timestamp, record) | ||
| -- put your lua code here. | ||
| end | ||
| ``` | ||
| The command removes all the Kubernetes components associated with the chart and deletes the release. | ||
|
|
||
| ## Configuration | ||
|
|
||
| The following tables lists the configurable parameters of the Fluent Bit chart and their default values. | ||
|
|
||
| For more details, read the [Fluent Bit documentation](../../../cmd/fluent-bit/README.md) | ||
|
|
||
| | Parameter | Description | Default | | ||
| |--------------------------|----------------------------------------------------------------------------------------------------|----------------------------------| | ||
| | `loki.serviceName` | The address of the Loki service. | `"${RELEASE}-loki"` | | ||
| | `loki.servicePort` | The port of the Loki service. | `3100` | | ||
| | `loki.serviceScheme` | The scheme of the Loki service. | `http` | | ||
| | `loki.user` | The http basic auth username to access the Loki service. | | | ||
| | `loki.password` | The http basic auth password to access the Loki service. | | | ||
| | `config.port` | the Fluent Bit port to listen. (This is mainly used to serve metrics) | `2020` | | ||
| | `config.tenantID` | The tenantID used by default to push logs to Loki | `''` | | ||
| | `config.batchWait` | Time to wait before send a log batch to Loki, full or not. (unit: secs) | `1` | | ||
| | `config.batchSize` | Log batch size to send a log batch to Loki. (unit: bytes) | `10240` (10KiB) | | ||
| | `config.loglevel` | the Fluent Bit log level (debug,info,warn,error). | `warn` | | ||
| | `config.lineFormat` | The line format to use to send a record (json/key_value) | `json` | | ||
| | `config.k8sLoggingParser`| Allow Kubernetes Pods to suggest a pre-defined Parser. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/filter/kubernetes#kubernetes-annotations). | `Off` | | ||
| | `config.k8sLoggingExclude`| Allow Kubernetes Pods to exclude their logs from the log processor. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/pipeline/filters/kubernetes) | `Off` | ||
| | `config.memBufLimit` | Override the default Mem_Buf_Limit [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/administration/backpressure#mem_buf_limit) | `5MB` | ||
| | `config.removeKeys` | The list of key to remove from each record | `[removeKeys,stream]` | | ||
| | `config.labels` | A set of labels to send for every log | `'{job="fluent-bit"}'` | | ||
| | `config.autoKubernetesLabels` | If set to true, it will add all Kubernetes labels to Loki labels | `false` | | ||
| | `config.labelMap` | Mapping of labels from a record. See [Fluent Bit documentation](../../../cmd/fluent-bit/README.md) | | | ||
| | `config.parsers` | Definition of extras fluent bit parsers. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/filter/parser). The format is a sequence of mappings where each key is the same as the one in the [PARSER] section of parsers.conf file | `[]` | | ||
| | `config.extraOutputs` | Definition of extras fluent bit outputs. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/pipeline/outputs/). The format is a sequence of mappings where each key is the same as the one in the [OUTPUT] | `[]` | | ||
| | `affinity` | [affinity][affinity] settings for pod assignment | `{}` | | ||
| | `annotations` | Annotations to add to Kubernetes resources. | `{}` | | ||
| | `deploymentStrategy` | The deployment strategy to use with the daemonset | `RollingUpdate` | | ||
| | `image.repository` | The Fluent Bit docker image repository | `grafana/fluent-bit-plugin-loki` | | ||
| | `image.tag` | The Fluent Bit docker image tag | `0.1` | | ||
| | `image.pullPolicy` | The Fluent Bit docker image pull policy | `IfNotPresent` | | ||
| | `nodeSelector` | Fluent Bit [node labels][nodeSelector] for pod assignment | `{}` | | ||
| | `podLabels` | additional Fluent Bit pod labels | `{}` | | ||
| | `podAnnotations` | additional Fluent Bit pod annotations | `Prometheus discovery` | | ||
| | `rbac.create` | Activate support for RBAC | `true` | | ||
| | `resources` | Resource requests/limit | | | ||
| | `tolerations` | [Toleration][toleration] labels for pod assignment | `no schedule on master nodes` | | ||
| | `volumes` | [Volume]([volumes]) to mount | `host containers log` | | ||
| | `volumeMounts` | Volume mount mapping | | | ||
| | `serviceMonitor.enabled` | Create a [Prometheus Operator](operator) serviceMonitor resource for Fluent Bit | `false` | | ||
| After that, the Lua scripts will be ready to be used as filters. So next step is to add your Fluent bit [filter](https://docs.fluentbit.io/manual/concepts/data-pipeline/filter) to `config.filters` in values.yaml, for example: | ||
|
|
||
| ```yaml | ||
| config: | ||
| filters: | | ||
| [FILTER] | ||
| Name lua | ||
| Match <your-tag> | ||
| script /fluent-bit/scripts/filter_example.lua | ||
| call filter_name | ||
| ``` | ||
| Under the hood, the chart will: | ||
| - Create a configmap using `luaScripts`. | ||
| - Add a volumeMounts for each Lua scripts using the path `/fluent-bit/scripts/<script>`. | ||
| - Add the Lua script's configmap as volume to the pod. | ||
|
|
||
| [toleration]: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ | ||
| [nodeSelector]: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector | ||
| [affinity]: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity | ||
| [volumes]: https://kubernetes.io/docs/concepts/storage/volumes/ | ||
| [operator]: https://github.com/coreos/prometheus-operator | ||
| ### Note | ||
| Remember to set the `script` attribute in the filter using `/fluent-bit/scripts/`, otherwise the file will not be found by fluent bit. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| logLevel: debug |
Oops, something went wrong.