charts,build,salt: Add fluent-bit pod memory limits to 1Gi

Add a fluent-bit pod memory limits to 1Gi and make it configurable

Re-render fluent-bit salt state using:
```
./charts/render.py fluent-bit \
  charts/fluent-bit.yaml charts/fluent-bit/ \
  --namespace metalk8s-logging \
  --service-config fluent-bit metalk8s-fluent-bit-config \
  metalk8s/addons/logging/fluent-bit/config/fluent-bit.yaml metalk8s-logging \
  > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls
```

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## Release 123.0.3 (in development)
 
+### Enhancements
+
+- Add default fluent-bit pod memory limits to 1Gi and add
+  ability to change the fluent-bit pod resources request
+  and limits
+  (PR[#3845](https://github.com/scality/metalk8s/pull/3845))
+
 ## Release 123.0.2
 
 ### Bug fixes

buildchain/buildchain/salt_tree.py

@@ -328,11 +328,13 @@ def task(self) -> types.TaskDict:
     Path("salt/metalk8s/addons/logging/deployed/files/fluent-bit.json"),
     Path("salt/metalk8s/addons/logging/deployed/files/logs.json"),
     Path("salt/metalk8s/addons/logging/deployed/files/loki.json"),
+    Path("salt/metalk8s/addons/logging/fluent-bit/config/fluent-bit.yaml"),
     Path("salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls"),
     Path("salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls"),
     Path("salt/metalk8s/addons/logging/fluent-bit/deployed/init.sls"),
     Path("salt/metalk8s/addons/logging/fluent-bit/deployed/post-upgrade.sls"),
     Path("salt/metalk8s/addons/logging/fluent-bit/deployed/post-downgrade.sls"),
+    Path("salt/metalk8s/addons/logging/fluent-bit/deployed/service-configuration.sls"),
     Path("salt/metalk8s/addons/logging/loki/config/loki.yaml"),
     Path("salt/metalk8s/addons/logging/loki/deployed/chart.sls"),
     Path("salt/metalk8s/addons/logging/loki/deployed/datasource.sls"),

charts/fluent-bit.yaml

@@ -9,10 +9,7 @@ podAnnotations:
   prometheus.io/port: "2020"
   prometheus.io/path: /api/v1/metrics/prometheus
 
-resources:
-  requests:
-    cpu: 100m
-    memory: 200Mi
+resources: '__var__(fluent-bit.spec.deployment.resources)'
 
 tolerations:
 - key: node-role.kubernetes.io/bootstrap

docs/operation/cluster_and_service_configuration.rst

@@ -87,6 +87,19 @@ The default configuration values for Loki are specified below:
    :lines: 3-
 
 
+Fluent-bit Default Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Fluent-bit is a logs collectors system, its job is to retrieve local logs to
+forward them to a log aggregation system (loki).
+
+The default configuration values for Fluent-bit are specified below:
+
+.. literalinclude:: ../../salt/metalk8s/addons/logging/fluent-bit/config/fluent-bit.yaml
+   :language: yaml
+   :lines: 3-
+
+
 UI Default Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -747,6 +760,34 @@ edited as follows:
    Due to internal implementation, ``retention_period`` must be a multiple of
    ``24h`` in order to get the expected behavior
 
+Fluent-bit Configuration Customization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Add Fluent-bit memory limit
+"""""""""""""""""""""""""""
+
+Fluent-bit consumes some memory to store logs input before processing them
+and logs chunks before sending them to Loki.
+Its memory consumption really depends on the usage, which is why you may want
+to change it.
+
+For example, to set the limit to 4 GiB, the ConfigMap must be
+edited as follows:
+
+.. code-block:: yaml
+
+    apiVersion: v1
+    kind: ConfigMap
+    data:
+      config.yaml: |-
+        apiVersion: addons.metalk8s.scality.com
+        kind: FluentBitConfig
+        spec:
+          deployment:
+            resources:
+              limits:
+                memory: "4Gi"
+
 .. _csc-ui-customization:
 
 Metalk8s UI Configuration Customization

salt/metalk8s/addons/logging/fluent-bit/config/fluent-bit.yaml

@@ -0,0 +1,13 @@
+#!yaml
+
+# Configuration of the fluent-bit service
+apiVersion: addons.metalk8s.scality.com
+kind: FluentBitConfig
+spec:
+  deployment:
+    resources:
+      requests:
+        cpu: 100m
+        memory: 200Mi
+      limits:
+        memory: 1Gi

salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

@@ -1,8 +1,8 @@
 #!jinja | metalk8s_kubernetes
 
 {%- from "metalk8s/repo/macro.sls" import build_image_name with context %}
-
-
+{%- set fluent-bit_defaults = salt.slsutil.renderer('salt://metalk8s/addons/logging/fluent-bit/config/fluent-bit.yaml', saltenv=saltenv) %}
+{%- set fluent-bit = salt.metalk8s_service_configuration.get_service_conf('metalk8s-logging', 'metalk8s-fluent-bit-config', fluent-bit_defaults) %}
 
 {% raw %}
 
@@ -136,10 +136,7 @@ spec:
           httpGet:
             path: /api/v1/health
             port: http
-        resources:
-          requests:
-            cpu: 100m
-            memory: 200Mi
+        resources: {% endraw -%}{{ fluent-bit.spec.deployment.resources }}{%- raw %}
         volumeMounts:
         - mountPath: /fluent-bit/etc/fluent-bit.conf
           name: config

salt/metalk8s/addons/logging/fluent-bit/deployed/init.sls

@@ -1,5 +1,6 @@
 include:
   - .configmap
+  - .service-configuration
   - .chart
 
 {#- Due to a change of fluent-bit DaemonSet labelSelector in 123.0, which is immutable field

salt/metalk8s/addons/logging/fluent-bit/deployed/service-configuration.sls

@@ -0,0 +1,33 @@
+include:
+  - ...deployed.namespace
+
+{%- set fluent-bit_config = salt.metalk8s_kubernetes.get_object(
+        kind='ConfigMap',
+        apiVersion='v1',
+        namespace='metalk8s-logging',
+        name='metalk8s-fluent-bit-config',
+    )
+%}
+
+{%- if fluent-bit_config is none %}
+
+Create metalk8s-fluent-bit-config ConfigMap:
+  metalk8s_kubernetes.object_present:
+    - manifest:
+        apiVersion: v1
+        kind: ConfigMap
+        metadata:
+          name: metalk8s-fluent-bit-config
+          namespace: metalk8s-logging
+        data:
+          config.yaml: |-
+            apiVersion: addons.metalk8s.scality.com
+            kind: FluentBitConfig
+            spec: {}
+
+{%- else %}
+
+metalk8s-fluent-bit-config ConfigMap already exists:
+  test.succeed_without_changes: []
+
+{%- endif %}

salt/metalk8s/service-configuration/deployed/init.sls

@@ -11,4 +11,5 @@ include:
   - metalk8s.addons.dex.deployed.service-configuration
 {%- endif %}
   - metalk8s.addons.logging.loki.deployed.service-configuration
+  - metalk8s.addons.logging.fluent-bit.deployed.service-configuration
   - metalk8s.addons.ui.deployed.ui-configuration