salt,charts: Grafana sidecar picks up dashboards in all namespaces

Configure k8s-sidecar in Grafana deployment:

* to search in `ALL` namespaces
* to place dashboards in folders according to
  `metalk8s.scality.com/grafana-folder-name` annotation
* to create Grafana folders from dashboard directory structure

Re-render chart to salt state using

```
./charts/render.py prometheus-operator \
  charts/kube-prometheus-stack.yaml \
  charts/kube-prometheus-stack/ \
  --namespace metalk8s-monitoring \
  --service-config grafana \
  metalk8s-grafana-config \
  metalk8s/addons/prometheus-operator/config/grafana.yaml \
  metalk8s-monitoring \
  --service-config prometheus \
  metalk8s-prometheus-config \
  metalk8s/addons/prometheus-operator/config/prometheus.yaml \
  metalk8s-monitoring \
  --service-config alertmanager \
  metalk8s-alertmanager-config \
  metalk8s/addons/prometheus-operator/config/alertmanager.yaml \
  metalk8s-monitoring \
  --service-config dex \
  metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  --drop-prometheus-rules charts/drop-prometheus-rules.yaml \
  --patch 'PrometheusRule,metalk8s-monitoring,prometheus-operator-kubernetes-system-kubelet,spec:groups:0:rules:1:for,"5m"' \
  > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls
```

Fixes: #3618

CHANGELOG.md

@@ -69,6 +69,11 @@
   highly-available when we have multiple Prometheus instances
   (PR[#3573](https://github.com/scality/metalk8s/pull/3573))
 
+- [#3618](https://github.com/scality/metalk8s/issues/3618) Detect Grafana
+  dashboard ConfigMaps in any namespace rather than just `metalk8s-monitoring`,
+  and enable Grafana folder generation from dashboard file structure (PR
+  [#3620](https://github.com/scality/metalk8s/pull/3620/commits))
+
 ## Bug fixes
 
 - [#3601](https://github.com/scality/metalk8s/issues/3601) - Marks

charts/kube-prometheus-stack.yaml

@@ -155,6 +155,12 @@ grafana:
     image:
       repository: '__image__(k8s-sidecar)'
 
+    dashboards:
+      searchNamespace: ALL
+      folderAnnotation: metalk8s.scality.com/grafana-folder-name
+      provider:
+        foldersFromFilesStructure: true
+
     datasources:
       # Service deployed by Thanos
       url: http://thanos-query-http:10902/

docs/developer/architecture/logs.rst

@@ -704,7 +704,14 @@ An example of what we want is `Loki dashboard`_.
 
   The ``grafana_datasource: "1"`` and ``grafana_dashboard: "1"`` labels are
   what is used by the Prometheus Operator to retrieve datasource and dashboard
-  for Grafana, resources must be deployed in `metalk8s-monitoring` namespace.
+  for Grafana.
+
+  Resources may be deployed in any namespace so long as it contains the above
+  labels.
+
+  The `metalk8s.scality.com/grafana-folder-name` annotation on dashboard
+  resources provide control over the folder in which the dashboard is placed in
+  Grafana.
 
 Loki Volume Purge
 ~~~~~~~~~~~~~~~~~

salt/metalk8s/addons/prometheus-operator/deployed/chart.sls

@@ -20796,13 +20796,12 @@ data:
     providers:
     - name: 'sidecarProvider'
       orgId: 1
-      folder: ''
       type: file
       disableDeletion: false
       allowUiUpdates: false
       updateIntervalSeconds: 30
       options:
-        foldersFromFilesStructure: false
+        foldersFromFilesStructure: true
         path: /tmp/dashboards
 kind: ConfigMap
 metadata:
@@ -57480,7 +57479,7 @@ spec:
           apiVersion="v1", namespace="metalk8s-monitoring", name="prometheus-operator-grafana",
           path="data:grafana.ini")
         checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
-        checksum/sc-dashboard-provider-config: 1fb938ae203ab04abaec138541fafa99df99f419c7c1567c8573f1cb0e9a487c
+        checksum/sc-dashboard-provider-config: 015250c97b5d357b599e212313ae96228466291847f8918b64b2a2ade947b8a7
         checksum/secret: 81a974a3dea4b80cdc27cccd08b7648f4175ec54526ce15faffd35b6535e0967
       labels:
         app.kubernetes.io/instance: prometheus-operator
@@ -57496,6 +57495,10 @@ spec:
           value: /tmp/dashboards
         - name: RESOURCE
           value: both
+        - name: NAMESPACE
+          value: ALL
+        - name: FOLDER_ANNOTATION
+          value: metalk8s.scality.com/grafana-folder-name
         image: {% endraw -%}{{ build_image_name("k8s-sidecar", False) }}{%- raw %}:1.12.2
         imagePullPolicy: IfNotPresent
         name: grafana-sc-dashboard