Setup Metrics Gathering on the New Observability Cluster

[schwesig]

• subtask of duplicate: Install OpenShift, Logging, Metrics & Observability on the new Logging, Metrics & Observability Cluster #177
• Zenhub Epic Setup a new Cluster to handle Metrics, Logging and Observability

timeline: 2023/Q4 - 2024/Q1

right now catching this as an issue for #177

• creating an idea diagram
• How do we manage custom dashboards? How do we manage custom dashboards? #54
• monitoring components
• define possible alerts

[computate]

@schwesig @jbasu01 Thanks @harshil-codes for the suggestion to follow the documentation for configuring an external grafana to query metrics with the Observatorium API. After trying out the right secrets this time, I managed to get a temporary Grafana datasource like this working. This will allow us to set up Grafana on the observability cluster, but point it to the observatorium Route on the infra cluster. There is CA Cert validation and TLS Client auth that will allow the observability cluster to connect to the infra cluster. We just need to put the credentials in Vault. Here is how to configure the GrafanaDatasource:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
  name: observability-metrics
  namespace: grafana
  labels:
    app.kubernetes.io/instance: grafana-infra
spec:
  datasources:
    - access: proxy
      editable: false
      jsonData:
        timeInterval: 5s
        tlsAuth: true
        tlsAuthWithCACert: true
      name: observability-metrics
      secureJsonData:
        tlsClientCert: |-
          # Use `oc --as system:admin -n open-cluster-management-observability get secret/observability-grafana-certs -o jsonpath='{.data.tls\.crt}' | base64 -d`
        tlsClientKey: |-
          # Use `oc --as system:admin -n open-cluster-management-observability get secret/observability-grafana-certs -o jsonpath='{.data.tls\.key}' | base64 -d`
        tlsCACert: |-
          # Use `oc --as system:admin -n open-cluster-management-observability get secret/observability-server-ca-certs -o jsonpath='{.data.ca\.crt}' | base64 -d`
      type: prometheus
      url: >-
        https://observatorium-api-open-cluster-management-observability.apps.nerc-ocp-infra.rc.fas.harvard.edu/api/metrics/v1/default
  name: observability-metrics

[schwesig]

@computate awesome, thanks for trying and creating a poc

[jbasu01]

Thanks @computate for running the poc and sharing the info.

[schwesig]

status: regular meetings @schwesig & @jbasu01 working on the PR

[schwesig]

check this for details about referencing secrets
OCP-on-NERC/nerc-ocp-config#317 (comment)

[schwesig]

split
Personas and use cases in a separate issue