Feature: Observability Stack

[omrishiv]

Have you searched for this feature request?

• I searched but did not find similar requests

Problem Statement

Currently, we do not have any observability components to check on metrics, logs, cost. We also have no ability to visualize or alert on them.

Possible Solution

I propose we add a stack that includes:

• Prometheus
• Loki
• Grafana
• Alertmanager
• OpenCost

These components will set the foundation for observability across CNOE components. They'll be able to monitor the core ones as well as any others added to the environment, including non-CNOE components.

We can have Grafana leverage keycloak and use the credentials there. If/when a service mesh becomes available, we can add Tempo to the stack for tracing observability.

Alternatives Considered

No response

[omrishiv]

I opened this as a tracking issue; I am already working on this but wanted to put it out there for visibility