[WIP] feat: add Observability Remote Write Proxy deployment templates

[miguelsorianod]

Description

Related to https://issues.redhat.com/browse/MGDSTRM-11146.

This PR adds the OCP templates needed to deploy Observability Remote Write Proxy.

The aim of Observability Remote Write Proxy in COS Fleet Manager context is for the related Prometheus metrics in the Data Planes to be sent to Observatorium indirectly through the proxy.
The motivation of sending the credentials indirectly through the proxy is to avoid needing to store Observatorium credentials in the Data Planes themselves.

This PR adds the following OCP templates:

• templates/observability-remote-write-proxy-oidc-secret.yml: Contains the needed K8s secret when OIDC token retrieval is enabled in Observability Remote Write Proxy.
• templates/observability-remote-write-proxy-route.yml: Contains an OCP Route used to access the Observability Remote Write Proxy API. This route is intended for development purposes only.
• templates/observability-remote-write-proxy.yml: Contains the Observability Remote Write Proxy K8s Deployment and related K8s objects like K8s Service, PodDisruptionBudget, ...

The templates have been intentionally been implemented separatedly from templates/service-template.yml, templates/secret-template.yml and templates/route-template.yml. The reason for that is that the Observability Remote Write Proxy is considered independent of the COS Fleet Manager deployment and its related elements themselves.

Aside from the templates, other changes made have been:

• Add Makefile targets to allow deploying Observability Remote Write Proxy more easily during development
• Add documentation around deploying Observability Remote Write Proxy in OpenShift

Once this is merged some additional tasks will be needed on AppInterface side to be able to use Observability Remote Write Proxy in the stage and production environments. This includes the creation of additional OCP routes dedicated to it

Verification Steps

Deploy Observability Remote Write Proxy and configure COS Fleet Manager to use it.
To do that:

1. Deploy Observability Remote Write Proxy in OCP. Read the documentation of this PR on how to do it. Additionally, temporarily edit the OCP template to use the 3d31c44 container image tag to get the latest functionality. Observatorium Stage can be used when setting the Observatorium related information. If there are any doubts about what values to set on some steps please let me know. Be careful with the metric name and labels being sent in that case as the metrics namespacing is shared and you don't want unintentional overwritting of metrics
2. Configure COS Fleet Manager to use Observability Remote Write Proxy. To do so, run COS Fleet Manager with the --observability-red-hat-sso-observatorium-gateway flag set to the URL provided by the OCP Route created for Observability Remote Write Proxy. The URL scheme has to be included as part of the URL. An Observability Operator version that includes the latest functionality that allows it to use Observability Remote Write Proxy has to be used
3. Verify that metrics are being sent to Observatorium through Observability Remote Write Proxy

Checklist (Definition of Done)

• All acceptance criteria specified in JIRA have been completed
• Unit and integration tests added that prove the fix is effective or the feature works (tested against emulated and non-emulated OCM environment)
• Documentation added for the feature
• CI and all relevant tests are passing
• Code Review completed
• Verified independently by reviewer
• Required metrics/dashboards/alerts have been added (or PR created).
• Required Standard Operating Procedure (SOP) is added.
• JIRA has created for changes required on the client side

[miguelsorianod]

cc @lgarciaaco we can test this changes to the dev environment you mentioned and if/when they look ok we can merge

[machi1990]

@miguelsorianod @pb82 maybe we could reconsider to move and centralize the OpenShift deployments from a service specific repo onto the https://github.com/bf2fc6cc711aee1a0c2a/observability-remote-write-proxy repo in the long run?

[miguelsorianod]

This secret is only used for development purposes.
For stage and production environment different secrets will be created and configured

[miguelsorianod]

This route is only used for development purposes.
For our stage and production environment different routes will be created and configured

[miguelsorianod]

The addition of a separate route than the one used by CFM is intentional. Details around the reason and motivations why this has been done for KFM are available in the https://issues.redhat.com/browse/MGDSTRM-10333 Jira issue

[miguelsorianod]

Hi @machi1990

@miguelsorianod @pb82 maybe we could reconsider to move and centralize the OpenShift deployments from a service specific repo onto the https://github.com/bf2fc6cc711aee1a0c2a/observability-remote-write-proxy repo in the long run?

This has been precisely intentionally duplicated to avoid sharing the template. Sharing the template adds coupling and wanted to keep the autonomy of changes. This is also referenced from app-interface, etc.

[miguelsorianod]

Hi @machi1990

@miguelsorianod @pb82 maybe we could reconsider to move and centralize the OpenShift deployments from a service specific repo onto the https://github.com/bf2fc6cc711aee1a0c2a/observability-remote-write-proxy repo in the long run?

This has been precisely intentionally duplicated to avoid sharing the template. Sharing the template adds coupling and wanted to keep the autonomy of changes.

However, if we want to provide a base deployment on the obs rw proxy repo itself as a sample deployment at some point in time that could be useful too.

[machi1990]

Hi @machi1990

@miguelsorianod @pb82 maybe we could reconsider to move and centralize the OpenShift deployments from a service specific repo onto the https://github.com/bf2fc6cc711aee1a0c2a/observability-remote-write-proxy repo in the long run?

This has been precisely intentionally duplicated to avoid sharing the template. Sharing the template adds coupling and wanted to keep the autonomy of changes.

Thanks for the reply. I understand the motive. I've a strong feeling that in the long run most services using the proxy will be wanting the same deployments files - it mainly the configuration that will change. Two duplication is mostly fine, so let's see how it goes in the long run.

[miguelsorianod]

Thanks for the reply. I understand the motive. I've a strong feeling that in the long run most services using the proxy will be wanting the same deployments files - it mainly the configuration that will change. Two duplication is mostly fine, so let's see how it goes in the long run.

If Obs Rw proxy ends up being used considerably among services I would go even further which would be providing it as a service on app-interface, and consumers of it wouldn't need to worry about an ocp deployment template of it, just about defining an "obs rw proxy resource" with the attributes it would offer configured to their needs.