Create advanced configuration for observability cluster

[schwesig]

replacing CCI-MOC/ops-issues#1196 (wrong repo)
follow up to #280

• auth
• certificates
• secrets
• etc

will set basics for and support:

• Attach observability cluster to argocd #309
• Provide Redhat contributors with admin access to nerc-ocp-obs cluster #308
• Setup Logging on the New Observability Cluster 2/2 #324
• Setup Observability on the New Observability Cluster #327
• Setup Metrics Gathering on the New Observability Cluster #326
• Create new metrics and logs backing stores with better volumes configuration #151

[schwesig]

reinstalling (thanks @larsks) the 3 workers and 3 controllers. some of them were stucked in an unhealthy state,
also dealing with switch, nic and power outages in the last days.
starting over again to connect them to the cluster,

[schwesig]

closed in favor of adding more details into this #280