Support ServiceMonitor CRD

[benoittoulme]

Prometheus uses ServiceMonitor CRDs to discover the endpoints to call to pull metrics.

It looks like otel also supports ServiceMonitor CRDs:
open-telemetry/opentelemetry-operator#653
open-telemetry/opentelemetry-operator#383

This support was added in release v0.61.0 https://github.com/open-telemetry/opentelemetry-operator/releases/tag/v0.61.0

Ask: Have the Splunk Otel Collector support ServiceMonitor CRDs. This is already supported in otel, so hopefully not too much work. It would make it easier for users to use the same standards to setup metrics ingest into Splunk; And the ServiceMonitor CRD contains additional features that the current annotation-based model does not support - for example, use authentication when executing requests to pull metrics.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[conrad784]

ServiceMonitors are very usefull as the normal prometheus annotation does not work if the monitored endpoint is behind authentication. Also a lot of helm charts only implement ServiceMonitor.

[atoulme]

Moving to a Splunk Idea: https://ideas.splunk.com/ideas/SFXIMMID-I-571

Please add your votes there to get this prioritized.

[atoulme]

Work continues on the Splunk idea.

[MB175]

@benoittoulme so how to I configure prometheus in the Otel config so it recognises CRDs (ServiceMonitors) ?

[mpepping]

so how to I configure prometheus in the Otel config so it recognises CRDs (ServiceMonitors) ?

@MB175 this should help: https://github.com/signalfx/splunk-otel-collector-chart/blob/splunk-otel-collector-0.113.0/helm-charts/splunk-otel-collector/values.yaml#L1278-L1280