[target-allocator] Allow the target allocator to use a label selector for Prometheus CRDs

[jaronoff97]

Right now the target allocator pulls in every prometheus CRD (servicemonitor, podmonitor, etc.) in a cluster. This may not be desired for users that want to split up scraping among pools of collectors.

[jaronoff97]

So there are a few options here on how we can implement this. The first is to match what the prometheus-operator does, which is a bit complex.

Prometheus-Operator

Prometheus uses a combination of selectors for watching and then selecting. There is the prometheus-instance-selector which the operator uses to watch specific Prometheus CRDs. There is then a slew of servicemonitor and podmonitor selectors. There are label and namespace selectors. Once the Prometheus operator's informers receive an update for any servicemonitor or podmonitor, it queries all prometheus instances from its prometheus watcher, and then checks each of their servicemonitor/podmonitor selectors to determine which prometheus instance should receive the configuration.

So ultimately the operator is watching a potential subset of Prometheus instances and all of the servicemonitor and podmonitors. From there the operator determines which Prometheus should receive the configuration from its selector lists (note I believe the default behavior when there are no selectors is give to it the first matching Prometheus)

Target Allocator

Because the TA is the instance doing the watching, not the Otel Operator, our querying should probably be a bit different. Right now every TA receives every ServiceMonitor/PodMonitor. I believe we should opt for a more simple approach than Prometheus as a result. I propose we add two new blocks of configuration to the TargetAllocator config file called service_monitor_selector and pod_monitor_selector of type metav1.LabelSelector. The default behavior will remain the same i.e. if these are unspecified, all ServiceMonitor and PodMonitor objects are queried.