Make collectors discover each other for scrapers sharding

[francois07]

Component(s)

receivers, extension/observer

Is your feature request related to a problem? Please describe.

Currently, scrapers sharding can only be achieved by using the OpenTelemetry Operator's Target Allocator.

Describe the solution you'd like

Similarly to how Grafana Alloy handles scrapers sharding, it would be very useful to make it so that collectors are able to discover each. This would eliminate the need for the Target Allocator especially for people that do not want to use the OpenTelemetry Operator.
Moreover, this would make the collector instances decentralized, which I believe is a benefit in cloud environments.

An interesting way to do this would be to make use of the observer extension to detect collector instances and shard the scrapers' targets in each of them accordingly.

Describe alternatives you've considered

Using the Target Allocator as a standalone agent (no documentation)

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[francois07]

/label extension/observer

[github-actions]

Pinging code owners for extension/observer: @dmitryax @rmfitzpatrick. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

What would the observer observe to find collector (k8s pods, something else)? Prometheus service discovery already has similar mechanisms to observer: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. Would this be different?

[francois07]

@dashpole The idea here is to shard the scrapers when there are multiple instances of the same collector. This is a functionality that is currently implemented using the Target Allocator which involves additional configuration as well as the need to use the OpenTelemetry Operator.

In Grafana Alloy, for example, there is no such need for any additional agent. Collectors are able to recognize each other and shard their scrapers accordingly. Implementing a similar feature would make it much easier to scale the collector.

To answer you question, the observer would observe pods similarly to the target allocator to determine which pods are collectors of the same instance. Then, a list of collectors could be retrieved and used to split the scraping between each instances.

For reference, the Target Allocator is configured the following way

collector_selector:
  matchlabels:
    app.kubernetes.io/component: otel
    app.kubernetes.io/part-of: demo

Another way to do this would be by using go-discover just like Alloy, but I believe using existing internal tools is better. What do you think ?

[dashpole]

I wonder if it is similar to how the prometheus operator does sharding? It uses prometheus k8s service discovery, and the hashmod function to split targets by the number of collectors running

[francois07]

I'm not sure about how Alloy does it exactly as they are using their own package to do it, but i know the Target Allocator is using consistent to distribute targets which works similarly to hashmod

[francois07]

To clarify, I'm envisioning a configuration similar to the following

extensions:
  k8s_observer:
    observe_pods: true
receivers:
  prometheus:
    sharding:
      enabled: true
      watch_observers: [k8s_observer]
      selector:
        matchlabels:
          app.kubernetes.io/component: otelcol
    config:
      scrape_configs: [...]

in the background, scrapers would only scrape targets discovered by scrape_configs that are "assigned" to them, so they would have t/n targets with t the total number of scraping targets and n the total number of detected collectors.

[seankhliao]

it would be a bit like embedding an instance of target allocator in the collector, it does seem somewhat less efficient in that each instance will have to do all the k8s service discovery independently.

[dashpole]

Based on https://github.com/open-telemetry/opentelemetry-operator/blob/9f1eb133ebaf83ce0ca822ea1198fe73ef4eca95/cmd/otel-allocator/target/discovery.go, the otel operator is just using prometheus service discovery under the hood. You should be able to use the hashmod operator as described above to achieve what you are trying to do.