MultiExporter

[Wondertan]

Problem Statement

In a distributed p2p network environment, several backends might collect telemetry from the network nodes. A simple example has one global monitoring backend for the network and one local to each network node, s.t. global backend collects objective telemetry from all the nodes, while each node keeps a subjective view locally. In this case, having the ability to configure the node to push telemetry data to several backends through several exporters would solve the problem.

Proposed Solution

Implement MultiExporter (name TBD) implementing the Exporter interface for both tracing and metrics, which allows aggregating multiple Exporters into one.

Alternatives

The alternative is to run Otel Collector along with the node with multiple exporters. The problem with this approach is that adds additional complexity to node infrastructure, which aims to be simple.

Prior Art

AFAIK, Otel Collector allows setting multiple telemetry backends

[MadVikingGod]

The current solution to this problem would be to have multiple processors for traces. This wouldn't solve the problem of having the exact same datasets sent to multiple locations, with each processor determining where to split the batch, but it also does solve the problem of one destination blocking sending to the other destinations.

This is not part of our specification, so there is little chance that it will end up in this repository, but that is what contrib is for. If you build something and can demo it, we would be delighted to see it at the SIG meeting.

[Aneurysm9]

I would worry about going too far down this path within the SDK as this is an area where the collector is designed to operate and will have clear advantages. In the example given there is already node-local infrastructure for receiving telemetry, that could as easily be a collector that handles the fanout.

In the end, there is irreduceable total complexity in the system, so making one part of the system simple will only push that complexity elsewhere. With the collector already being designed to handle this type of complexity it is probably a better fit than the SDK.

[MrAlias]

Thanks for proposing the suggestion @Wondertan! As mentioned above, this is considered out-of-scope for this project. I'm going to close this issue as it is not something we plan to include.

[trajan0x]

For anyone finding this through google, e have an example in synapsecns/sanguine#3099 here

This implementation works for traces, not metrics (doesn't make much sense for metrics imo b/c merics are never push)

but leaving here in case anyone finds useful