Prototype: OpenTelemetry components for Flow #1843
Closed
+1,968
−2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This (large) draft PR demonstrates how OpenTelemetry components can be added as Flow components.
This is just a draft to demonstrate the capability, and needs a lot of attention before it's mergeable.
The implementation of these components completely bypasses the OpenTelemetry Collector pipeline system, instead relying on the Flow controller to bind OTel components together, passing around consumers as interfaces.
Each component which can receive telemetry data is bundled as a generic
TelemetryConsumer
, which accepts metrics, logs, and traces. It is a combination of the three consumer interfaces from OpenTelemetry Collector. If the underlying consumer does not support one of the telemetry types, the request will fail. In the future, this should probably be changed to be validated at load time instead of failing at runtime.The following example components are included:
otel.reciever_jaeger
: The upstream Jaeger receiverotel.exporter_otlp
: The upstream OTLP exporterotel.receiver_otlp
: The upstream OTLP receiverotel.receiver_zipkin
: The upstream Zipkin receiver.otel.exporter_logging:
A custom component which uses go-kit's logger for logging about received traces.There is generic code for constructing and running the various OpenTelmetry Collector components. The remainder of the code involves recreating config structs for unmarshaling and converting it to the upstream types on-demand. While this is more repetitive than the pkg/traces approach of unmarshaling into a
map[string]interface{}
, I personally find it preferable since it is more discoverable when examining the code and more likely to fail fast at compile time if upstream changes the type definition unexpectedly.Example config:
After running, follow Tempo's guide to push Zipkin traces using cURL to see Spans appear in Tempo.
Note that running these components will break the /-/config endpoint, since rfratto/gohcl isn't able to properly serialize a list of capsule values. The fix for that doesn't seem straightforward, and a long-term fix may need to wait for #1839.