Skip to content

Commit

Permalink
RFC - Pipeline Component Telemetry (#11406)
Browse files Browse the repository at this point in the history 
This PR adds a RFC for normalized telemetry across all pipeline
components. See
#11343

edit by @mx-psi:
- Announced on #otel-collector-dev on 2024-10-23:
https://cloud-native.slack.com/archives/C07CCCMRXBK/p1729705290741179
- Announced on the Collector SIG meeting from 2024-10-30

---------

Co-authored-by: Alex Boten <[email protected]>
Co-authored-by: Damien Mathieu <[email protected]>
Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
  • Loading branch information
@djaglowski @codeboten
@dmathieu @wildum @evan-bradley
5 people authored Nov 27, 2024
1 parent 1b41510 commit 79357e8
Showing 1 changed file with 218 additions and 0 deletions.
218 changes: 218 additions & 0 deletions docs/rfcs/component-universal-telemetry.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Pipeline Component Telemetry

## Motivation and Scope

The collector should be observable and this must naturally include observability of its pipeline components. Pipeline components
are those components of the collector which directly interact with data, specifically receivers, processors, exporters, and connectors.

It is understood that each _type_ (`filelog`, `batch`, etc) of component may emit telemetry describing its internal workings,
and that these internally derived signals may vary greatly based on the concerns and maturity of each component. Naturally
though, there is much we can do to normalize the telemetry emitted from and about pipeline components.

Two major challenges in pursuit of broadly normalized telemetry are (1) consistent attributes, and (2) automatic capture.

This RFC represents an evolving consensus about the desired end state of component telemetry. It does _not_ claim
to describe the final state of all component telemetry, but rather seeks to document some specific aspects. It proposes a set of
attributes which are both necessary and sufficient to identify components and their instances. It also articulates one specific
mechanism by which some telemetry can be automatically captured. Finally, it describes some specific metrics and logs which should
be automatically captured for each kind of pipeline component.

## Goals

1. Define attributes that are (A) specific enough to describe individual component[_instances_](https://github.com/open-telemetry/opentelemetry-collector/issues/10534)
and (B) consistent enough for correlation across signals.
2. Articulate a mechanism which enables us to _automatically_ capture telemetry from _all pipeline components_.
3. Define specific metrics for each kind of pipeline component.
4. Define specific logs for all kinds of pipeline component.

## Attributes

All signals should use the following attributes:

### Receivers

- `otelcol.component.kind`: `receiver`
- `otelcol.component.id`: The component ID
- `otelcol.signal`: `logs`, `metrics`, `traces`, `profiles`

### Processors

- `otelcol.component.kind`: `processor`
- `otelcol.component.id`: The component ID
- `otelcol.pipeline.id`: The pipeline ID
- `otelcol.signal`: `logs`, `metrics`, `traces`, `profiles`

### Exporters

- `otelcol.component.kind`: `exporter`
- `otelcol.component.id`: The component ID
- `otelcol.signal`: `logs`, `metrics` `traces`, `profiles`

### Connectors

- `otelcol.component.kind`: `connector`
- `otelcol.component.id`: The component ID
- `otelcol.signal`: `logs`, `metrics` `traces`
- `otelcol.signal.output`: `logs`, `metrics` `traces`, `profiles`

Note: The `otelcol.signal`, `otelcol.signal.output`, or `otelcol.pipeline.id` attributes may be omitted if the corresponding component instances
are unified by the component implementation. For example, the `otlp` receiver is a singleton, so its telemetry is not specific to a signal.
Similarly, the `memory_limiter` processor is a singleton, so its telemetry is not specific to a pipeline.

## Auto-Instrumentation Mechanism

The mechanism of telemetry capture should be _external_ to components. Specifically, we should observe telemetry at each point where a
component passes data to another component, and, at each point where a component consumes data from another component. In terms of the
component graph, every _edge_ in the graph will have two layers of instrumentation - one for the producing component and one for the
consuming component. Importantly, each layer generates telemetry ascribed to a single component instance, so by having two layers per
edge we can describe both sides of each handoff independently.

Telemetry captured by this mechanism should be associated with an instrumentation scope corresponding to the package which implements
the mechanism. Currently, that package is `service/internal/graph`, but this may change in the future. Notably, this telemetry is not
ascribed to individual component packages, both because the instrumentation scope is intended to describe the origin of the telemetry,
and because no mechanism is presently identified which would allow us to determine the characteristics of a component-specific scope.

### Instrumentation Scope

All telemetry described in this RFC should include a scope name which corresponds to the package which implements the telemetry. If the
package is internal, then the scope name should be that of the module which contains the package. For example,
`go.opentelemetry.io/service` should be used instead of `go.opentelemetry.io/service/internal/graph`.

### Auto-Instrumented Metrics

There are two straightforward measurements that can be made on any pdata:

1. A count of "items" (spans, data points, or log records). These are low cost but broadly useful, so they should be enabled by default.
2. A measure of size, based on [ProtoMarshaler.Sizer()](https://github.com/open-telemetry/opentelemetry-collector/blob/9907ba50df0d5853c34d2962cf21da42e15a560d/pdata/ptrace/pb.go#l11).
These may be high cost to compute, so by default they should be disabled (and not calculated). This default setting may change in the future if it is demonstrated that the cost is generally acceptable.

The location of these measurements can be described in terms of whether the data is "consumed" or "produced", from the perspective of the
component to which the telemetry is attributed. Metrics which contain the term "produced" describe data which is emitted from the component,
while metrics which contain the term "consumed" describe data which is received by the component.

For both metrics, an `outcome` attribute with possible values `success` and `failure` should be automatically recorded, corresponding to
whether or not the corresponding function call returned an error. Specifically, consumed measurements will be recorded with `outcome` as
`failure` when a call from the previous component the `ConsumeX` function returns an error, and `success` otherwise. Likewise, produced
measurements will be recorded with `outcome` as `failure` when a call to the next consumer's `ConsumeX` function returns an error, and
`success` otherwise.

```yaml
otelcol.receiver.produced.items:
enabled: true
description: Number of items emitted from the receiver.
unit: "{item}"
sum:
value_type: int
monotonic: true
otelcol.processor.consumed.items:
enabled: true
description: Number of items passed to the processor.
unit: "{item}"
sum:
value_type: int
monotonic: true
otelcol.processor.produced.items:
enabled: true
description: Number of items emitted from the processor.
unit: "{item}"
sum:
value_type: int
monotonic: true
otelcol.connector.consumed.items:
enabled: true
description: Number of items passed to the connector.
unit: "{item}"
sum:
value_type: int
monotonic: true
otelcol.connector.produced.items:
enabled: true
description: Number of items emitted from the connector.
unit: "{item}"
sum:
value_type: int
monotonic: true
otelcol.exporter.consumed.items:
enabled: true
description: Number of items passed to the exporter.
unit: "{item}"
sum:
value_type: int
monotonic: true

otelcol.receiver.produced.size:
enabled: false
description: Size of items emitted from the receiver.
unit: "By"
sum:
value_type: int
monotonic: true
otelcol.processor.consumed.size:
enabled: false
description: Size of items passed to the processor.
unit: "By"
sum:
value_type: int
monotonic: true
otelcol.processor.produced.size:
enabled: false
description: Size of items emitted from the processor.
unit: "By"
sum:
value_type: int
monotonic: true
otelcol.connector.consumed.size:
enabled: false
description: Size of items passed to the connector.
unit: "By"
sum:
value_type: int
monotonic: true
otelcol.connector.produced.size:
enabled: false
description: Size of items emitted from the connector.
unit: "By"
sum:
value_type: int
monotonic: true
otelcol.exporter.consumed.size:
enabled: false
description: Size of items passed to the exporter.
unit: "By"
sum:
value_type: int
monotonic: true
```
### Auto-Instrumented Logs
Metrics provide most of the observability we need but there are some gaps which logs can fill. Although metrics would describe the overall
item counts, it is helpful in some cases to record more granular events. e.g. If a produced batch of 10,000 spans results in an error, but
100 batches of 100 spans succeed, this may be a matter of batch size that can be detected by analyzing logs, while the corresponding metric
reports only that a 50% success rate is observed.
For security and performance reasons, it would not be appropriate to log the contents of telemetry.
It's very easy for logs to become too noisy. Even if errors are occurring frequently in the data pipeline, only the errors that are not
handled automatically will be of interest to most users.
With the above considerations, this proposal includes only that we add a DEBUG log for each individual outcome. This should be sufficient for
detailed troubleshooting but does not impact users otherwise.
In the future, it may be helpful to define triggers for reporting repeated failures at a higher severity level. e.g. N number of failures in
a row, or a moving average success %. For now, the criteria and necessary configurability is unclear so this is mentioned only as an example
of future possibilities.
### Auto-Instrumented Spans
It is not clear that any spans can be captured automatically with the proposed mechanism. We have the ability to insert instrumentation both
before and after processors and connectors. However, we generally cannot assume a 1:1 relationship between consumed and produced data.
## Additional Context
This proposal pulls from a number of issues and PRs:
- [Demonstrate graph-based metrics](https://github.com/open-telemetry/opentelemetry-collector/pull/11311)
- [Attributes for component instancing](https://github.com/open-telemetry/opentelemetry-collector/issues/11179)
- [Simple processor metrics](https://github.com/open-telemetry/opentelemetry-collector/issues/10708)
- [Component instancing is complicated](https://github.com/open-telemetry/opentelemetry-collector/issues/10534)

0 comments on commit 79357e8

Please sign in to comment.