Support dynamic enrichment and customization for telemetry collected by instrumentation libraries

[lmolkova]

Some of the issues raised in spec and semconv are related to interaction between application and instrumentation code (or different layers of instrumentations):

• classifying errors - it's context-dependent
• enriching telemetry - e.g. add specific attributes to metric depending on the specific context call is made in.
• suppressing all nested instrumentations or duplicates in certain scope

See open-telemetry/opentelemetry-specification#4131 (comment) for the context.

Such customizations are sometimes possible for spans (using SpanProcessors), but not in general possible for metrics.

I'd like to propose tackling this issue in scope of the devex project.

[tsloughter]

Agreed, but I think these should be broken into 3 separate issues, right?

Sidebar: I think we need a label for issues that are proposals of items to actually work on, but not sure what to call it.

[lmolkova]

I agree that this is a big area that will need multiple sub-issues, but if we're going to work on any of them, we should keep the wider area in mind.

We should avoid designing APIs for each individual problem that might not work or look ugly together.

So I think we need to have a general idea on how to solve the big problem and then we can break it down into smaller pieces that play nicely with each other.

[lmolkova]

An example to demonstrate how different areas are related.

Let's say we want to define enrichment API (open-telemetry/oteps#207), e.g.:

try (var scope = Instrumentation.withAttributes("url.template", "https://endpoint/user/{user_id}")) {
  httpClient.get("https://endpoint/user/alice");
}

suppression (open-telemetry/oteps#172) API might look like

try (var scope = Instrumentation.suppressSpans(tracer -> isHttpTracer(tracer))) {
  httpClient.get("https://endpoint/user/alice");
}

general purpose customization might look like

var urlTemplate = "https://endpoint/user/{user_id}";
try (var scope = Instrumentation.customize(
        span -> span
            .setAttribute("url.template", urlTemplate)
            .setAttribute("user.id", "alice"), // might as well adjust span status here
        measurement -> measurement
            .setAttribute("url.template", urlTemplate))) {
  httpClient.get("https://endpoint/user/alice");
}

I'm not proposing any of these APIs in particular - they all have some issues, but I'm saying that they are related to each other.