Selective Tag propagation

[semistrict]

When instrumenting a library, you want to be as generous as possible with tags that you provide so that the Operator (whoever ends up using your library in an application) has maximum flexibility in how they construct views to break down stats.

However, currently all tags are propagated (for example in gRPC) by default. This means that there's a cost added for every new tag, a cost that is only visible to the Instrumentor (person instrumenting the library) but really paid by the Operator (application developer).

To solve this, we need a way for the Operator to limit tag propagation.

[semistrict]

Initial idea discussed with @bogdandrutu is by default not propagate any tags and then to provide an API similar to Views to enable their propagation, probably under a different name.

For example, you may want to propagate the gRPC service name tag under a different tag name (so that it doesn't conflict with downstream services).

Under this model, there's no cost to adding more tags if they aren't propagated and they aren't used in views.

[codefromthecrypt]

Another way of saying this is that there's a whitelist approach, right? IOTW tags exist in a namespace, regardless, and we are filtering some to propagate to the next hop, subject to a potential rename policy.

One interesting bit about this is that tags currently are validated and constrained constants, due to downstream propagation limitations. Not all of these limitations apply to in-process tags, particularly if you are inheriting a separate set of tags from a different context. Just something to think about when we consider teasing things out. If we aggressively up-front validate for remote propagation, we may end up dropping valid tags for correlation

[SergeyKanzhelev]

Should there be separate collection for Trace-Context tags? Those are quite special as you want to allow tracing libraries modify them, but not expose to end user of the tracing library.

For the rest of the tags - one way to limit those may be set TTL=0 on a tag. Similar to TTL=1 here.

[semistrict]

Another way of saying this is that there's a whitelist approach, right? IOTW tags exist in a namespace, regardless, and we are filtering some to propagate to the next hop, subject to a potential rename policy.

Yes, I think this is functionally equivalent to a whitelist although the name "whitelist" doesn't feel right to me, maybe because I think of it in the same way as the distinction between Measures/Views i.e. the Instrumentor/Operator audience distinction.

One interesting bit about this is that tags currently are validated and constrained constants, due to downstream propagation limitations. Not all of these limitations apply to in-process tags, particularly if you are inheriting a separate set of tags from a different context. Just something to think about when we consider teasing things out. If we aggressively up-front validate for remote propagation, we may end up dropping valid tags for correlation

One way to resolve this would be to assign a very short name to the propagated tag, maybe even a meaningless unique integer. Then on the "receiving" side, "unpack" the tag from the "baggage" using a new naming scheme again. The disadvantage of this approach is that it requires configuring both the originator and the receiver. However, you probably need to configure the receiver anyway to do anything meaningful with the received tag (construct a view).

[savaki]

One of the advantages of OT is its simplicity. I worry that some of what you're discussing here may be overly complex for the average user. Whether one agrees with it or not, the OT levels of propagation are clear and easy to understand; baggage and tags.

What I hear primarily is that there's a need for another level for users who are more savvy to prevent remote propagation. The other things like whitelists seem to me like user level add ons rather than part of the core census library.

So what if propagation worked as follows:

• tags require a Scope parameter of either:
  • Trace (context and all child contexts, but not remote contexts)
  • Propagate (same as Trace but propagates remotely)
  • Span* (for current context only)
• TagMap may be iterated over
• When a Span is created, it reads over the Tags that are currently in the context and creates a new TagMap, discarding tags from the parent context that were Span scoped
• Tags become attributes in the Span

As a new user of OC that has a good degree of familiarity with OT, I love the separation of tags, traces, and stats that OC defines, but I worry about the higher level of cognitive load of the OC interface.

[semistrict]

@savaki tags are not really related to tracing in the model right now. Tags are what we use to group stats by, they ultimately turn into labels in Prometheus. I think what you're referring to are attributes on Spans. These are indeed simpler. But there is currently no connection between Tags and Tracing / Spans / Span Attributes.

[songy23]

Fixed in #233.