Spans emitted by linkerd-proxy make it hard to use SpanMetrics due to spanName cardinality explosion

[Tyrion85]

What problem are you trying to solve?

According to Jaeger docs, when adding SpanMetrics you can expect the metrics cardinality to be:

max = 324 * num_operations
typical = 72 * num_operations

However, by default spanNames as emitted by linkerd-proxy have a full URL path set (which is a bad practice, according to Opentelemetry specs), which results in the following situation, when URL path contains variables (pretty common for APIs):

It is possible to rename spanNames, and extract variables to attributes, by using Opentelemetry Collector's SpanProcessor, for example in case above:

processors:
...
  span/to_attributes:
        name:
          to_attributes:
            rules:
            - ^\/api\/v1\/users\/(?P<userId>.*)\/own_permissions\/(?P<businessGid>.*)
...
service:
...
  pipelines:
        traces:
          processors: [ batch,span/to_attributes ]

But that would mean you have to hardcode all the paths of all APIs that have variables in URL paths, or devise of some mechanism to automatically update the central Collector's config based on what individual services have/use.

I'm really curious how do other people use SpanMetrics when using linkerd. Do you just live with this issue and cardinality explosion? Is there some other config I'm missing, or some extension/project that deals with this? As far as I can see, linkerd-proxy specifically makes SpanMetrics really hard to use.

How should the problem be solved?

I have no idea what the alternative to using full URL path as spanName would be, considering Linkerd (probably?) doesn't know which parts of the path are fixed, and which are variable.

Any alternatives you've considered?

I gave the OpentelemetryCollector SpanProcessor config above, but like I said, I would have to hardcode the pattern of every such API call. Doesn't seem very maintainable.

How would users interact with this feature?

No response

Would you like to work on this feature?

None

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[NickAdolf]

This is causing us grief as well.