[Feature]: SPM Provide option to fill null gaps with zero in monitor tab

[Ashmita152]

Requirement

Currently, if there is no error in a traces(ie. no error=true tag in spans), otel-collector doesn't send any error metrics to prometheus (i.e. it doesn't create any metric with prometheus label as status_code = "STATUS_CODE_ERROR"). This generates empty dashboard on Jaeger monitor tab.

Problem

It creates empty dashboards on monitor tab which can create confusion whether:

• data is not being ingested through the pipeline
• data has no error

Proposal

In order to prevent confusion, it will be better if instead of showing empty dashboard, we show dashboard containing data value as zero.

cc @albertteoh

[yurishkuro]

If I understand the situation correctly, this results in no data being written to Prometheus. In this case, how can Jaeger know if it's an ingestion problem or no-errors traffic? Filling gaps with zeros by Jaeger does not answer that question, it just masks the problem, making it look like there is data with zeros.

[albertteoh]

If I understand the situation correctly, this results in no data being written to Prometheus. In this case, how can Jaeger know if it's an ingestion problem or no-errors traffic? Filling gaps with zeros by Jaeger does not answer that question, it just masks the problem, making it look like there is data with zeros.

Error metrics are subset of all calls/requests using the query:

calls_total{service_name =~ "...", status_code = "STATUS_CODE_ERROR"}

If there is data resulting from the request count query:

calls_total{service_name =~ "..."}

Should that give us enough information to know that there were no ingestion issues? Similarly, the absence of request count data means we can't assume that there is no-errors traffic, in which case, we should display "no data" instead of filling in zeroes.

What do you think?

[yurishkuro]

strictly speaking, error metric is a distinct time series stored independently, but that's too in the weeds, so I agree if calls_total has data it's reasonably safe to assume no ingestion issues.