How can I programmatically find clock skew warnings and orphaned spans warnings reported by the UI

[johnmayock]

We have instrumented OTEL within a number of micro-services and ingresses. They are now building up substantial span counts for each trace.

We are seeing reports from users of the OTEL data, of clock skew warnings and of traces with orphaned spans (which report in the JaegerUI of an invalid parent span id warning).

I'm trying to create a manual (or automated) script that can flag these up on a dashboard so that we can investigate and resolve.

Does any know how I can gather tracesIds what contain either of these two warnings?

The http/grpc trace retrieval API doesn't appear to support. nor did data retrieved from opensearch db (using curl) appear to indicate these (they may not form part of the models/schema), I assume this in inferred in Jaeger Query or the UI, upon receipt of all spans within a given trace requested on the UI.

[yurishkuro]

Jaeger performs clock skew adjustments/calculations at trace retrieval time, so there is no data in the database that allows to query by that.

I'd like to better understand what you're trying to do and what the issue is that the users are complaining about.

[johnmayock]

re clock skew
We have reports of traces with large clock skew deltas reported and I wanted to obtain traces without manual hunting, to further investigate drift by isolating host using the telemetry in particular (using k8sattributes processor).

re orphaned spans
I believe users may be hitting #2231 on occasion reporting
spans being broken and reporting invalid parent spanIDs but I have seen cases where someone team
may have added an OTTL based drop filter on the OTEL collector and dropped a parent span without consideration that child spans still exist and become orphaned (reporting error in Jager UI of invalid parent span ID).

The use case in this one, is to search for traceIDs that contain this warning. So that a someone can detect and then investigate triage if its a case of erroneous span filtering or something else such as issue on collector.