[APM] Trace timeline: Show message when transactions are missing

[sorenlouv]

It can happen that one or more spans are pointing to a transaction that is missing. In this case it is not possible to place the spans on the timeline.

When this occurs a message in the UI should indicating that the data is incomplete.

Design

This trace contains spans from missing transactions. As a result these spans are not displayed in the timeline.

[elasticmachine]

Pinging @elastic/apm-ui

[formgeist]

@elastic/apm-ui I'll assign this to me, as I will create explicit designs for showing this tooltip for either missing and dropped spans as an enhancement over the current solution.

[formgeist]

@sqren Just bringing this issue back up to the surface, but we want to warn/notify the user at trace view level (AFAICT when looking at the referenced issue) that spans are not shown because they have no parent transaction, right?

[sorenlouv]

we want to warn/notify the user at trace view level

Yes. If by trace level you mean in contrast to on transaction/span level. So it should probably be a message at the top or bottom of the trace (but not a tooltip on a specific span/transaction).

[formgeist]

Meaning it's only relevant when viewing the full/root trace view?

[sorenlouv]

Since we don't know where in the trace the missing transactions belong we don't know if it's relevant or not to display the message in a partial trace. It's probably better to show the message than not, in case the user is expecting some transactions that are missing.

[formgeist]

Right, so displaying the message for the trace whether or not you're viewing the full trace or only a given transaction within the trace. Thanks for the feedback 👍

[sorenlouv]

Right, so displaying the message for the trace whether or not you're viewing the full trace or only a given transaction within the trace. Thanks for the feedback 👍

Yes, that sounds right.
np :)

[formgeist]

I reckon we should display a small warning badge with a tooltip that warns the user that the trace has spans that are not displayed in the timeline visualization.

Suggested copy:

This trace contains spans from missing transactions. As a result these spans are not displayed in the timeline.

[sorenlouv]

I like it 👍

[formgeist]

@sqren Updated description with the solution

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mykolaharmash]

Moving back to "Ready" in order to prioritize Profiling integration into Infra.

[jennypavlova]

I discussed with @kpatticha the scenarios here and the easiest way to verify the spans is to check if the parent.id is present - that way we know that the trace is incomplete (draft PR).

We were also discussing the scenario when a transaction is missing but it's still a parent of a span (so the parent.id is the id of this transaction) - for that case just checking if the parent id is present won't work and the spans of the missing transaction won't be shown in the waterfall.

@elastic/apm-server I have some questions: Is there a way to know if a trace is incomplete and which property should I check? In case there is no way to check on a trace level how do we validate a transaction? I am looking for a way to check that based on the trace data returned. Also, any example of an incomplete trace will be useful.

[sorenlouv]

I discussed with @kpatticha the scenarios here and the easiest way to verify the spans is to check if the parent.id is present - that way we know that the trace is incomplete (#171196).

This doesn't sound right. The problem I suggest we focus on is when items are dropped by an APM agent, are lost in transit or are still being ingested and are thus not available to be rendered (see example)
I suggest you look for items (transactions or spans) that are orphans, because their parent is missing. Orphans are items where parent.id is pointing to a non-existing item in the trace.

[kpatticha]

@sqren Thanks for chiming in.

so orphans are

• Transactions or Spans with a parent.id that doesn't exist in the trace

What about the use case where the parent.id field is not present?(expect from the root) Or we can assume that it will be present. ?

By any chance, do you have a reproducible example ?

[sorenlouv]

What about the use case where the parent.id field is not present?(expect from the root) Or we can assume that it will be present?

I'm not aware of this use case. I don't see why this would happen unless there is a bug in the APM agent (which we should then fix instead).

By any chance, do you have a reproducible example ?

Easy: Create a distributed using synthtrace, then manually delete one of the items in the trace.
Better: Use synthtrace to create a distributed trace, and use synthtrace to delete some items.

[jennypavlova]

Thank you @sqren and @kpatticha!

I suggest you look for items (transactions or spans) that are orphans, because their parent is missing. Orphans are items where parent.id is pointing to a non-existing item in the trace.

So we want to show the message if the parent id of any item in the trace is not an id of any other trace item, right? I will try it out and see - probably create a similar case using synthtrace (it was hard to remove an item the last time I tried because once I filtered out a transaction it removed all its children)

What about the use case where the parent.id field is not present? (except from the root) Or we can assume that it will be present?

I'm not aware of this use case. I don't see why this would happen unless there is a bug in the APM agent (which we should then fix instead).

Ok, so it's not something I will check for - I was thinking at the beginning that the parent id is not set if a span/transaction is an orphan as it is an optional field but I guess it is not correct.