Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare of two trace instances showing diff at wrong point #252

Open
objectiser opened this issue Sep 25, 2018 · 5 comments
Open

Compare of two trace instances showing diff at wrong point #252

objectiser opened this issue Sep 25, 2018 · 5 comments

Comments

@objectiser
Copy link
Contributor

Requirement - what kind of business use case are you trying to solve?

Compare should only show differences as red or green.

Problem - what in Jaeger blocks you from solving the requirement?

Firstly wanted to say that the new compare feature is great. However when trying it out on the Istio bookinfo example I noticed that it is showing a particular node (istio-mixer op: /istio.mixer.v1.Mixer/Check) as being added and removed under the same node - the only difference is that the green version of the node has a child span.

istio-jaeger-compare

The same issue occurs in multiple places in this particular example - shown twice in this image.

Proposal - what do you suggest to solve the problem or improve the existing situation?

If there is no difference between these nodes, then the compare should only be showing the addition of the child span. If there is a difference between these nodes that results in the green/red distinction, then the difference also needs to be included in the graphical representation, otherwise it is not clear.

Any open questions to address

@objectiser objectiser changed the title Compare of two trace instances showing diff are wrong point Compare of two trace instances showing diff at wrong point Sep 25, 2018
@tiffon
Copy link
Member

tiffon commented Sep 25, 2018

@objectiser thanks for reporting this.

The spans for istio-mixer.v1.Mixer/Check are not combined because they are not both leaves or both parents; leafs and parents are not combined.

The current treatment is intentional, but may not be correct.

A bit of context

Spans are first grouped within a trace to simplify and condense the representation, and then the condensed traces are compared. This is how nodes in the comparison DAG have numbers of spans associated with them.

When condensing a trace, two spans are in the same group iff:

  1. They have the same ordered set of ancestors as defined by the service and operation of the ancestors
  2. They are both either parents or leafs

When comparing a trace, the same heuristic is used to match groups of spans. A group of spans from trace A is matched with a group of spans from trace B iff:

  1. They have the same ordered set of ancestors as defined by the service and operation of the ancestors
  2. The spans within both groups are both either parents or leafs

Options for creating groups

Given the following trace, with A being the root node:

     C
    /
   B - C 
 /
A – B - D
 \  
  B

More dense

I believe the approached you described is achived by removing the second grouping condition and would result in the following structure:

      C(x2)
     /
A – B(x3) - D

Where the (x2) indicates the node has 2 members in the group.

Dense

The current implementation would result in:

     C(x2)
    /
   B(x2) - D
 /
A – B

Less dense

There is the possibility of changing the second grouping condition to:

  1. They have the same set of children as defined by the service and operation of each child

And, would result in the following:

   B - C(x2)
 /
A – B - D
 \  
  B

The two B nodes which are parents are not combined because they do not have the same set of children.

Which is correct?

I think "correct" is defined by which is most useful. We discussed it internally and settled on the "dense" variation. But, it's arbitrary.

The "less dense" variation preserves the structure of the trace with the most authenticity, but the more separated elements are (like parent vs child being in different groups), the more work it takes to see they're actually related. OTOH, if you over-group the spans, then the comparison becomes kind of pointless.

We could have a checkbox or radio button to toggle between the variations?


What do you think?

@yurishkuro, do you want to weigh-in?

@objectiser
Copy link
Contributor Author

@tiffon May be it would be good to have a way to select between variations, to allow us to try it out with different scenarios.

@tiffon
Copy link
Member

tiffon commented Sep 25, 2018

@objectiser Thanks for the feedback.

I wanted to add two more grouping approaches that recently came up:

  • Group on service-operation, i.e. for any service-operation in one of the traces, depict it in the graph once (akin to a dependency graph) and match service-operations from trace A to B based only on the service and operation

The current comparison is very ancestor oriented, which can be problematic. For instance, if two traces are exactly the same with the exception of the root node, they will show up as two completely different trees with no overlap. Depicting a service-operation once is a very simple approach which would alleviate this (at the cost of quite a bit of detail).

The second option is to do the same thing but consolidate on the service level instead of service-operation.

So far, the list of comparison options might be:

  • Service dependencies graph and spans are counted by service
  • Service-operation dependencies and spans are counted by service-operation
  • Ordered set of ancestors (aka "more dense", from my previous comment)
  • Ordered set of ancestors and parent vs leaf (aka "dense")
  • Ordered set of ancestors and set of children (aka "less dense")

@yurishkuro
Copy link
Member

The current comparison is very ancestor oriented, which can be problematic.

So an interesting interview question - given two compressed trace trees, can we determine through some algorithm if they should be matched at roots or at a subtree of one of them?

@tiffon
Copy link
Member

tiffon commented Sep 27, 2018

I want to note @yurishkuro's suggestion of showing the options as a series of checkboxes. Enabling a lower-checkbox would automatically enable any checkboxes above. I.E. you cannot have only the first and last checkboxes enabled.

For instance:


Two spans are in the same group iff:

  • They are from the same service
  • They have the same service & operation
  • They have the same ancestral lineage (by service & operation)
  • They are both either parents or leaves
  • They have the same set of children (by service & operation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants