Can not filter by “Duration (μs)> =” in the UI

[codefromthecrypt]

From @longinhit on October 24, 2016 9:50

Copied from original issue: openzipkin/zipkin#1349

[codefromthecrypt]

the duration query depends a lot on how your apps are instrumented and
which storage is in use. For example, we no longer support duration query
on the "cassandra" storage type.

[codefromthecrypt]

From @michaelsembwever on October 26, 2016 6:42

@longinhit it's supported in the cassandra3 storage, if you're able to upgrade.

[codefromthecrypt]

From @dangets on November 21, 2016 17:17

Is this the same issue with the Elasticsearch backend? Results that shouldn't be are getting filtered out.

[codefromthecrypt]

what instrumentation are you using? are the traces you are sending
reporting span.duration? You can check by using the api as
/api/v1/trace/trace-id-in-hex?raw

[codefromthecrypt]

fyi here's a status report on libraries that properly address
timestamp and duration
openzipkin/openzipkin.github.io#49

[codefromthecrypt]

From @dangets on November 22, 2016 16:29

We are using brave v3.15.1. We were previously using v3.9.1 until a few weeks ago and that is around when we started seeing the problem (upgraded brave and zipkin-server jar).

We do use a custom jmeter setup to run automated tests and we generate a traceId before sending the initial request so that we can log traceIds alongside request logs. Using the raw endpoint I am seeing sr and ss on the top-level span but it looks like the duration field is missing from that span.

Again, this was working previously - is this something that has changed during the deprecation of the SpanCollector classes? I can try switching to the Reporter classes to see if that fixes it.

[codefromthecrypt]

From @dangets on November 22, 2016 19:12

I verified that when I hit an endpoint using curl and specifying a traceId in the headers the duration does not show up. Without the headers it does.

And we found this comment that says a timestamp/duration won't be kept if a traceId is sent - perhaps the cause?

brave/brave-core/src/main/java/com/github/kristofa/brave/ServerRequestInterceptor.java

Line 50 in 181a2f4

// In the RPC span model, the client owns the timestamp and duration of the span. If we

Does a backfilling as mentioned in the comment ever happen?

[codefromthecrypt]

I'll look into this now. It it is a root span, we should be setting
duration for sure. Stand by and thanks for gathering the details!

[codefromthecrypt]

@dangets we are missing a feature, which is to accommodate externally provisioned root spans. The trick is to use a heuristic which doesn't accidentally double-report duration in the root span.

The only thing I can think of is to look for absence of the X-B3-Sampled header. When trace identifiers are externally generated, this header will be absent. When X-B3-Sampled: 1, we can assume the caller reported the span.

In other words, don't have jmeter set X-B3-Sampled: 1!

Change coming in a bit

[dangets]

@adriancole Thanks! We can definitely do that.

[codefromthecrypt]

#278

[codefromthecrypt]

going out now as 3.15.4

lemme know if it doesn't work!

[dangets]

Just tested it, looks good. Duration on root trace is showing up and filtering in the UI works as expected. Thanks!