Does this support propagating context from an incoming request?

[emmanuel]

Hi,

I was trying to set up and run through the example that @aledbf ported from OpenZipkin, which starts traces in the browser and propagates them through two backend services. So far, I haven't been able to get full trace propagation through Nginx.

With tracing support turned on in Nginx, the traces I see in Zipkin start with Nginx spans, and when I turn off tracing in Nginx, the traces start with the browser's spans. In both cases, the spans from the example 'frontend' and 'backend' services are correctly stitched to the traces. I infer from this that the nginx-opentracing module is not correctly extracting the trace context from the browser's requests and using that context to create child spans.

I'm confused, however, because I see what I believe is the logic for extracting context from an incoming request. Perhaps the issue is specifically related to CORS? I don't understand how that would be so, though, because the setup I described works when turning off the tracing module in nginx (specifically, I have configured Nginx to allow the trace context headers in CORS requests: x-b3-sampled, x-b3-spanid, and x-b3-traceid).

Is there something specific that needs to be configured in order for this module to propagate Zipkin context headers from incoming to outgoing (downstream to upstream) requests? And if you think this should work, do you have any suggestions about what I might do to troubleshoot?

Thank you!

[EDIT: renamed from "Does this support propagating context from an incoming request?" to "Does this support extracting trace context from an incoming request?"]

[rnburn]

nginx-opentracing does support span propagation, but I think what's going on here is that the zipkin javascript library is using V1 style spans with CS/SR/SS/CR annotations and a single span per RPC call; whereas the zipkin c++ library currently only supports the V2 style spans. It can interoperate with something like openzipkin/zipkin-go-opentracing (so long as this option is turned off), but if you try to mix it with libraries that are expecting to use a single span for an RPC call things get problematic.

I'm certainly open to extending zipkin-cpp-opentracing to support V1 style spans. You might add an issue for it (or even better a PR if you're interested in working on it). It would need something like the clientServerSameSpan option that's available in the zipkin Go tracer and also some logic like this around the span.kind tag.

[emmanuel]

Thanks @rnburn, that makes total sense. I had missed that @aledbf's fork of openzipkin/zipkin-js-example was branched immediately prior to Zipkin v2 span support landing in both the JS library and the example. Nor did I have any inkling that that was a possible or probable cause of the propagation failure that I was observing.

I'm working on rebasing my changes, so I haven't actually seen context propagation work from the browser through this nginx module. I'll come back and close this once I've got my example working e2e (also to add a link so that others might benefit from the Kubernetes portions of the updated example).

Thanks for setting me on the path!

[emmanuel]

... well that was a bit longer than I expected. I've gone back and updated my "test harness" to use the latest version of openzipkin/zipkin-js-example, which is shipping spans in the V2 format. Also, I upgraded to a newer version of this module, still by way of the Kubernetes Nginx ingress; I'm running 0.9.0-beta.17, which includes kubernetes/ingress-nginx#1580 (your PR to include the latest release of this module).

I'm still not seeing this Nginx module continue traces that start in the browser.

After performing that upgrade, it struck me that the span format shipped from the browser component shouldn't have anything to do with whether the Nginx successfully continues an in-progress trace. IIUC, continuing a trace should be entirely on the basis of the x-b3-* headers present on the request.

Here's a screenshot of an example request:

I'm running the browser front-end locally in order to avoid building a docker image and deploying that to my cluster (hence http://localhost:8080), but that shouldn't change the trace context propagation.

[rnburn]

I looked into this.

The problem is caused by a bug in zipkin's javascript library where it checks if a trace is sampled by comparing the value of the x-b3-sampled header to "1"; so it will treat traces with x-b3-sampled == "true" as unsampled.

[rnburn]

I pushed some changes to rnburn/zipkin-cpp-opentracing to make it compatible with zipkin-javascript.

I also added an example that shows how to set up nginx with client-side javascript.

[emmanuel]

Thanks @rnburn. Just for my own closure, I'll run through my set up again and report my results.