Add trace ids to nginx log

[Sennahoi]

Is it possible to add the trace ids to the nginx log file?

[nwest1]

I've been logging the headers as per nginx norm: $http_x_b3_traceid. That's not going to get you a full trace based on other opentracing directives (like anything created with opentracing_tag or automatically-added metadata in locations)

It would be nice have the ability to log out the full trace(s) - or at least have access to the variables that might make those up. Are those available?

If there's a better way to do this, that'd be great to see.

Currently strategy: (var_* are lua-calculated)

  log_format tracing '[{'
    '"traceId": "$http_x_b3_traceid",'
    '"name": "nginx",'
    '"id": "$http_x_b3_spanid",'
    '"parentId": "$http_x_b3_parentspanid",'
    '"timestamp": $var_epoch_micros,'
    '"duration": $var_request_time_micros,'
    '"binaryAnnotations":['
      '{"key":"lc","value":"nginx","endpoint":{"serviceName":"gateway-nginx","ipv4":"$server_addr"}}'
    ']'
    '}]'
  ;

[geekeren]

This example can help you:
https://github.com/geekeren/docker-zipkin-nginx-example/blob/b0a363f23fb4125caf0f8e1d77a39cdd29663604/files/nginx.conf#L18

[Sund3]

Does anyone have an example for jaeger trace id for the $opentracing_context_ ?

[sdanbury]

@Sund3 did you work it out in the end?

So if I am looking to log the trace IDs and I don't pass in any trace header in the request then I would expect this plugin to generate one for me. In order to log that trace ID generated by the plugin, can I just do this in the usual way, such as $http_*? For example, if the header was uber-trace-id, I could log $http_uber_trace_id?

For that to be true, then I would assume the plugin works by adding a request header, as opposed to adding a proxied header or something? Is that correct?

[wosc]

If you're using the w3c propagation format and want to log the trace/parent/span ID, it is doable, but somewhat cumbersome: nginx-opentracing provides variables for whatever directly exists in the opentracing SpanContext, and jaeger-client-cpp (which is the tracer implementation used in nginx-ingress) currently only puts the whole propagated header in there, so you have to parse it out into individual variables again manually. I've got it to work like this:

map $http_traceparent $my_trace_id {
  default "00000000000000000000000000000000";
  ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$trace_id";
}
map $http_traceparent $my_parent_id {
  default "0000000000000000";
  ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$parent_id";
}
map $opentracing_context_traceparent $my_span_id {
  default "0000000000000000";
  ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$parent_id";
}

log_format tracing '{"trace_id": "$my_trace_id", "parent_id": "$my_parent_id", "span_id": "$my_span_id"}';

If anyone knows an easier workaround, I'm all ears, and of course I'd be even more happy if nginx-opentracing could provide standardized variables for the three IDs, independent of tracer implementation and propagation format.

[ubonagiri]

Does anyone have an example for jaeger trace id for the $opentracing_context_ ?

in your logs refer variable - $opentracing_context_uber_trace_id . where as traceContextHeaderName configured with 'uber_trace_id' as below

  {
  "service_name": "JAEGER-SERVICE-NAME",
  "sampler": {
    "type": "JAEGER-SAMPLER-TYPE",
    "param": JAEGER-SAMPLER-PARAM
  },
  "reporter": {
    "localAgentHostPort": "JAEGER-REPORTER-LOCAL-AGENT-HOST-PORT"
  },
  "headers": {
    "jaegerDebugHeader": "jaeger-debug-id",
    "jaegerBaggageHeader": "jaeger-baggage",
    "traceBaggageHeaderPrefix": "uberctx-",
    "traceContextHeaderName": "uber-trace-id"
  },
  "baggage_restrictions": {
    "denyBaggageOnInitializationFailure": false,
    "hostPort": ""
  }
}

[ghost]

Based on @wosc's comment, am I wrong to think that this issue should be opened on https://github.com/jaegertracing/jaeger-client-cpp/ ?

[james-callahan]

in your logs refer variable - $opentracing_context_uber_trace_id . where as traceContextHeaderName configured with 'uber_trace_id' as below

That results in the entire trace context header (which is 4 colon separated fields).
As a workaround until this issue is solved we're using:

# Parse an uber-trace-id header
# It consists of four colon separated hexidecimal values
# e.g. bb67795d1171a727:e3368d224589f8d4:bb67795d1171a727:1
map $opentracing_context_uber_trace_id $trace_id {
    default "";
    ~^([0-9a-f]+):[0-9a-f]+:[0-9a-f]+:[0-9a-f]+$ "$1";
}

Additional note: we're doing this in the context of ingress-nginx; that map snippet goes into the http-snippet, and then in log-format-upstream we use $trace_id.

[miry]

Contribution are welcome. There are no active developers that would work on it as far I know.

[james-callahan]

I had a quick look at how to do it; for whoever gets around to it:

• Add a new variable (similar to opentracing_binary_context)
• In the handler/fetcher, call active_span
• Then call the ToTraceID() method to get the trace ID as a string