No Request ID in opentelemetry trace output

[Bazer47]

Hi Maintainers,

Firstly, thanks for the great work! I would like to deal with the following issue.

Description
When I enable Triton Server tracing in triton mode, a Request ID is part of a JSON Trace output. But, when I use opentelemetry mode, I cannot see any Request ID in any span.

Example of triton mode trace output:
[{"id":2,"timestamps":[{"name":"HTTP_RECV_START","ns":409405920611710}]},{"id":2,"timestamps":[{"name":"HTTP_RECV_END","ns":409405920852250}]},{"id":2,"model_name":"model_example","model_version":1,"request_id":"250"},{"id":2,"timestamps":[{"name":"REQUEST_START","ns":409405920945849}]},{"id":2,"timestamps":[{"name":"QUEUE_START","ns":409405920951197}]},{"id":2,"timestamps":[{"name":"INFER_RESPONSE_COMPLETE","ns":409408919258919}]},{"id":2,"timestamps":[{"name":"HTTP_SEND_START","ns":409408919282319}]},{"id":2,"timestamps":[{"name":"HTTP_SEND_END","ns":409408919296258}]},{"id":2,"timestamps":[{"name":"COMPUTE_START","ns":409406878052138}]},{"id":2,"timestamps":[{"name":"COMPUTE_INPUT_END","ns":409406878139021}]},{"id":2,"timestamps":[{"name":"COMPUTE_OUTPUT_START","ns":409408919217595}]},{"id":2,"timestamps":[{"name":"COMPUTE_END","ns":409408919289501}]},{"id":2,"timestamps":[{"name":"REQUEST_END","ns":409408919321027}]}]

Example of opentelemetry trace output exported from grafana (using otlp collector, tempo db, grafana):
[{"traceId":"4b6d38a26a358b7710bd4f068689bc3d","spanId":"6586bccc3e1d5e91","parentSpanId":"e3ddf64f76f3668e","traceState":"","name":"model_example","kind":"SPAN_KIND_SERVER","startTimeUnixNano":1698973646139838500,"endTimeUnixNano":1698973650441209300,"attributes":[{"key":"triton.model_name","value":{"stringValue":"model_example"}},{"key":"triton.model_version","value":{"intValue":1}},{"key":"triton.trace_id","value":{"intValue":8}},{"key":"triton.trace_parent_id","value":{"intValue":0}}],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"status":{"code":0,"message":""},"events":[{"timeUnixNano":1698973646139838500,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"REQUEST_START"}}}],"droppedAttributesCount":0,"name":""},{"timeUnixNano":1698973646139847200,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"QUEUE_START"}}}],"droppedAttributesCount":0,"name":""},{"timeUnixNano":1698973650441209300,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"REQUEST_END"}}}],"droppedAttributesCount":0,"name":""}]},{"traceId":"4b6d38a26a358b7710bd4f068689bc3d","spanId":"8b39223cd2bff441","parentSpanId":"6586bccc3e1d5e91","traceState":"","name":"compute","kind":"SPAN_KIND_SERVER","startTimeUnixNano":1698973649319284700,"endTimeUnixNano":1698973650440223500,"attributes":[],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"status":{"code":0,"message":""},"events":[{"timeUnixNano":1698973649319284700,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"COMPUTE_START"}}}],"droppedAttributesCount":0,"name":""},{"timeUnixNano":1698973649319803400,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"COMPUTE_INPUT_END"}}}],"droppedAttributesCount":0,"name":""},{"timeUnixNano":1698973650440165000,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"COMPUTE_OUTPUT_START"}}}],"droppedAttributesCount":0,"name":""},{"timeUnixNano":1698973650440223500,"attributes":[{"key":"message","value":{"stringValue":{"Type":"STRING","Value":"COMPUTE_END"}}}],"droppedAttributesCount":0,"name":""}]}],"instrumentationLibrary":{"name":"triton-server","version":""}}]}]}

Triton Information
I am using Docker image nvcr.io/nvidia/tritonserver:23.09-py3. The Triton container is running together with otlp collector, tempo db, and grafana.

To Reproduce
Run tritonserver with following arguments: --trace-config mode=opentelemetry --trace-config opentelemetry,url=otel-collector:4318/v1/traces --trace-config rate=1 --trace-config level=TIMESTAMPS --trace-config count=100

Expected behavior
Request ID is included somewhere in the opentelemetry trace/span info.

[krishung5]

Created an enhancement ticket DLIS-5814. cc @oandreeva-nv

[oandreeva-nv]

@Bazer47 IIRC, "id":2 is a trace_id, not request id. Would you like to see Request ID in both Trace APIs?
Regarding OpenTelemetry: {"key":"triton.trace_id","value":{"intValue":8}} corresponds to id field in Triton Trace API

[Bazer47]

@oandreeva-nv I mean the request id in ...,"model_version":1,"request_id":"250"},.... A request id is available in trace mode output. But, it is not in the OpenTelemetry mode output.

According to this doc site OpenTelemetry trace support a request id should be part of each span.
Citing:

OpenTelemetry APIs produce spans that collect the same timestamps as Triton’s Trace APIs. Each span also includes model_name, model_version, request_id, and parent_id as an attribute.

[oandreeva-nv]

ah, got you! thank you for clarification and for reporting this issue!

[Bazer47]

No problem. Thank you, too!

[oandreeva-nv]

@Bazer47 , this issue should be resolved in 23.12 release, so closing now. Feel free to follow up with any questions and we will re-open this issue.