-
Notifications
You must be signed in to change notification settings - Fork 448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Infinite trace generator #761
Comments
I think the answer here is that you're seeing a lot of spam in the output because you haven't included any filtering in the registry stack. The default behavior is to include everything, even items that are tagged with In this case the A typical application will have some filtering mechanism baked into the registry and would normally pick
|
Just found this issue, before i opened #1171. Same solution should work for both, I think.. |
Sounds like this might be related to #473 |
There was some attempt made to fix this as part of #1330, if someone wants to have a look further. Otherwise, I will get back to this issue shortly. The fix was to suppress the logs invoked in the context of the (async) task doing the export. |
Trying to document few options we discussed in various forums to address this issue:
Would like to discuss this further in the community meeting. |
From 5/21 Community Call:
|
It causes infinite cascading events to be emitted. See open-telemetry/opentelemetry-rust#761
* fix(stackable-telemetry): disable export of h2 events via OTLP It causes infinite cascading events to be emitted. See open-telemetry/opentelemetry-rust#761 * chore(stackable-telemetry): update changelog
This feels like it boils down to propagating a piece of information through a stack of spawned tasks. Or having access to parent tasks inside a spawned task to get information like this. Something like looking up ppid of a process to check that it was not spawned by the exporter task/process. I don't know about anything like that in tokio but it might be useful in other cases so it might be worth it following up there. But otherwise, I don't think it's currently possible to propagate information through |
Yes @mladedav, this is something we are trying to achieve. I think the problem is not specific to Most of the OpenTelemetry language implementations provide their own tracing library for instrumentation, and they add this piece of information (as a suppression flag) into the context. Then all the instrumented libraries will propagate the context into spawned tasks, and this flag can be eventually checked in the subscriber and dropped. As done in opentelemetry-js:
In the case of rust, the libraries are instrumented using tokio-rs
opentelemetry-rust provides an abstraction over the async runtimes, so any such mechanism if available should be there across all the async implementations. Tokio runtime provides an option (as mentioned in option 3 in the listed suggestions), but this has its performance overhead, and not sure if a similar is there in async-std. |
Other subscribers I've worked with usually don't need a spawned task or don't produce any further tracing information. See the I think the tasks can be assigned a slightly different subscriber that would suppress the otel layer, but it would ideally leave other layers intact. I'm not sure if that's possible, maybe the task would have to just have its tracing completely disabled. But then the libraries also need to propagate the subscriber if they spawn further tasks. I'm not exactly sure about the details of this though, I vaguely remember someone advising not to do that, but I don't remember why. I know that otel is runtime agnostic, but some kind of local-storage preserving some kind of structural concurrency ownership concept might be interesting to all of them. Not sure though. Maybe it's a bad idea, maybe there would be technical limitations. |
Any subscriber which need to make a grpc/http call should face this. For example: https://github.com/krojew/tracing-elastic-apm/blob/master/src/apm_client.rs#L103-L104 (not just grpc/http call, but pretty much any API that is already instrumented with |
I didn't mean it as that the problem isn't valid or that no one has tried to solve it before, it's just I don't have personal experience with it. By the way the example you linked seems to also go the way of spawning a single-threaded runtime, but I think it should be possible to use standard multi-threaded one with multiple threads and on_thread_start where you set the It might still be a bit wasteful since you're starting new threads for the runtime but at least it seems we're not constrained to a single thread. |
When I use the
opentelemetry_datadog
library withreqwest
as the client, each batch of spans will generate an additional 2 spans. These spans look like this:I believe this is from the http request to datadog to save the original spans. Now, the request to send these 2 spans also generates 2 new spans, which generate 2 new spans, etc.
To reproduce:
encode_headers
/parse_headers
traces get generated over and over again.I would expect to have some way to disable tracing of the tracer. Is this a bug? Do I just need to configure trace levels more precisely?
The text was updated successfully, but these errors were encountered: