Disabling completely instrumentation at runtime?

[tbolon]

I would like to disable completely instrumentation at runtime, after having configured the Sdk.

Initially, I was trying to use the AlwaysOff sampler on traces, but discovered that Activity are still returned by the ActivitySource.
It seems intentional because even when then sampler should discard, the root activity is emitted to preserve and propagate the trace id.

But SuppressInstrumentationScope will drop all traces, and I would like to apply this behavior globally.

My suggestion could be to simply add a public setter on Sdk.SuppressInstrumentation with a bool backend. Combine SuppressInstrumentationScope.IsSuppressed and the backing field on get, and set the backing field on set:

Current:

public static class Sdk
{
    /// <summary>
    /// Gets a value indicating whether instrumentation is suppressed (disabled).
    /// </summary>
    public static bool SuppressInstrumentation => SuppressInstrumentationScope.IsSuppressed;
}

My suggestion:

public static class Sdk
{
    private static bool suppressed;

    /// <summary>
    /// Gets or sets a value indicating whether instrumentation is suppressed (disabled).
    /// </summary>
    public static bool SuppressInstrumentation
    {
        get => suppressed || SuppressInstrumentationScope.IsSuppressed;
        set => suppressed = value;
    }
}

I don't think this value is expected to be cached by the caller (because SuppressInstrumentationScope rely on this), so I don't see any downside on this feature.

[cijothomas]

Could you describe what you are trying to achieve? Disabling instrumentation at runtime is not something supported by the OTel specification, unless you use custom sampler.
Suppress instrumentation is used to "suppress" telemetry in a execution context only (typically used to prevent live loops, where an Otel exporter could be triggering instrumentation..)

[cijothomas]

I don't see any reason why it is not supported. (Even creating a TracerProvider, using it for just one activity, then disposing it is supported, but its highly discouraged. Your use case of a "controlled" create/dispose TracerProvider seems okay to me)

[tbolon]

I just had the wrong idea that everything was coupled to the application lifetime. Certainly a bias based on my experience using Application Insights, and the tendency to read all tutorials about starting the instrumentation with the startup of the application, and forgetting about it.

I was also convinced that some instrumentation components could require that the component is created after the instrumentation is started. See redis for (actually the contrary) example: the instrumentation requires that the connection is created before the SDK is configured and that the connection is passed as a parameter.

I expect that is some cases, when there are static or singleton components, adding the instrumentation after the app have started will not be supported, or will require some tricks. I need to investigate more.

This is a good topic to get some thoughts from the specification SIG as well, as I think this behavior would be same issue in other languages as well, so you won't be able to achieve what you want in any languages.

Regarding this suggestion, could you direct me where I should raise this topic?

[cijothomas]

https://github.com/open-telemetry/opentelemetry-specification - repo
https://cloud-native.slack.com/archives/C01N7PP1THC - Slack channel

[cijothomas]

I expect that is some cases, when there are static or singleton components, adding the instrumentation after the app have started will not be supported, or will require some tricks

This is true. Sorry for not calling this out. (The common instrumentations like HttpClient, ASP.NET Core should be fine)
Note that you'll be experimenting with this idea for the 1st time - would love to hear back here, for the benefit of others having similar needs.

[tbolon]

I am trying to wrap my ideas before starting a discussion on the specification repo. Basically there are multiple topics I would like to raise, and I am not sure if they apply to the specification repo:

• Could OpenTelemetry be used as a temporary instrumentation system, where we want to start the application without any instrumentation overhead (which means max close to 0 overhead), then instrumentation is started and traces/metrics/logs are emitted on demand. Is it something covered by the specification? Should a SDK support that?

• Should a SDK support being created and disposed multiple times in an application lifetime? Is it something that should be written at the specification level? The current specification for the SDK is quite scarse.

• When a trace SDK is started, should the TraceAdapter always emit activity (read traceid and propagate it), even if the sampler is asking to drop all traces (AlwaysOffSampler)? Currently in .NET even with AlwaysOff, the sampler is attached to the ActivitySource listener (causing all the activity tracing overhead), and propagate activity. If the SDK is supposed to be tied to the lifetime of the application, there should be a way to lower the overhead of the instrumentation to the bare minimum.

Perhaps could you give me advice about how to raise a constructive discussion based on these points?

On my side, I will try to play with the examples project to inspect the behavior when I start/stop the sdk when the application is running.