Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make OpenTelemetry-API Bridge and dependent instrumentations work with indy #11578

Merged
merged 11 commits into from
Sep 4, 2024
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class KotlinCoroutinesInstrumentationModule extends InstrumentationModule {
public class KotlinCoroutinesInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {

public KotlinCoroutinesInstrumentationModule() {
super("kotlinx-coroutines");
Expand All @@ -25,12 +27,9 @@ public boolean isHelperClass(String className) {
}

@Override
public boolean isIndyModule() {
// java.lang.LinkageError: bad method type alias: (CoroutineContext)Object[] not visible from
// class
// io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines.KotlinCoroutinesInstrumentation$ContextAdvice
// CoroutineContext is loaded from agent class loader not application
return false;
public String getModuleGroup() {
// This module uses the api context bridge helpers, therefore must be in the same classloader
return "opentelemetry-api-bridge";
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,17 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.Collections;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.0");
}

@Override
public boolean isIndyModule() {
return false;
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return asList(
Expand All @@ -31,4 +29,17 @@ public List<TypeInstrumentation> typeInstrumentations() {
new OpenTelemetryInstrumentation(),
new SpanInstrumentation());
}

@Override
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}

@Override
public List<String> agentPackagesToHide() {
// These are helper classes injected by api-version specific instrumentation modules
// We don't want to fall back to accidentally trying to load those from the agent CL
// when they haven't been injected
return Collections.singletonList("io.opentelemetry.javaagent.instrumentation.opentelemetryapi");
Comment on lines +40 to +43
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't follow why this is only a problem for this particular instrumentation? thanks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this instrumentation is the only one using this mechanism: All other api-bridge instrumentations seem to only provide the helpers, the opentelemetry-api-1.0 instrumentation is the one glueing them to the application code.

To give a bit more detail about the problem, here is an example lookup of such a helper class:

The 1.0 instrumentation looks for a OpenTelemetry implementation class (=the helper) provided by the 1.27 instrumentation to use that if available.

So with non-indy, this helper class will only be available if the 1.27 instrumentation was actually applied, because only then this helper is injected. E.g. if the application ships the API in version 1.26, this lookup would fail.

This check doesn't quite work that way for invokedynamic instrumentations. In that case, we have three classloaders:

  • The application classloader containing the application OpenTelemetry API
  • The agent classloader (containing all the .class resources of the instrumentations)
  • The InstrumentationModuleClassloader which will have the application classloader and agent classloader both as parents

When now an instrumentation is applied, its advice code and helpers are loaded into the InstrumentationModuleClassloader. The module-grouping feature ensures that all opentelemetry-api instrumentations use the same InstrumentationModuleClassloader instance.

Going back to our example where the application ships the OpenTelemetry API in 1.26, the following would happen:

  • The 1.0 instrumentation tries to lookup the helper provided by the 1.27 instrumentation
  • This helper is not injected into the InstrumentationModuleClassloader, because the instrumentation wasn't applied
  • However, the InstrumentationModuleClassloader will now look in its parents for said helper. The helper will now be loaded by the agent classloader (because it contains the .class files of the helper and therefore can load them)

The mechanism of package hiding introduced in this PR prevents the last step from happening: We disallow the helper to be looked up from the agent classloader and therefore the class loading fails as it should.

Note that once we commit to invokedynamic and remove non-invokedynamic instrumentations, we can simplify this.
A lot of this "trickery" is only required because right now when we inject helpers into application classloaders, those classes are also loaded immedatiely. With invokedynamic and the InstrumentationModuleClassloader this follows the classic java model of the classloading knowing how to load its classes, but only actually loading them on demand.

For the OpenTelemetry API bridge this means we can then just have the bridge for all versions included and simply at runtime (instead of instrumentation time) select the proper bridge implementation. We have actually done exactly that in the elastic-apm-agent for our OpenTelemetry Metrics API Bridge.
This means that the bridge code is actually completely independent of the agent/instrumentation and can be in my opinion be more easily maintained. I'm planning/proposing to take the same approach for the Otel-agent once we've dropped support for non-invokedynamic instrumentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI another neat thing about that bridge structure is that it allows to add a test for detecting unimplemented default methods:
https://github.com/elastic/apm-agent-java/blob/main/apm-agent-plugins/apm-opentelemetry/apm-opentelemetry-metrics-bridge-parent/apm-opentelemetry-metrics-bridge-latest/src/test/java/co/elastic/apm/agent/opentelemetry/metrics/bridge/BridgeExhaustivenessTest.java#L43

So that test would fail if we upgrade the SDK/API and the bridge is missing a newly added method. I don't know if something like that is already in place for the current bridge implementation.

Copy link
Member

@trask trask Aug 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, that helps me a lot, a couple of questions:

The helper will now be loaded by the agent classloader (because it contains the .class files of the helper and therefore can load them)

would we ever want any helper classes to be loaded by agent classloader? (just wondering if we can do this universally somehow instead of per instrumentation module)

Note that once we commit to invokedynamic and remove non-invokedynamic instrumentations, we can simplify this.

does that mean we can remove agentPackagesToHide() at that point?

I don't know if something like that is already in place for the current bridge implementation.

yes, this is handled by muzzle, check out #1193

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would we ever want any helper classes to be loaded by agent classloader? (just wondering if we can do this universally somehow instead of per instrumentation module)

Yes, this should be the preferred mechanism for communicating across instrumentation modules.
The "module-group" feature where we place multiple instrumentations in the same classloader was mainly introduced to have an easy way of getting the existing instrumentations to work. They currently communicate by explicitly using classes which would have been injected into the app CL before.

Ideally instrumentation which offer a service to other instrumentations should rather do this by publishing an API to the agent-CL which will then be accessible to all other instrumentations, independent of their classloader.
E.g. if an instrumentation is responsible for wrapping some application types in order to preserve context, they could offer just an API for unwrapping instead of having all their implementation details and dependencies exposed to other instrumentation modules by sharing the classloader. This should make instrumentations and their interactions easier to understand, because then there are clear borders between them.

At the moment, classes can be explicitly loaded into the agent CL by returnign false for those via the InstrumentationModule.isHelperClass, which might look a bit confusing due to the old naming.

That said, what we could do is ensure that no class for which isHelperClass is true is ever loaded from the agent-CL for a given module-group. Do you want me to try that maybe?

does that mean we can remove agentPackagesToHide() at that point?

agentPackagesToHide will actually be required when we remove shading: As soon as we do that, there will be two opentelemetry-APIs with the same name visible to the bridge InstrumentationModuleClassloader: The opentelemetry-api from the agent and from the app.
Normally, classes from the agent take precedence (so that application dependencies don't accidentally replace stuff we ship in the agent). In this case, we have the edge case that we actually do want to link against the application opentelemetry-api instead of the one from the agent. Therefore we need to hide the one from the agent and call it via some proxy-api (classes injected into the agent-cl delegating to the agent opentelemetry-API) instead.

yes, this is handled by muzzle, check out #1193

IINM that one only covers abstract methods, inherited default methods are not considered as abstract. But let's not go down that rabbit hole, that is a topic for the future

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of this "trickery" is only required because right now when we inject helpers into application classloaders, those classes are also loaded immedatiely.

FYI we used to do this, but now we only use this for the boot loader. See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/internal/internal-class-loader/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/internal/classloader/LoadInjectedClassInstrumentation.java

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't aware of that, thanks for pointing this out!

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,24 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {

public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.10");
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,23 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.15");
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,23 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.27");
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,17 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.31", "opentelemetry-api-incubator-1.31");
}

@Override
public boolean isIndyModule() {
return false;
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
Expand All @@ -35,4 +32,9 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
return hasClassesNamed(
"application.io.opentelemetry.extension.incubator.metrics.ExtendedDoubleHistogramBuilder");
}

@Override
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.32");
}
Expand All @@ -31,12 +33,12 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiIncubatorInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiIncubatorInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiIncubatorInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.32", "opentelemetry-api-incubator-1.32");
}
Expand All @@ -29,12 +31,12 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryIncubatorInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryIncubatorInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.37", "opentelemetry-api-incubator-1.37");
}
Expand All @@ -27,12 +29,12 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.38");
}
Expand All @@ -31,12 +33,12 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;
import net.bytebuddy.matcher.ElementMatcher;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiIncubatorInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiIncubatorInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiIncubatorInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.38", "opentelemetry-api-incubator-1.38");
}
Expand All @@ -29,12 +31,12 @@ public ElementMatcher.Junction<ClassLoader> classLoaderMatcher() {
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryIncubatorInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryIncubatorInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,23 @@
import com.google.auto.service.AutoService;
import io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule;
import io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation;
import io.opentelemetry.javaagent.extension.instrumentation.internal.ExperimentalInstrumentationModule;
import java.util.List;

@AutoService(InstrumentationModule.class)
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule {
public class OpenTelemetryApiInstrumentationModule extends InstrumentationModule
implements ExperimentalInstrumentationModule {
public OpenTelemetryApiInstrumentationModule() {
super("opentelemetry-api", "opentelemetry-api-1.4");
}

@Override
public boolean isIndyModule() {
return false;
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
}

@Override
public List<TypeInstrumentation> typeInstrumentations() {
return singletonList(new OpenTelemetryInstrumentation());
public String getModuleGroup() {
return "opentelemetry-api-bridge";
}
}
Loading
Loading