Classloading issue with custom instrumentation

[patsonluk]

Hi all,

We are experimenting with adding our custom instrumentation as an extension (-Dotel.javaagent.experimental.initializer.jar), and have some success by tagging our Instrumentation Module with @AutoService(InstrumentationModule.class).

By overriding additionalHelperClassNames, we can get the advice to reference helper class of ours. Unfortunately, in more complex usage, which we reference more 3rd party library from our helper class (which the 3rd party libraries are packaged in our extension jar), it's unpractical to list out all the dependencies.

Therefore, we modified our extension build to integrate the Muzzle plugin and override isHelperClass to make it scan all the usage of our helper class and all the 3rd party library dependencies (which we shaded to have our prefix, hence isHelperClass can really catch them all). The issue though, is that the list grows too long (1k+ classes, since we reference frameworks like Guava), and exception of Method too large: is thrown during muzzle code generation.

By looking at the OT agent itself, it uses Muzzle plugin only to find the "helper" class (and not dependencies on other modules), as those dependencies are made available by appending the agent jar itself to the bootstrap loader path search https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/v1.1.0/javaagent-bootstrap/src/main/java/io/opentelemetry/javaagent/OpenTelemetryAgent.java#L129 , which should make all the dependencies provided via the agent jar available to almost all classloader (except some oddball classloader, which might skip bootstrap loader path search/load class by resource?)

That means for us, we might be able to get around it by using non standard jvm option -Xbootclasspath/a (though non-standard option is bad?). So our questions are:

1. Is tagging @AutoService(InstrumentationModule.class) a legit way to add custom instrumentation via extension (-Dotel.javaagent.experimental.initializer.jar). Or is forking this repo the only way?
2. Assuming we can use extension to provide custom instrumentation (which is good, since that's less dependency), any suggestions of solving this classloading issue? Would it be possible if OT append extension jar to bootstrap classpath as well much like the the OT java agent itself? (this does not work, at least not in any trivial way, for example if we have a custom ConfigurablePropagatorProvider implementation , and such implementation will be loaded by java.util.ServiceLoader, since the implementation is found in bootstrap classpath, it will be loaded by bootstrap class loader, however, when JVM tries to load the interface, which is ConfigurablePropagatorProvider, it runs into java.lang.NoClassDefFoundError due to such an interface not available to bootstrap class loader (it's only visible to AgentClassLoader)

Many thanks for your kind attention!

[mateuszrzeszutek]

Hey @patsonluk ,

1. Yes, using InstrumentationModules in extension jars is definitely a legit way of adding custom instrumentations.
2. Do I understand it correctly that those 3rd party libraries are unavailable in the instrumented applications and you have to inject them using the instrumentation?
   Since you're shading your code anyway, try using splitting the dependencies and the actual instrumentation, e.g. keep all dependencies shaded in a com.myextension.bootstrap package and in a separate jar, and the instrumentation code in a com.myextension.instrumentation package, also in a separate jar. You could append the first (deps) jar to the bootstrap classpath and only use the second one as the extensions/initializer jar; the isHelperClass() in your instrumentation module would also only check for the instrumentation package.
   This way muzzle would only collect references only to the instrumentation classes (and the library classes that they reference) and assume that all dependencies have to be present in the instrumented application - and they should always be present, since you're adding them to the bootstrap classpath.

I'd also really recommend avoiding depending on too many libraries/frameworks and keeping the instrumentation code minimal; but if you're not able to do it then you can try splitting the jars/packages - hope that it works for you.

[mateuszrzeszutek]

Providing separate jars and using extra non standard xbootclasspath means a bit less convenient for user, though it's probably of very minor concern and we might be able to address that with some wrapper scripts from us :)

There's another approach that you might want to consider if you want to keep things in a single jar: you can repackage otel instrumentation agent as a custom distro and add whatever you need to the bootstrap loader. This requires a bit more complex build process and is a bit of a hassle, but might fit your use case.

1. Be given the java.lang.instrument.Instrumentation object, so the extension can choose to append jar in a more flexible way - for example the extension jar can actually create a "jar-in-jar" during the build process. And during runtime, extract that jar to be appended to the bootstrap path search.

We sort-of have something like that: check out the BootstrapPackagesProvider SPI. Unfortunately it doesn't work with extensions currently, only javaagent distros - and is not able to add resources to the bootstrap classpath, only classes. I have an idea how to make this better though, perhaps in the next release this'll be ready for extension usage.

[patsonluk]

Ah many thanks for the custom distro suggestion! From my quick read, it seems to provide us more control over the behavior of OT agent (comparing to rolling just our own extension) while still provides a good separation of code logic? (ours vs OT's)

So for agent - contains the main repackaging functionality and, optionally, an entry point to the agent, if one wishes to customize that, is it referring to the fact that we can modify the premain class to point to ours instead hence modifying the entry point? Or are there some other hooks out there that we should use? :)

We sort-of have something like that: check out the BootstrapPackagesProvider SPI. Unfortunately it doesn't work with extensions currently, only javaagent distros - and is not able to add resources to the bootstrap classpath, only classes. I have an idea how to make this better though, perhaps in the next release this'll be ready for extension usage.

That sounds cool, we will keep checking out for further updates!

[mateuszrzeszutek]

So for agent - contains the main repackaging functionality and, optionally, an entry point to the agent, if one wishes to customize that, is it referring to the fact that we can modify the premain class to point to ours instead hence modifying the entry point?

Yep, it's exactly that.

[patsonluk]

We have also tried the custom distro approach. While adding our custom instrumentation, it's found the muzzling is not executed by default with the current custom repo setup, the example DemoServlet3Instrumentation does not seem to need muzzling as the advice does not reference "helper class" like BaseTracer.

So we are wondering if any example of muzzle usage will be added to the example custom distro?

We are trying to get muzzle plugin to run in our custom distro (which we did successfully on our extension), but in a rather ugly way as we are not that familiar with the javaagent gradle build with opentelemetry-java-instrumentation.

Many thanks again! 😊

[mateuszrzeszutek]

Yeah, muzzle code generation currently works only in this repo, if you need it then unfortunately you need to copy the muzzle build logic over to your distro (which I'm guessing should be pretty similar to what you already did in the extension). You can take a look at files changed in PR #3401, you'll probaby need to replicate a similar setup in your project.

There's issue #2778 for extracting and publishing the muzzle codegen gradle plugin so that it can be used in external projects. This is something that'll benefit all distro/extension authors - once it's done it'll be definitely added to distro and extension examples hosted in this repo.