The agent is unable to instrument ManagedExecutorService and crashes the thread

[mdindoffer]

The APM agent is unable to instrument the ManagedExecutorService.
Whenever a Callable is submitted invocation crashes with:

SEVERE [org.glassfish.enterprise.concurrent] (EE-ManagedExecutorService-default-Thread-1) java.lang.ClassCastException: co.elastic.apm.agent.impl.async.ContextInScopeRunnableWrapper cannot be cast to org.glassfish.enterprise.concurrent.internal.ManagedFutureTask

Environment
OpenJDK 1.8.0_191 (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
Wildfly 9.0.2.Final
APM agent 1.5.0, 1.6.0-SNAPSHOT at fb3c7cd

Default Wildfly EE configuration:

Click to expand

<subsystem xmlns="urn:jboss:domain:ee:3.0">
    <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
    <concurrent>
        <context-services>
            <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
        </context-services>
        <managed-thread-factories>
            <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
        </managed-thread-factories>
        <managed-executor-services>
            <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" core-threads="5" max-threads="25" keepalive-time="5000"/>
        </managed-executor-services>
        <managed-scheduled-executor-services>
            <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" core-threads="2" keepalive-time="3000"/>
        </managed-scheduled-executor-services>
    </concurrent>
    <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" jms-connection-factory="java:/activemq/ConnectionFactory" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
</subsystem>

Nothing to see in debug logs, except for loads of these:

ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - co.elastic.apm.api.CaptureTransaction was loaded by org.jboss.modules.ModuleClassLoader
ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - co.elastic.apm.api.CaptureSpan was loaded by org.jboss.modules.ModuleClassLoader
ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - javax.ws.rs.Path was loaded by org.jboss.modules.ModuleClassLoader
ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - javax.jws.WebService was loaded by org.jboss.modules.ModuleClassLoader
ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - javax.faces.lifecycle.Lifecycle was loaded by org.jboss.modules.ModuleClassLoader
ERROR [stderr] (ServerService Thread Pool -- 73) [ServerService Thread Pool -- 73] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - javax.faces.lifecycle.Lifecycle was loaded by org.jboss.modules.ModuleClassLoader

[felixbarny]

Our guard against Executors which expect a certain implementation type for the submitted Runnables doesn't work in this case.

apm-agent-java/apm-agent-plugins/apm-java-concurrent-plugin/src/main/java/co/elastic/apm/agent/concurrent/ExecutorInstrumentation.java

Lines 95 to 100 in 2ca66d6

	if (original != null && (exception instanceof ClassCastException || exception instanceof IllegalArgumentException)) {
	// seems like this executor expects a specific subtype of Callable
	runnable = original;
	// repeat only if submitting a task fails for the first time
	return excluded.add(thiz);
	} else {

This only guards against queues which don't accept a certain implementation.

However, the ManagedThreadPoolExecutor casts in the beforeExecute method on the executing thread.

https://github.com/eclipse-ee4j/concurrency-ri/blob/311305e2ed48c60d65592f2d22ee03e7fdd16c7a/src/main/java/org/glassfish/enterprise/concurrent/internal/ManagedThreadPoolExecutor.java#L81-L87

A workaround is to add (?-i)org.glassfish.enterprise.concurrent.internal.ManagedThreadPoolExecutor to classes_excluded_from_instrumentation

[felixbarny]

I have drafted a fix for this. Could you try out the attached snapshot build to see if it works for you? #596 (comment)

[mdindoffer]

I have drafted a fix for this. Could you try out the attached snapshot build to see if it works for you? #596 (comment)

Thanks for the quick response. Unfortunately it behaves the same way.

[felixbarny]

Hmm, that's strange. Is there a full stack trace available for this message?

SEVERE [org.glassfish.enterprise.concurrent] (EE-ManagedExecutorService-default-Thread-1) java.lang.ClassCastException: co.elastic.apm.agent.impl.async.ContextInScopeRunnableWrapper cannot be cast to org.glassfish.enterprise.concurrent.internal.ManagedFutureTask

[mdindoffer]

Well not in logs. But I've caught the stacktrace on exception breakpoint:

at org.glassfish.enterprise.concurrent.internal.ManagedThreadPoolExecutor.beforeExecute(ManagedThreadPoolExecutor.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)

[felixbarny]

Oh, right. It's not enough to exclude ManagedThreadPoolExecutor from instrumentation. This class is actually already not instrumented as we only instrument the execute and submit methods. These methods are inherited from the superclass java.util.concurrent.ThreadPoolExecutor.

A temporary workaround is to disable context propagation for thread pools altogether by setting disable_instrumentations=incubating,executor.

I'll think about a proper solution.

[cpmoore]

@felixbarny
Thanks for the workaround, I was facing the same problem in our environment with a java.util.concurrent.FutureTask class. The agent ended up causing the thread to die which resulted in a NullPointerException being thrown.

2019-04-18 08:21:55.571 [ERROR] 503a1248-8abf-4e52-852a-846cb3c0edb0 SVDRNETBRACDEV Search [Default Executor-thread-31] com.asg.mobius.view.core.search.processors.AbstractSearchProcessor:collectResultsWithNoCache - java.lang.NullPointerException
java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:133) ~[?:1.8.0]
        at java.util.concurrent.FutureTask.get(FutureTask.java:203) ~[?:1.8.0]
        at com.asg.mobius.view.core.common.security.execution.MobiusFutureTask.get(MobiusFutureTask.java:50) ~[common-security-8.0.8.jar:?]
        at com.asg.mobius.view.core.search.processors.AbstractSearchProcessor.collectResultsWithNoCache(AbstractSearchProcessor.java:173) ~[search-impl-8.0.8.jar:?]
        at com.asg.mobius.view.core.search.processors.AbstractSearchProcessor.executeTasks(AbstractSearchProcessor.java:156) ~[search-impl-8.0.8.jar:?]

[btsinfo]

he i had a problem with java agent in JBOSS, the log show error
(ActiveMQ-client-global-threads)) java.lang.ClassCastException: co.elastic.apm.agent.jdbc.helper.ConnectionMetaData cannot be cast to co.elastic.apm.agent.jdbc.helper.ConnectionMetaData

[eyalkoren]

@btsinfo this have been fixed in #1864 but not released yet.
Until it is released, you can use our latest published snapshot.