Intermittent ClassNotFoundException for MBeanBuilder

[smillidge]

In the Payara-web distribution we are seeing intermittent exceptions below when restarting a server.

If Hazelcast is enabled this will cause the server to shutdown again as it tries to load MBeans.

[#|2015-01-16T18:40:08.437+0000|SEVERE|Payara 4.1||_ThreadID=22;_ThreadName=Thread-9;_TimeMillis=1421433608437;_LevelValue=1000;|
javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder: java.lang.ClassNotFoundException: com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder

Just running the start-domain command again often resolves the problem.

[smillidge]

Looks like this only occurs if you do a Ctrl-C and RAPID restart. We think it is file locking in the OSGI-Cache and not a real issue.

[smillidge]

Seems like this is a bigger problem than we thought.
We need to change when MBeans for Hazelcast are installed as there are problems at boot.

[smillidge]

Looks like this is due to the original classloader hack which is now no longer needed due to Hazelcast incorporating hazelcast/hazelcast#3922 into 3.4

[dmatej]

Behavior with 4.1.151:

1. When I only enable Hazelcast on clean new domain, restarts are correct
2. When I configure domain (Xmx, Xss, JDBC, JMS, realms, HTTP, HTTP thread pools, ...) and enable Hazelcast, restart always fails with this exception.
3. Then when I copy this domain to the second computer with faster HD and much more free memory, domain starts there without problem. Always.
4. Then when I restart the domain on original computer, it is without problem, (but the start is a bit slower). Always.
5. Then I stop the second domain and I'm again in state 2). And then steps 3-4 have same results.

Behavior with 4.1.152-SNAPSHOT with hack replaced is without problems! :)

diff --git a/nucleus/payara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java b/nucleus/payara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java
index 33d8ab4..4a49c4e 100644
--- a/nucleus/payara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java
+++ b/nucleus/payara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java
@@ -176,13 +176,9 @@ public class HazelcastCore implements EventListener {

     private void bootstrapHazelcast() {
         Config config = buildConfiguration();
-        // hack to prevent Hazelcast barfing on multiple classloaders for portable hooks during boot
-        ClassLoader tccl = Thread.currentThread().getContextClassLoader();
-        Thread.currentThread().setContextClassLoader(Hazelcast.class.getClassLoader());
         theInstance = Hazelcast.newHazelcastInstance(config);
         theInstance.getCluster().getLocalMember().setStringAttribute(INSTANCE_ATTRIBUTE, context.getInstanceName());
         hazelcastCachingProvider = HazelcastServerCachingProvider.createCachingProvider(theInstance);
-        Thread.currentThread().setContextClassLoader(tccl);
     }

     private void bindToJNDI() {