Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot load Hazelcast and use Cache on new Payara with JDK8 #196

Closed
dmatej opened this issue Feb 5, 2015 · 27 comments
Closed

Cannot load Hazelcast and use Cache on new Payara with JDK8 #196

dmatej opened this issue Feb 5, 2015 · 27 comments

Comments

@dmatej
Copy link
Contributor

dmatej commented Feb 5, 2015

I don't know if it is a bug of Payara, Hazelcast or JDK, but in unit test CachingProvider.class.isAssignableFrom(HazelcastCachingProvider.class) returns true. Following code is placed in a library deployed to Payara in EAR. Library is then used from EAR's EJB module.

System used: Kubuntu Linux 64bit
JDK used: OpenJDK 1.8.0u40b12, 1.8.0u40b23, Oracle JDK 1.8.0u31 - always same problem.
Code used:

    final CachingProvider cachingProvider = Caching.getCachingProvider();

Problematic code in Cache API (javax.cache.Caching):

          Class<?> clazz = classLoader.loadClass(fullyQualifiedClassName);
          if (CachingProvider.class.isAssignableFrom(clazz)) {
            return ((Class<CachingProvider>) clazz).newInstance();
          } else {
            throw new CacheException("The specified class [" + fullyQualifiedClassName + "] is not a CachingProvider");
          }
Caused by: javax.cache.CacheException: Failed to load the CachingProvider [com.hazelcast.cache.HazelcastCachingProvider]
        at javax.cache.Caching$CachingProviderRegistry.loadCachingProvider(Caching.java:492)
        at javax.cache.Caching$CachingProviderRegistry.getCachingProviders(Caching.java:429)
        at javax.cache.Caching$CachingProviderRegistry.getCachingProvider(Caching.java:370)
        at javax.cache.Caching$CachingProviderRegistry.getCachingProvider(Caching.java:351)
        at javax.cache.Caching.getCachingProvider(Caching.java:142)
        at cz.i.xxxxx
        ... 71 more
Caused by: javax.cache.CacheException: The specified class [com.hazelcast.cache.HazelcastCachingProvider] is not a CachingProvider
        at javax.cache.Caching$CachingProviderRegistry.loadCachingProvider(Caching.java:489)
        ... 85 more
@jerrinot
Copy link
Contributor

jerrinot commented Feb 5, 2015

Hi David,

can you double-check your EAR doesn't contain a JAR with JCache API or Hazelcast? I's just a shot in the dark, but it's worth checking.

Is it working when you let Payara to inject the CachingProvider to your bean via CDI @Inject instead?

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

No, Hazelcast is not present in EAR file. The second question is only for experiment, because this class is not a bean. But I will try it in some bean.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

CacheProvider is not injected at all (tried servlet, stateless bean, message listener) - it is null and no error is reported, except MessageListener, where it reported "unsatisfied ependency" to server.log.

Injecting CacheManager (see jsr107/jsr107spec#208 ) causes stacktrace even in stateless bean (other beans are injected correctly):

  @Inject
  private CacheManager cacheManager;
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB
        at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:435)
...
Caused by: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB
        at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:700)
        at com.sun.ejb.containers.util.pool.NonBlockingPool.getObject(NonBlockingPool.java:246)
        at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:430)
        ... 79 more
Caused by: javax.ejb.CreateException: Could not create stateless EJB
        at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:514)
        at com.sun.ejb.containers.StatelessSessionContainer.access$000(StatelessSessionContainer.java:97)
        at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:698)
        ... 81 more
Caused by: java.lang.NullPointerException
        at fish.payara.cdi.jsr107.JSR107Producer.getCacheManager(JSR107Producer.java:50)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.jboss.weld.injection.MethodInjectionPoint.invokeOnInstanceWithSpecialValue(MethodInjectionPoint.java:90)
...

I have no idea what to do.

@smillidge
Copy link
Contributor

I can't reproduce with our test JCache application running on JDK1.8. OTOH all our test applications are Servlets and session beans in wars. Do you have a simple test case we can review as we may not have captured your specific deployment scenario in testing.

I do know, which I note is missing from our documentation. Is that you must copy payara-jsr107.jar from glassfish/modules to domains/domain1/lib/applibs and reference it as a library during deployment to get injection to work in an EJB Session Bean. This is part of the EJB spec.

screenshot from 2015-02-05 12 57 07

@smillidge
Copy link
Contributor

Also dumb question I know.
Is Hazelcast enabled on the instances and do you get the messages and can you see the cluster members in the admin console;
[2015-02-05T12:50:09.774+0000] [Payara 4.1] [INFO] [] [fish.payara.nucleus.hazelcast.HazelcastCore] [tid: _ThreadID=54 _ThreadName=admin-listener(3)] [timeMillis: 1423140609774] [levelValue: 800] [[
Hazelcast Instance Bound to JNDI at payara/Hazelcast]]
[2015-02-05T12:50:09.774+0000] [Payara 4.1] [INFO] [] [fish.payara.nucleus.hazelcast.HazelcastCore] [tid: _ThreadID=54 _ThreadName=admin-listener(3)] [timeMillis: 1423140609774] [levelValue: 800] [[
JSR107 Caching Provider Bound to JNDI at payara/CachingProvider]
[2015-02-05T12:50:09.775+0000] [Payara 4.1] [INFO] [] [fish.payara.nucleus.hazelcast.HazelcastCore] [tid: _ThreadID=54 _ThreadName=admin-listener(3)] [timeMillis: 1423140609775] [levelValue: 800] [[
JSR107 Default Cache Manager Bound to JNDI at payara/CacheManager]]

@smillidge
Copy link
Contributor

Also the discussions in jsr107/jsr107spec#208 are NOT in the JSR107 spec as is but are deferred to the next JCache release.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Injection to bean works, but it is needed to enable Hazelcast in the server configuration also with "Dynamic" checked. Then comes another problem - server cannot be restarted, start ends with stopping the instance and this stacktrace:

  Shutdown required
MultiException stack 1 of 1
MultiException stack 1 of 2
javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder: java.lang.ClassNotFoundException: com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
        at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:503)
        at javax.management.MBeanServerFactory.getNewMBeanServerBuilder(MBeanServerFactory.java:539)
        at javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:316)
        at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:231)
        at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:192)
        at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:468)
        at com.hazelcast.jmx.ManagementService.<init>(ManagementService.java:74)
        at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:128)
        at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:153)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:136)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:112)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:182)
        at fish.payara.nucleus.hazelcast.HazelcastCore.postConstruct(HazelcastCore.java:80)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1041)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:335)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:377)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:471)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:227)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:84)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2064)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:105)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1223)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.felix.framework.BundleWiringImpl.doImplicitBootDelegation(BundleWiringImpl.java:1738)
        at org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1675)
        at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1511)
        at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
        at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at javax.management.MBeanServerFactory.loadBuilderClass(MBeanServerFactory.java:446)
        at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:488)
        ... 31 more
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.hazelcast.HazelcastCore
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:395)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:471)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:227)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:84)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2064)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:105)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1223)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
]]

@smillidge
Copy link
Contributor

Clear out your osgi-cache.

@smillidge
Copy link
Contributor

We have seen this intermittently with a Ctrl-C and immediate restart
#154
I thought it had gone away :-(
Looks like we may have to lazy load the Hazelcast MBeans to prevent it for good.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Removing the osgi-cache does not help :(
Startup fails even with all ear applications undeployed. When I disable the Hazelcast by setting enabled="false" in domain.xml, it starts.
When I tried to copy the payara-jsr107.jar to applibs and did a clean deploy, it caused another stacktrace when accessing the original code instead of injection (Caching.getCachingProvider()):

javax.cache.CacheException: Multiple CachingProviders have been configured when only a single CachingProvider is expected
        at javax.cache.Caching$CachingProviderRegistry.getCachingProvider(Caching.java:376)
        at javax.cache.Caching.getCachingProvider(Caching.java:160)

Stopping and starting the domain is impossible without disabling hazelcast via domain.xml.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Funny is that this exception I got even with Hazelcast disabled (I know, it's another mechanism, but ...).

@smillidge
Copy link
Contributor

Do you have a test case EAR file we can inviestigate with?

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Now I used this code and I got "multiple cachingproviders" error again ...

Caching.getCachingProvider(Thread.currentThread().getContextClassLoader());

But then I used your hack from HazelcastCore, it inspired me and I tried a bit of "shotgun debugging":

        // hack to prevent Hazelcast barfing on multiple classloaders for portable hooks during boot
        ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(Hazelcast.class.getClassLoader());

But this is not possible from an application:

Caused by: java.lang.LinkageError: javax/cache/spi/CachingProvider

Another attempt:

Caching.getCachingProvider(Hazelcast.class.getClassLoader());
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of org/glassfish/javaee/full/deployment/EarLibClassLoader) previously initiated loading for a different type with name "javax/cache/spi/CachingProvider"

And once more ...

Caching.getCachingProvider(CachingProvider.class.getClassLoader());
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "javax.cache.Caching.getCachingProvider(Ljava/lang/ClassLoader;)Ljavax/cache/spi/CachingProvider;" the class loader (instance of org/glassfish/javaee/full/deployment/EarLibClassLoader) of the current class, cz/i/xxx/CodeDao, and the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) for the method's defining class, javax/cache/Caching, have different Class objects for the type javax/cache/spi/CachingProvider used in the signature

And surprise ...

Caching.getCachingProvider(Cache.class.getClassLoader());
javax.cache.CacheException: No CachingProviders have been configured

But this behavior is really weird:
This works:

HazelcastServerCachingProvider provider  = new HazelcastServerCachingProvider();
provider.getCacheManager();

And this causes exception:

CachingProvider provider  = new HazelcastServerCachingProvider();
provider.getCacheManager();
java.lang.IncompatibleClassChangeError: Class com.hazelcast.cache.HazelcastCachingProvider does not implement the requested interface javax.cache.spi.CachingProvider

This seems that CachingProvider was compiled after HazelcastCachingProvider, but - then it should not work in any case. But when enabling the Hazelcast dynamically via appserver GUI it works without problems ... until restart of appserver. I don't get it.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

I will try to create some.

@smillidge
Copy link
Contributor

What is your deployment structure and which jar in your deployment are you trying to load the caching provider in?

Can you access the caching provider from JNDI?

@smillidge
Copy link
Contributor

I have no problem deploying this https://github.com/smillidge/JCacheTestEAR

As long as I ensure I reference payara-jsr107.jar as a libary
screenshot from 2015-02-05 20 24 14

If I then reference the servlet I get hex System.currentTimeMillis which doesn't change (as it is cached) and I get the System.out.println
[#|2015-02-05T20:24:26.003+0000|INFO|Payara 4.1|javax.enterprise.system.core|_ThreadID=49;_ThreadName=admin-listener(2);_TimeMillis=1423167866003;_LevelValue=800;|
JCacheEarTest-ear-4.1.152-SNAPSHOT was successfully deployed in 966 milliseconds.|#]

[#|2015-02-05T20:24:37.873+0000|INFO|Payara 4.1||_ThreadID=34;_ThreadName=Thread-9;_TimeMillis=1423167877873;_LevelValue=800;|
HazelcastServerCachingProvider{hazelcastInstance=HazelcastInstance{name='glassfish-web.server', node=Address[192.168.79.136]:5900}}|#]

[#|2015-02-05T20:24:37.873+0000|INFO|Payara 4.1||_ThreadID=34;_ThreadName=Thread-9;_TimeMillis=1423167877873;_LevelValue=800;|
HazelcastCacheManager{hazelcastInstance=HazelcastInstance{name='glassfish-web.server', node=Address[192.168.79.136]:5900}, cachingProvider=HazelcastServerCachingProvider{hazelcastInstance=HazelcastInstance{name='glassfish-web.server', node=Address[192.168.79.136]:5900}}}|#]

[#|2015-02-05T20:24:37.873+0000|INFO|Payara 4.1||_ThreadID=34;_ThreadName=Thread-9;_TimeMillis=1423167877873;_LevelValue=800;|
com.hazelcast.cache.impl.CacheProxy@3bcf8db0|#]
screenshot from 2015-02-05 20 24 46

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Structure is standard:
xxx.ear/lib/.jar (libraries, no JEE interfaces, no libraries that are in Payara, except SLF4J)
xxx.ear/
.jar (EJB modules)
xxx.ear/.war (WAR modules)
xxx.ear/META-INF/

The caching provider is loaded by one of libraries in the EAR.
On this is machine one restart from ten was successful. Now I copied the JDK8 and application server (no EARs deployed) to another notebook and there it starts without problems.
These problems with restarts - it seems the port remained somehow blocked by some deleted instance (and killed process) or something like that. Last message in log is "Hazelcast JMX agent enabled." and then it is folowed by "Terminating forcefully...". If it would be alright, there comes "Address [192.168.1.2]:5900 is STARTED"

Now on my notebook I have changed port from 5900 to 5950. Hurray, domain started, but instead one port I got two and I don't understand how, but the first item is the second notebook. No, I don't have domain configured as cluster ...:

Members [2] {
        Member [192.168.0.190]:5900
        Member [192.26.1.2]:5950 this
}
]]

Okay, now the JNDI ...

@smillidge
Copy link
Contributor

If you are loading the CachingProvider directly in the jar using the api Caching.getCachingProvider() you will get a different CachingProvider instance than the Payara in-built one, due to classloading. You need to inject via CDI, or lookup via JNDI to consistently get the correct CachingProvider instance.

Hazelcast JCache cluster is independent of the Payara cluster and is determined by the multicast Group and multicast port. Therefore the two servers will form a JCache cluster even across separate Payara domains if the multicast settings are the same.

@dmatej
Copy link
Contributor Author

dmatej commented Feb 5, 2015

Okay, if the server starts (asadmin stop-domain, wait until the process really stopped after asadmin reported it stopped, and run asadmin start-domain), then JNDI works perfectly. If I run asadmin restart-domain, Hazelcast does not start or starts broken (see previous stacktraces).

Thanks for explanation, now everything seems logical, except one thing - simply, this is not true:

java.lang.IncompatibleClassChangeError: Class com.hazelcast.cache.HazelcastCachingProvider does not implement the requested interface javax.cache.spi.CachingProvider

@smillidge
Copy link
Contributor

I think the restart bug is due to the classloader hack used to get around a problem in Hazelcast. This is no longer needed in Hazelcast 3.4. Removing the hack seems at initial first sight to solve the restart domain intermittent failure.

See #154 for tracking the restart problem

@trajano
Copy link

trajano commented Aug 14, 2015

Perhaps an upgrade to Hazelcast 3.5 would help as well. I had an app that worked on Glassfish but was using Hazelcast embedded in the application as JCache is not part of the Java EE standard yet. Having Hazelcast be a part of the core JARs for the application server can introduce issues.

@trajano
Copy link

trajano commented Aug 14, 2015

specifically mine on JDK8 is

Caused by: java.util.ServiceConfigurationError: javax.cache.spi.CachingProvider: Provider com.hazelcast.cache.HazelcastCachingProvider not a subtype

and on JDK7

Caused by: java.lang.ClassCastException: Cannot cast com.hazelcast.cache.HazelcastCachingProvider to javax.cache.spi.CachingProvider

I do use the jcache-1.0.0 dependency so that may add to the issue.

@albanf
Copy link

albanf commented Jun 7, 2017

I am also getting this error with Payara 4.1.2.172.
The cache api is duplicated in the modules directory, in cache-api.jar and jsr107-repackaged.jar

@parml
Copy link

parml commented Jul 12, 2017

@trajano Were you able to find a solution? I am getting that same error message on JDK8.
Removing cache-1.0.0 dependency from my war file does not fix the issue.
I'm using 4.1.2.172

@albanf
Copy link

albanf commented Jul 25, 2017

You also need to remove cache-api.jar from the Payara installation (from glassfish/modules/) and then clear the OSGI cache directory (rm -rf glassfish/domains/domain1/osgi-cache/felix/), then start your domain.

@lprimak
Copy link
Contributor

lprimak commented Jul 25, 2017

@smillidge
Copy link
Contributor

cache-api and jsr107-repackaged should not both be in the modules directory this is a bug on our side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants