Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE in ConnectorXAResource.getResourceHandle causes connection pool problems #3025

Closed
escay opened this issue Aug 8, 2018 · 6 comments
Closed
Assignees
Labels
PR: TESTS REQUIRED PR Requires Tests to be merged
Milestone

Comments

@escay
Copy link

escay commented Aug 8, 2018

Description


I run very often into similar issues as described in #769, especially the situation described in issue #2310 seems to be the same as the one in my server.log file:

Caused by: javax.transaction.xa.XAException: com.sun.appserv.connectors.internal.api.PoolingException: java.lang.NullPointerException
	at com.sun.enterprise.resource.ConnectorXAResource.handleResourceException(ConnectorXAResource.java:116)
	at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:175)

Reproduction is very difficult but I can reproduce it often when:

  • I make a large number of Remote stateless bean calls using a standalone client.
  • The Remote stateless beans make a large number of calls internally to other stateless beans, which make a large number of calls to the database.
  • Most methods are annotated using SUPPORTS or REQUIRES, not REQUIRES_NEW.
  • Problem occurs on both Oracle 12.2.1.2 and Postgresql 9.2 using jdbc type 4 driver, it does not seem to be database version related.

Expected Outcome

No NPEs.
Remote bean calls that keep working.

Current Outcome

NPEs and the exception:

Caused by: javax.transaction.xa.XAException: com.sun.appserv.connectors.internal.api.PoolingException: java.lang.NullPointerException
	at com.sun.enterprise.resource.ConnectorXAResource.handleResourceException(ConnectorXAResource.java:116)
	at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:175)

Steps to reproduce (Only for bug reports)

I debugged Payara and added a breakpoint on NPE.
This eventually shows a NPE occurs in com.sun.enterprise.resource.ConnectorXAResource on line 255

In my case the values are:

  • beginTxIfNeeded = true
  • j2eetran = (JavaEETransactionImpl) 389 JavaEETransactionImpl: txId=25612 nonXAResource=null jtsTx=null localTxStatus=0 syncs=[com.sun.ejb.containers.ContainerSynchronization@447ec6a1, com.sun.ejb.containers.SimpleEjbResourceHandlerImpl@626aa694]
  • j2eetran.getNonXAResource() = null
  • h = null
  • localHandle_ is available and localHandle_.isShareable() is true

The handling of the 'h == null' situation is missing:

    private ResourceHandle getResourceHandle(boolean beginTxIfNeeded) throws PoolingException {
        try {
            ResourceHandle h = null;
            JavaEETransaction j2eetran = getCurrentTransaction();
            if (j2eetran == null) {      //Only if some thing is wrong with tx manager.
                h = localHandle_;        //Just return the local handle.
            } else {
                h = (ResourceHandle)j2eetran.getNonXAResource();
            //make sure that if local-tx resource is set as 'unshareable', only one resource
            //can be acquired. If the resource in question is not the one in transaction, fail
            if (!localHandle_.isShareable()){
                   if(h != localHandle_){
                        throw new ResourceAllocationException("Cannot use more than one local-tx resource in unshareable scope");
                    }
                }
            }
            if (beginTxIfNeeded && h.getResourceState().isUnenlisted()) {
                ManagedConnection mc = (ManagedConnection) h.getResource();
                // begin the local transaction if first time
                // this ManagedConnection is used in this JTA transaction
                mc.getLocalTransaction().begin();
            }
            return h;
        } catch (Exception ex) {
            _logger.log(Level.SEVERE, "poolmgr.system_exception", ex);
            throw new PoolingException(ex.toString(), ex);
        }
    }

As far as I can see this line of code was introduced in the fix for #769

https://github.com/payara/Payara/blame/189529b7fb3b00f282ab7c3315ce53f96a6f93a7/appserver/connectors/connectors-runtime/src/main/java/com/sun/enterprise/resource/ConnectorXAResource.java#L255

Perhaps the code should be:

            if (j2eetran == null || j2eetran.getNonXAResource() == null) {      //Only if some thing is wrong with tx manager.
                h = localHandle_;        //Just return the local handle.
            } else {
                h = (ResourceHandle)j2eetran.getNonXAResource();

To make it null pointer safe? But I do not have enough knowledge to understand why there would not be a j2eetran.getNonXAResource() is some cases. EJB Pool or database pool limit reached in an earlier stage?

The stacktrace at the moment the NPE occurs is:

"orb-thread-pool-1 (pool #1): worker-3"
	at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:255)
	at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandleAndBeginTxIfNeeded(ConnectorXAResource.java:232)
	at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:160)
	at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.delistResource(JavaEETransactionManagerSimplified.java:545)
	at com.sun.enterprise.resource.rm.ResourceManagerImpl.unregisterResource(ResourceManagerImpl.java:277)
	at com.sun.enterprise.resource.rm.ResourceManagerImpl.delistResource(ResourceManagerImpl.java:235)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:378)
	at com.sun.enterprise.resource.listener.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:77)
	at com.sun.gjc.spi.ManagedConnectionImpl.connectionClosed(ManagedConnectionImpl.java:860)
	at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:217)
	at com.sun.gjc.spi.jdbc40.ConnectionHolder40.close(ConnectionHolder40.java:608)
	at com.mycompany.database.settings.dbbase.MycompanySettingDbBase.getMycompanySetting(MycompanySettingDbBase.java:128)
	at sun.reflect.GeneratedMethodAccessor247.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy308.getMycompanySetting
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySettingValue(MycompanySettingInf.java:524)
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySettingValue(MycompanySettingInf.java:508)
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySetting(MycompanySettingInf.java:392)
	at sun.reflect.GeneratedMethodAccessor429.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy398.fillMycompanySetting
	at com.mycompany.settings.__EJB31_Generated__MycompanySettingInf__Intf____Bean__.fillMycompanySetting
	at com.mycompany.settings.SettingsBean.fillMycompanyConfiguration(SettingsBean.java:356)
	at com.mycompany.settings.SettingsBean.getDeviceAdapterConfiguration(SettingsBean.java:142)
	at sun.reflect.GeneratedMethodAccessor473.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy400.getDeviceAdapterConfiguration
	at com.mycompany.clientremote.platform.PlatformServiceBean.getDeviceAdapterConfiguration(PlatformServiceBean.java:262)
	at sun.reflect.GeneratedMethodAccessor472.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.mycompany.clientremote.common.abstractclasses.MycompanyClientRemoteBase.intercept(MycompanyClientRemoteBase.java:101)
	at sun.reflect.GeneratedMethodAccessor466.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.BeanAroundInvokeInterceptor.intercept(InterceptorManager.java:936)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)
	at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
	at com.sun.proxy.$Proxy568.getDeviceAdapterConfiguration
	at sun.reflect.GeneratedMethodAccessor471.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:143)
	at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:173)
	at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
	at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1565)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1441)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:946)
	at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:710)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:509)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2238)
	at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$TaskRunner.run(ThreadPoolImpl.java:189)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Eventually this NPE is handled in handleResourceException and the server.log shows the PoolingException: java.lang.NullPointerException:

javax.ejb.EJBException
	at com.sun.ejb.containers.EJBContainerTransactionManager.processSystemException(EJBContainerTransactionManager.java:750)
	at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:700)
	at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:505)
	at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4647)
	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2123)
	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2093)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy308.getMycompanySetting(Unknown Source)
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySettingValue(MycompanySettingInf.java:524)
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySettingValue(MycompanySettingInf.java:508)
	at com.mycompany.settings.MycompanySettingInf.fillMycompanySetting(MycompanySettingInf.java:392)
	at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy398.fillMycompanySetting(Unknown Source)
	at com.mycompany.settings.__EJB31_Generated__MycompanySettingInf__Intf____Bean__.fillMycompanySetting(Unknown Source)
	at com.mycompany.settings.SettingsBean.fillMycompanyConfiguration(SettingsBean.java:356)
	at com.mycompany.settings.SettingsBean.getDeviceAdapterConfiguration(SettingsBean.java:142)
	at sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
	at com.sun.proxy.$Proxy400.getDeviceAdapterConfiguration(Unknown Source)
	at com.mycompany.clientremote.platform.PlatformServiceBean.getDeviceAdapterConfiguration(PlatformServiceBean.java:262)
	at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.mycompany.clientremote.common.abstractclasses.MycompanyClientRemoteBase.intercept(MycompanyClientRemoteBase.java:101)
	at sun.reflect.GeneratedMethodAccessor466.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.BeanAroundInvokeInterceptor.intercept(InterceptorManager.java:936)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)
	at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
	at com.sun.proxy.$Proxy568.getDeviceAdapterConfiguration(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor471.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:143)
	at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:173)
	at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
	at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1565)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1441)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:946)
	at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:710)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:509)
	at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2238)
	at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$TaskRunner.run(ThreadPoolImpl.java:189)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Got exception during XAResource.end: javax.transaction.xa.XAException: com.sun.appserv.connectors.internal.api.PoolingException: java.lang.NullPointerException
	at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.delistResource(JavaEETransactionManagerSimplified.java:547)
	at com.sun.enterprise.resource.rm.ResourceManagerImpl.unregisterResource(ResourceManagerImpl.java:277)
	at com.sun.enterprise.resource.rm.ResourceManagerImpl.delistResource(ResourceManagerImpl.java:235)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:378)
	at com.sun.enterprise.resource.listener.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:77)
	at com.sun.gjc.spi.ManagedConnectionImpl.connectionClosed(ManagedConnectionImpl.java:860)
	at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:217)
	at com.sun.gjc.spi.jdbc40.ConnectionHolder40.close(ConnectionHolder40.java:608)
	at com.mycompany.database.settings.dbbase.MycompanySettingDbBase.getMycompanySetting(MycompanySettingDbBase.java:128)
	at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4867)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
	at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
	at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4839)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4827)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
	... 126 more
Caused by: javax.transaction.xa.XAException: com.sun.appserv.connectors.internal.api.PoolingException: java.lang.NullPointerException
	at com.sun.enterprise.resource.ConnectorXAResource.handleResourceException(ConnectorXAResource.java:116)
	at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:175)
	at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.delistResource(JavaEETransactionManagerSimplified.java:545)
	... 162 more

Environment

  • Payara Version: 5.182
  • Edition: Full
  • JDK Version: 1.8.0.182_64bit
  • Operating System: Windows 10
  • Database: Oracle 12.2.1.2 and Postgresql 9.2 using jdbc type 4 driver.
@escay
Copy link
Author

escay commented Aug 9, 2018

I tested changing the code to:

if (j2eetran == null || j2eetran.getNonXAResource() == null) {
                h = localHandle_;
            } else {
                h = (ResourceHandle)j2eetran.getNonXAResource();

I replaced the /glassfish/modules/connectors-runtime.jar and my remote bean calls keep working / I cannot reproduce the problem anymore.

This might not be the correct fix, but it avoids the NullPointerException and remote bean calls keep working.

@escay
Copy link
Author

escay commented Sep 3, 2018

I will try to create a test case to reproduce this. I cannot provide the current code. This could take a while.

@marcreichman
Copy link

@escay I was the poster of #2310. I actually have seen this issue go away since using more default connection pool settings, as what was recommended in my ticket. I'm not sure if it will help you, but I just wanted to add a data point. Here is my current connection pool setting (for Oracle):

<jdbc-connection-pool validation-table-name="TABLEVALIDATION"
						  connection-validation-method="custom-validation"
						  allow-non-component-callers="true"
						  statement-cache-size="200"
						  associate-with-thread="false"
						  statement-timeout-in-seconds="-1"
						  idle-timeout-in-seconds="0"
						  lazy-connection-association="false"
						  max-wait-time-in-millis="0"
						  connection-creation-retry-attempts="5"
						  validate-atmost-once-period-in-seconds="30"
						  lazy-connection-enlistment="false"
						  max-pool-size="96"
						  res-type="javax.sql.DataSource"
						  name="MyPool"
						  is-connection-validation-required="true"
						  validation-classname="org.glassfish.api.jdbc.validation.OracleConnectionValidation"
						  datasource-classname="oracle.jdbc.xa.client.OracleXADataSource">
		<property name="DataSourceName" value="OracleDataSource"/>
		<property name="ImplicitCachingEnabled" value="false"/>
		<property name="NetworkProtocol" value="tcp"/>

		<property name="DatabaseName" value="transitem"/>
		<property name="LoginTimeout" value="0"/>
		<property name="User" value="myuser"/>
		<property name="Password" value="mypass"/>
		<property name="URL" value="jdbc:oracle:thin:@localhost:1521:ORCL"/>
		<property name="PortNumber" value="1521"/>

		<property name="ExplicitCachingEnabled" value="false"/>
		<property name="dynamic-reconfiguration-wait-timeout-in-seconds" value="960"/>
		<property name="MaxStatements" value="0"/>
	</jdbc-connection-pool>

@escay
Copy link
Author

escay commented Nov 12, 2018

@marcreichman Thank you for the information. In the meantime I have created a reproduction scenario and created a support ticket at Payara directly using a support contract. They are able to reproduce the problem. I hope the cause of the problem can be found.

@smillidge
Copy link
Contributor

I believe this is fixed in 5.184

@smillidge smillidge added this to the Payara 5.184 milestone Dec 2, 2018
@escay
Copy link
Author

escay commented Dec 3, 2018

See #3344 for the commits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: TESTS REQUIRED PR Requires Tests to be merged
Projects
None yet
Development

No branches or pull requests

4 participants