[4.1.2.173] payara-embedded-all unable to start

[ctabin]

Description

---

When calling glassfish.start();, payara crashes because of a NoClassDefFoundError of the class fish/payara/nucleus/healthcheck/stuck/StuckThreadsStore.

Here is the full stacktrace obtained just after the startup:

INFO: Realm [myrealm] of classtype [com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm] successfully created.
Aug 14, 2017 5:32:04 PM org.glassfish.security.services.impl.authorization.AuthorizationServiceImpl initialize
INFO: Authorization Service has successfully initialized.
Aug 14, 2017 5:32:04 PM com.sun.enterprise.v3.services.impl.GrizzlyProxy start0
INFO: Grizzly Framework 2.3.31 started in: 15ms - bound to [/0.0.0.0:3700]
Aug 14, 2017 5:32:04 PM com.sun.enterprise.v3.server.AppServerStartup postStartupJob
INFO: Undefined Product Name - define product and version info in config/branding  0.0.0.0.0  (0) startup time : Embedded (376ms), startup services(765ms), total(1,141ms)
Aug 14, 2017 5:32:04 PM com.sun.enterprise.v3.server.AppServerStartup postStartupJob
SEVERE: Shutting down server due to startup exception
java.lang.NoClassDefFoundError: fish/payara/nucleus/healthcheck/stuck/StuckThreadsStore
	at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:111)
	at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:164)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:420)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:409)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:305)
	at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:125)
	at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:591)
	at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:542)
	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:786)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpAllTheWay.go(CurrentTaskFuture.java:585)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpAllTheWay.access$100(CurrentTaskFuture.java:467)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture.go(CurrentTaskFuture.java:157)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture.go(CurrentTaskFuture.java:152)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.proceedTo(AsyncRunLevelContext.java:535)
	at org.glassfish.hk2.runlevel.internal.RunLevelControllerImpl.proceedTo(RunLevelControllerImpl.java:72)
	at com.sun.enterprise.v3.server.AppServerStartup.proceedTo(AppServerStartup.java:556)
	at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:348)
	at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:234)
	at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:223)
	at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:81)
	at ch.saierp.sainet.embedded.server.SAINetServer.start(SAINetServer.java:586)
	at ch.saierp.sainet.testing.server.gui.MainFrame.startServer(MainFrame.java:209)
	at ch.saierp.sainet.testing.server.gui.MainFrame.access$1500(MainFrame.java:50)
	at ch.saierp.sainet.testing.server.gui.MainFrame$6.doInBackground(MainFrame.java:657)
	at ch.saierp.sainet.testing.server.gui.MainFrame$6.doInBackground(MainFrame.java:625)
	at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at javax.swing.SwingWorker.run(SwingWorker.java:334)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: fish.payara.nucleus.healthcheck.stuck.StuckThreadsStore
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 41 more

Aug 14, 2017 5:32:04 PM fish.payara.nucleus.notification.NotificationEventBus unregister
WARNING: Tried to unregisterfish.payara.notification.eventbus.core.CDIEventbusNotifierService@11744878, it may not have been previously registered
Aug 14, 2017 5:32:04 PM fish.payara.nucleus.notification.NotificationEventBus unregister
WARNING: Tried to unregisterfish.payara.nucleus.notification.log.LogNotifierService@5d60cb3d, it may not have been previously registered
Aug 14, 2017 5:32:04 PM fish.payara.nucleus.notification.NotificationEventBus unregister
WARNING: Tried to unregisterfish.payara.notification.eventbus.core.EventbusNotifierService@6ceb766f, it may not have been previously registered
Aug 14, 2017 5:32:04 PM com.sun.enterprise.v3.admin.StopServer doExecute
INFO: Server shutdown initiated
Aug 14, 2017 5:32:04 PM com.sun.enterprise.v3.server.AppServerStartup doStart
SEVERE: Failed to start, exiting
An error has occurred during startup.
org.glassfish.embeddable.GlassFishException: Server failed to start

Steps to reproduce (Only for bug reports)

I'm using this simple piece of code to start the server:

GlassFishProperties gfprops = new GlassFishProperties();
gfprops.setConfigFileURI(genDomainFile.toURI().toString());
        
BootstrapProperties bsprops = new BootstrapProperties();
GlassFishRuntime runtime = GlassFishRuntime.bootstrap(bsprops);
GlassFish glassfish = runtime.newGlassFish(gfprops);
glassfish.start(); //crash

After searching for this class, i found it here, but it is not packaged at all with the payara-embedded-all distribution. Moreover, I coudn't find it in any other payara distribution ?

IMHO there are two cases:

• either the class is really missing in the shipping of the version 4.1.2.173
• there is a configuration somewhere that references this class/module

It doesn't seems to be our code, since it works well in 4.1.1.171.1. However it also crashes in version 4.1.2.172 (but the reason is different and seems unrelated with this issue: NoClassDefFoundError: fish/payara/notification/eventbus/EventbusMessage).

Context (Optional)

We are using Payara to run test within a maven deployment, but also as a full embedded server. In both maven tests and the server, it crashes with the same error.

Tested with this dependency.

Note, there was no change in our code and it worked with Payara-4.1.1.171.1.

Environment

• Payara Version: 4.1.2.173
• Edition: embedded-full
• JDK Version: Oracle JDK 1.8.0_121 / OpenJDK 8u141
• Operating System: Linux Debian
• Database: MySQL

[mikecroft]

Yes, I think this is a packaging issue, we'll look into it. Thanks for the report!

[ctabin]

Thanks for the fix !
Is there any chance it will be backported to 4.1.2.173 ?

[mikecroft]

Backported fixes are part of our Enterprise and Professional tier support plans:
https://www.payara.fish/support

We make all fixes available to our customers monthly. You can either buy support online, or email [email protected] for more information.

[ctabin]

OK thanks for your quick response !

[fcaputo]

So this makes 173 not usable. You should not have released it.

[lprimak]

@fcaputo Unfortunately, we do not have the resources to check all the possible use cases.
Yes, this bug did slip though and we did not notice, we are in the process of enhancing the test suite to capture more use cases.

[Pandrex247]

We did actually release Payara Embedded 173.0.1 which should have fixed this issue.
It's not tagged on GitHub (which I'll fix in a bit), but it should be available from Maven central and our website (as 173).

[fcaputo]

Wow, that's fast. Thank you. Everything is working again!