Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Report: Payara 5.2021.2 *does not start* / FISH-1297 #5192

Closed
pbelbin opened this issue Apr 10, 2021 · 18 comments · Fixed by #5193
Closed

Bug Report: Payara 5.2021.2 *does not start* / FISH-1297 #5192

pbelbin opened this issue Apr 10, 2021 · 18 comments · Fixed by #5193
Assignees
Labels
Status: Accepted Confirmed defect or accepted improvement to implement, issue has been escalated to Platform Dev Type: Bug Label issue as a bug defect

Comments

@pbelbin
Copy link

pbelbin commented Apr 10, 2021

Description


After creating a domain using asadmin create-domain das, trying to start it with asadmin start-domain das FAILS miserably.

The same result happens using the latest JDK 8 from Oracle, and the latest Zulu JDK 11.

Running on Centos 6.8 x86.

Expected Outcome

The domain should start.

Current Outcome

The domain fails to start and the output contains stack trace details.

Steps to reproduce

unpack the downloaded payara distribution zip file
configure payara5/glassfish/config/asenv.conf with AS_JAVA having location of java folder
navigate to payara5/glassfish/bin folder
issue command: ./asadmin create-domain das
issue command: ./asadmin start-domain das

see error output instead of result message indicating success.

In contrast, payara-5-2020.4.zip works just fine.

Environment

  • Distribution: Server Full Profile
  • JDK Version: 8 u281 Oracle, OpenJDK, Zulu
  • Operating System: Linux (centos 6.8)
@pbelbin pbelbin added the Type: Bug Label issue as a bug defect label Apr 10, 2021
@lprimak
Copy link
Contributor

lprimak commented Apr 10, 2021

There is not much to go on here. Can you paste the stack trace?

@pbelbin
Copy link
Author

pbelbin commented Apr 10, 2021

Here is a sample of what I see when it fails:

[root@s5 bin]# ./asadmin start-domain das
Waiting for das to start .......Error starting domain das.
The server exited prematurely with exit code 1.
Before it died, it produced the following output:

Launching Payara Server on Felix platform
Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishRuntime@4261a02d in service registry.



#!## LogManagerService.postConstruct : rootFolder=/opt/payara5/glassfish
#!## LogManagerService.postConstruct : templateDir=/opt/payara5/glassfish/lib/templates
#!## LogManagerService.postConstruct : src=/opt/payara5/glassfish/lib/templates/logging.properties
#!## LogManagerService.postConstruct : dest=/opt/payara5/glassfish/domains/das/config/logging.properties
Completed shutdown of Log manager service
Completed shutdown of GlassFish runtime
We are in non-embedded mode, so fish.payara.server.internal.core.glassfish [176] has nothing to do.
Apr 10, 2021 1:47:38 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner createBundleProvisioner
INFO: Create bundle provisioner class = class com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.
[#|2021-04-10T13:47:41.680-0500|INFO|Payara 5.2021.2|javax.enterprise.logging|_ThreadID=26;_ThreadName=RunLevelControllerThread-1618080461557;_TimeMillis=1618080461680;_LevelValue=800;_MessageID=NCLS-LOGGING-00009;|
  Running Payara Version: Payara Server  5.2021.2 #badassfish (build 3387)|#]

[#|2021-04-10T13:47:41.684-0500|INFO|Payara 5.2021.2|javax.enterprise.logging|_ThreadID=26;_ThreadName=RunLevelControllerThread-1618080461557;_TimeMillis=1618080461684;_LevelValue=800;_MessageID=NCLS-LOGGING-00010;|
  Server log file is using Formatter class: com.sun.enterprise.server.logging.ODLLogFormatter|#]

[#|2021-04-10T13:47:42.021-0500|INFO|Payara 5.2021.2|org.hibernate.validator.internal.util.Version|_ThreadID=28;_ThreadName=RunLevelControllerThread-1618080461563;_TimeMillis=1618080462021;_LevelValue=800;|
  HV000001: Hibernate Validator 6.1.5.Final|#]

[#|2021-04-10T13:47:42.681-0500|INFO|Payara 5.2021.2|org.glassfish.ha.store.spi.BackingStoreFactoryRegistry|_ThreadID=28;_ThreadName=RunLevelControllerThread-1618080461563;_TimeMillis=1618080462681;_LevelValue=800;|
  Registered fish.payara.ha.hazelcast.store.HazelcastBackingStoreFactoryProxy for persistence-type = hazelcast in BackingStoreFactoryRegistry|#]

[#|2021-04-10T13:47:44.274-0500|SEVERE|Payara 5.2021.2|com.hazelcast.internal.cluster.impl.DiscoveryJoiner|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464274;_LevelValue=1000;|
  [216.224.176.36]:4900 [development] [4.1] inetAddress can't be null
java.lang.NullPointerException: inetAddress can't be null
        at com.hazelcast.internal.util.Preconditions.checkNotNull(Preconditions.java:59)
        at com.hazelcast.cluster.Address.<init>(Address.java:72)
        at com.hazelcast.cluster.Address.<init>(Address.java:56)
        at fish.payara.nucleus.hazelcast.DomainDiscoveryService.discoverNodes(DomainDiscoveryService.java:176)
        at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:69)
        at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:58)
        at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:136)
        at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:96)
        at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)
        at com.hazelcast.instance.impl.Node.join(Node.java:808)
        at com.hazelcast.instance.impl.Node.start(Node.java:471)
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:123)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.275-0500|SEVERE|Payara 5.2021.2|com.hazelcast.instance.impl.Node|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464275;_LevelValue=1000;|
  [216.224.176.36]:4900 [development] [4.1] Could not join cluster. Shutting down now!|#]

[#|2021-04-10T13:47:44.278-0500|WARNING|Payara 5.2021.2|com.hazelcast.instance.impl.Node|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464278;_LevelValue=900;|
  [216.224.176.36]:4900 [development] [4.1] Terminating forcefully...|#]

[#|2021-04-10T13:47:44.301-0500|WARNING|Payara 5.2021.2|com.hazelcast.internal.util.PhoneHome|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464301;_LevelValue=900;|
  [216.224.176.36]:4900 [development] [4.1] Could not schedule phone home task! Most probably Hazelcast failed to start.|#]

[#|2021-04-10T13:47:44.305-0500|INFO|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464305;_LevelValue=800;_MessageID=NCLS-CORE-00015;|
  Shutdown requested
MultiException stack 1 of 2
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.305-0500|SEVERE|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=25;_ThreadName=RunLevelControllerThread-1618080461556;_TimeMillis=1618080464305;_LevelValue=1000;_MessageID=NCLS-CORE-00016;|
  Startup service failed to start
MultiException stack 1 of 2
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.306-0500|INFO|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=26;_ThreadName=RunLevelControllerThread-1618080461557;_TimeMillis=1618080464306;_LevelValue=800;_MessageID=NCLS-CORE-00015;|
  Shutdown requested
MultiException stack 1 of 9
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 3 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.nucleus.requesttracing.RequestTracingService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 4 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.nucleus.requesttracing.RequestTracingService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 5 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 6 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 7 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on com.sun.enterprise.v3.server.ApplicationLifecycle
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 8 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of com.sun.enterprise.v3.server.ApplicationLoaderService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 9 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on com.sun.enterprise.v3.server.ApplicationLoaderService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.309-0500|SEVERE|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=26;_ThreadName=RunLevelControllerThread-1618080461557;_TimeMillis=1618080464309;_LevelValue=1000;_MessageID=NCLS-CORE-00016;|
  Startup service failed to start
MultiException stack 1 of 9
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 3 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.nucleus.requesttracing.RequestTracingService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 4 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.nucleus.requesttracing.RequestTracingService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 5 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 6 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 7 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on com.sun.enterprise.v3.server.ApplicationLifecycle
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 8 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of com.sun.enterprise.v3.server.ApplicationLoaderService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 9 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on com.sun.enterprise.v3.server.ApplicationLoaderService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.324-0500|INFO|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=27;_ThreadName=RunLevelControllerThread-1618080461563;_TimeMillis=1618080464324;_LevelValue=800;_MessageID=NCLS-CORE-00087;|
  Grizzly 2.4.4 started in: 1,742ms - bound to [http-listener-1:8080, http-listener-2:8181, admin-listener:4848]|#]

[#|2021-04-10T13:47:44.402-0500|INFO|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=28;_ThreadName=RunLevelControllerThread-1618080461563;_TimeMillis=1618080464402;_LevelValue=800;_MessageID=NCLS-CORE-00101;|
  Network Listener iiop-service started in: 8ms - bound to [/0.0.0.0:3700]|#]

[#|2021-04-10T13:47:44.416-0500|SEVERE|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1618080464416;_LevelValue=1000;_MessageID=NCLS-CORE-00014;|
  Shutdown required
MultiException stack 1 of 2
MultiException stack 1 of 2
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
MultiException stack 1 of 9
java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
        at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
        at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
        at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:505)
        at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:240)
        at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.config.ClusteredConfig
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 3 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.nucleus.requesttracing.RequestTracingService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 4 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.nucleus.requesttracing.RequestTracingService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
        at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
        at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
        at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:671)
        at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:593)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 5 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 6 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on fish.payara.monitoring.runtime.MonitoringConsoleRuntimeImpl
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1482)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:800)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServices(ServiceLocatorImpl.java:788)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.postConstruct(ApplicationLifecycle.java:229)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 7 of 9
java.lang.IllegalStateException: Unable to perform operation: post construct on com.sun.enterprise.v3.server.ApplicationLifecycle
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:680)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 8 of 9
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of com.sun.enterprise.v3.server.ApplicationLoaderService errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 9 of 9
java.lang.IllegalStateException: Unable to perform operation: resolve on com.sun.enterprise.v3.server.ApplicationLoaderService
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
|#]

[#|2021-04-10T13:47:44.419-0500|SEVERE|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1618080464419;_LevelValue=1000;|
  Failed to start, exiting|#]

[#|2021-04-10T13:47:44.422-0500|INFO|Payara 5.2021.2||_ThreadID=1;_ThreadName=main;_TimeMillis=1618080464422;_LevelValue=800;|
  FileMonitoring shutdown|#]

[#|2021-04-10T13:47:44.492-0500|INFO|Payara 5.2021.2|javax.enterprise.system.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1618080464492;_LevelValue=800;_MessageID=NCLS-CORE-00013;|
  Shutdown procedure finished|#]

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:109)
        at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54)
Caused by: org.glassfish.embeddable.GlassFishException: Server failed to start
        at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:247)
        at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:226)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:84)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
        at com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl.start(EmbeddedOSGiGlassFishImpl.java:75)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
        at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:130)
        ... 6 more

Command start-domain failed.
[root@s5 bin]#

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

From what I can see, this is an issue with network not being configured properly.
Due to the way Hazelcast 4+ works, there needs to be a non-loopback network interface besides localhost network
operational on the machine for Hazelcast to work and Payara to start

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

This behavior can probably be improved thought to make it more lenient to network configurations though.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

On that machine, if I 'ping localhost', it happily reports successful results hitting '127.0.0.1'.

I also see in the /etc/hosts, that it has entries there for:

127.0.0.1 localhost
::1 localhost

And, there is a single non-loopback IP interface functional here too, via ifconfig -a

I do notice that the single actual interface only has an IPv4 address. I wonder if not having an available IPv6 address is contributing here?

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

Can you paste complete output of “ifconfig -a” ?
You can X out actual IP addresses if you wish

@lprimak lprimak added the Status: Accepted Confirmed defect or accepted improvement to implement, issue has been escalated to Platform Dev label Apr 11, 2021
@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

here you go.

The only change I have made is to tweak the IP address details of venet0:0. Everything else is exactly as it is.

[root@s5 bin]# ifconfig -a
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:377471343 errors:0 dropped:0 overruns:0 frame:0
          TX packets:377471343 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:228618122137 (212.9 GiB)  TX bytes:228618122137 (212.9 GiB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:127.0.0.1  P-t-P:127.0.0.1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:48248129 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42172251 errors:0 dropped:31 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15873120025 (14.7 GiB)  TX bytes:9191349074 (8.5 GiB)

venet0:0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:200.100.50.36  P-t-P:200.100.50.36  Bcast:200.100.50.255  Mask:255.255.255.0
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1

[root@s5 bin]#

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

I have been able now to get it to start, but, in order to do that, I have to modify domain.xml so that the hazelcast config is disabled entirely.

I have not yet found a way to have hazelcast enabled, and for it to start.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

I did also try adding to domain.xml:

 <jvm-options>-Djava.net.preferIPv4Stack=true</jvm-options>

but this did not make any difference.

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

Ah, the issue is that the only true (non-localhost) network is venet0:0 and it's a virtual interface.
This is why Payara is having trouble determining 'connectable' IP address. There needs to be a real network
with the real IP address for it to work. I am working on a fix for this so it will at least start with localhost in configuratino in this case.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

Hmmm. Yeah, this is a 'VPS' (virtual machine) on a hosting provider, so, it only has whatever it has. I imagine this would be a very common situation these days with hosted virtual machines.

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

Apparently not. First time anyone has seen this.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

Ok. Well, I guess this has highlighted some kind of deficiency in the testing scenarios. I appreciate the effort in getting a fix out for it.

@lprimak
Copy link
Contributor

lprimak commented Apr 11, 2021

Which provider are you using? This is certainly not the case with AWS or Azure.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

This particular one is on 'myhosting.com', but, let me check one of my other providers, and see what it looks like there too.

@pbelbin
Copy link
Author

pbelbin commented Apr 11, 2021

Hmmm. Yeah, another VPS provider I have, has interfaces that appear like non-virtual interfaces, I suppose:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 200.100.40.56  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 0a:30:56:41:bc:b9  txqueuelen 1000  (Ethernet)
        RX packets 2878813  bytes 1890772612 (1.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1785982  bytes 1424699733 (1.4 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 337  bytes 39579 (39.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 337  bytes 39579 (39.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@vmi534016:~#

@pbelbin
Copy link
Author

pbelbin commented Apr 13, 2021

any word of a fix?

@AlanRoth AlanRoth changed the title Bug Report: Payara 5.2021.2 *does not start* Bug Report: Payara 5.2021.2 *does not start* / FISH-1297 Apr 13, 2021
@pbelbin
Copy link
Author

pbelbin commented Apr 13, 2021

Cool. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Accepted Confirmed defect or accepted improvement to implement, issue has been escalated to Platform Dev Type: Bug Label issue as a bug defect
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants