FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks #6848

lprimak · 2024-07-28T09:47:27Z

Description

Bugfix. Application Deployment failure results Class Loader leak and extra error messages.
The root cause is the message stops undeployment process from proceeding further.
This is now fixed and leaks are removed.

[#|2024-07-28T04:01:23.573-0500|SEVERE|Payara 6.2024.7|jakarta.enterprise.web.core|_ThreadID=97;_ThreadName=admin-thread-pool::admin-listener(1);_TimeMillis=1722157283573;_LevelValue=1000;_MessageID=AS-WEB-CORE-00175;|
  Exception during cleanup after start failed
org.apache.catalina.LifecycleException: Manager has not yet been started
	at org.apache.catalina.session.StandardManager.stop(StandardManager.java:872)
	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:5954)
	at com.sun.enterprise.web.WebModule.stop(WebModule.java:648)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:5769)
	at com.sun.enterprise.web.WebModule.start(WebModule.java:619)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:982)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:965)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694)
	at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1813)
	at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1565)
	at com.sun.enterprise.web.WebApplication.start(WebApplication.java:107)
	at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123)
	at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:292)
	at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:361)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:633)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:574)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:453)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:453)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
	at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
	at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
	at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
	at java.base/java.lang.Thread.run(Thread.java:1570)
|#]

Important Info

Testing

Testing Performed

Using ClassloaderDataAPI to test for leaks

…ure, and leaking ClassLoader resources in the process

lprimak · 2024-08-03T05:05:20Z

@Pandrex247 This one's ready to go :)

Pandrex247 · 2024-08-06T09:35:40Z

This won't make the imminent release as we've just frozen for it, but I Should™ get round to reviewing it for the next one.

Pandrex247

Do you have any kind of reproducer for the scenario you're attempting to address here?

I can easily see that the LifecycleException being thrown will cause it to essentially skip all remaining unloading, but the force loading of JSP during exception handling and only logging undeployment errors if there's no deployment context seem a bit odd to me and I'd like to debug it.

appserver/web/web-core/src/main/java/org/apache/catalina/core/StandardContext.java

Pandrex247 · 2024-08-20T14:07:49Z

appserver/web/web-glue/src/main/java/com/sun/enterprise/web/WebContainer.java

@@ -2058,7 +2061,7 @@ public void unloadWebModule(String contextRoot, String appName, String virtualSe
            }
        }

-        if (!hasBeenUndeployed) {
+        if (!hasBeenUndeployed && deployment.getCurrentDeploymentContext() == null) {


I don't understand what this meant to be addressing.
&& operator means you're restricting this log message down to an even more specific case.

Currently, this message is printed during deployment, and that's an error.
The message is intended to be printed during undeployment only.
This change will check if the deployment process is running, and if it is, it will inhibit this message.

The result is that the message will only be printed during "true" undeployment as it was intended to be originally.

lprimak · 2024-08-21T06:07:38Z

Pretty much any application that fails deployment will be a valid reproducer for this

Here is the specific one I was testing with: https://github.com/flowlogix/shiro-env-vars

…StandardContext.java Co-authored-by: Andrew Pielage <[email protected]>

Pandrex247

Thanks!

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks

lprimak force-pushed the failed-deployment-cleanup branch from 3d791b2 to f3f7abf Compare August 2, 2024 02:48

bugfix: prevent deployment failures from causing an undeployment fail…

f899f65

…ure, and leaking ClassLoader resources in the process

lprimak force-pushed the failed-deployment-cleanup branch from f3f7abf to f899f65 Compare August 3, 2024 05:00

lprimak marked this pull request as ready for review August 3, 2024 05:01

lprimak changed the title ~~bugfix: prevent deployment failures from causing an undeployment failure~~ bugfix: prevent deployment failures from causing an undeployment failure and leaks Aug 3, 2024

Pandrex247 changed the title ~~bugfix: prevent deployment failures from causing an undeployment failure and leaks~~ FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks Aug 6, 2024

Pandrex247 reviewed Aug 20, 2024

View reviewed changes

Update appserver/web/web-core/src/main/java/org/apache/catalina/core/…

1e70b7d

…StandardContext.java Co-authored-by: Andrew Pielage <[email protected]>

Pandrex247 approved these changes Aug 21, 2024

View reviewed changes

Pandrex247 merged commit d7b7c98 into payara:main Aug 21, 2024
1 check passed

lprimak deleted the failed-deployment-cleanup branch August 21, 2024 10:27

Pandrex247 added a commit to Pandrex247/Payara that referenced this pull request Sep 16, 2024

Merge pull request payara#6848 from flowlogix/failed-deployment-cleanup

e1f837b

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks #6848

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks #6848

lprimak commented Jul 28, 2024 •

edited

Loading

lprimak commented Aug 3, 2024

Pandrex247 commented Aug 6, 2024

Pandrex247 left a comment

Pandrex247 Aug 20, 2024

lprimak Aug 21, 2024

lprimak commented Aug 21, 2024 •

edited

Loading

Pandrex247 left a comment

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks #6848

FISH-9171 bugfix: prevent deployment failures from causing an undeployment failure and leaks #6848

Conversation

lprimak commented Jul 28, 2024 • edited Loading

Description

Important Info

Testing

Testing Performed

lprimak commented Aug 3, 2024

Pandrex247 commented Aug 6, 2024

Pandrex247 left a comment

Choose a reason for hiding this comment

Pandrex247 Aug 20, 2024

Choose a reason for hiding this comment

lprimak Aug 21, 2024

Choose a reason for hiding this comment

lprimak commented Aug 21, 2024 • edited Loading

Pandrex247 left a comment

Choose a reason for hiding this comment

lprimak commented Jul 28, 2024 •

edited

Loading

lprimak commented Aug 21, 2024 •

edited

Loading