Permgen memory leak due to ClassInfo caching in java.beans.Introspector on JDK 11/17

[webbres]

I am currently trying to resolve an issue we're seeing with redeploying a spring boot application that looks to be a problem in the spring boot end not ours?

Links:

• Stack overflow question: https://stackoverflow.com/questions/70174850/spring-boot-tomcat-redeploy-permgen-leak-due-to-logging
• Git repo: https://github.com/webbres/spring-boot-permgen-issue/tree/master/debugging

Config:
Java 11
OS: Windows & Centos
Spring boot: 2.5
Tomcat: 9.0.27

---

The problem/bug:

When deploying on tomcat 9 using java 11 reloading an application results in a PermGen memory leak with the classloader for the application not being GCd. After adding additional configuration only weak and soft references are found in the heap.

Summary of findings:

Reloading a spring boot application deployed as a war in tomcat 9 results in a PermGen memory leak (reported by the find leaks check and checked in VisualVM). In our actual application there is a strong reference to a log4j JMX bean which is removed when running tomcat with -Dlog4j2.disable.jmx=true though even then the org.apache.catalina.loader.ParallelWebappClassLoader for the application is marked as DESTROYED but is not garbage collected.

I attempted to recreate the issue on a simple spring boot starter (see repo linked). Here I found strong references to logging shutdown hooks which are removed when adding logging.register-shutdown-hook=false to the application.properties. Though after adding this property like above the classloader still isn't being garbage collected.

Cheers

Sam

[wilkinsona]

As far as I can tell, this problem is specific to Java 11.

Running with Java 8, when an application is undeployed and GC is forced, all of its classes are unloaded and its ClassLoader is garbage collection. Running with Java 11, the ClassLoader is never garbage collected. In both cases, there are no strong references to the ClassLoader. In Java 8, classes loaded by the class loader are only referenced weakly. In Java 11, some of the classes loaded by the class loader are referenced softly. The presence of soft references is key to the difference in behaviour.

Generally speaking, weak references will be cleaned up whenever garbage collection is forced. By contrast, soft references will only be cleaned up when the JVM is under memory pressure. Examining in YourKit the heap of a basic Spring Boot web application running on Java 11 reveals soft references from com.sun.beans.introspect.ClassInfo to various Spring and application classes. These ClassInfo instances are being created by java.beans.Introspector via org.springframework.beans.CachedIntrospectionResults.

CachedIntrospectionResults used to call java.beans.Introspector.flushFromCaches(Class<?>) immediately after introspecting a class. This was removed as, due to changes in the JDK, it was no longer needed. It looks to me like it may be needed again following these changes in the JDK which introduced ClassInfo and a soft reference cache.

[wilkinsona]

Assuming that my analysis of the cause is correct, I am not sure that reinstating the call to flushFromCaches will be sufficient. As far as I can tell, it does not clear the cache used by the new com.sun.beans.introspect.ClassInfo. This was reported as a JDK-8207331 which was closed as a duplicate of JDK-8231454. JDK-8231454 was fixed in Java 16 by these changes.

[webbres]

Thank you for looking into it. It doesn't sound like there's much I can do my end in terms of support but do let me know if this isn't the case.

[rstoyanchev]

It sounds like this requires a fix in the JDK. @wilkinsona, I'm wondering if there anything to be done in the Spring Framework, i.e. should this remain open?

[wilkinsona]

The fix has already been made in the JDK but only in Java 16 and later. For that fix to be effective Framework's call to flushFromCaches needs to be reinstated as far as I can tell. For Java 11, I'm not sure what can be done. At best, a change here would need to be a reflective hack as just calling flushFromCaches won't help. The ideal would be a backport of JDK-8231454 to Java 11.

[bclozel]

This has been backported in Java 11.0.16 by JDK-8287343. The 6.0.x generation has moved to JDK17+ in the meantime.

I'm closing this issue as a result.

[wilkinsona]

@bclozel has the call to flushFromCaches been reinstated? If my analysis above is correct, it is required to take advantage of the JDK changes.

[jhoeller]

As of 6.0, this only affects StandardBeanInfoFactory since our lean SimpleBeanInfoFactory (which does not call java.beans.Introspector) is used by default now. Nevertheless, the Introspector.flushFromCaches() is worth reinstating there as well in case someone explicitly configures StandardBeanInfoFactory for backwaeds compatibility.

More importantly, this is worth backporting to 5.3.x for avoiding the memory leak on JDK 11.0.16+.