AnnotationFormatError: java.lang.IllegalArgumentException: Wrong type at constant pool index post class retransformation by agent

[bvsvas]

I get the error AnnotationFormatError: java.lang.IllegalArgumentException: Wrong type at constant pool index It comes while hitting REST Endpoint in Spring Boot application deployed in WebSphere Liberty. It only comes when call was made after class retransformation by java agent. This error doesn't come if endpoint hits before class retransformation.

We have verified class bytecode pre and post class retransformation and constant pool's constant index are intact and no change was observed. (refer to the class-dump.zip in attachments, it will have before and after .class files)

JDK used:
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 8.0.8.0 - pxa6480sr8-20230314_01(SR8))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20230313_47323 (JIT enabled, AOT enabled)
OpenJ9 - 4d03a8c
OMR - 49526cb
IBM - 7187a01

Error stack trace:

Caused by: java.lang.IllegalArgumentException: Wrong type at constant pool index
	at sun.reflect.ConstantPool.getClassAt0(Native Method)
	at sun.reflect.ConstantPool.getClassAt(ConstantPool.java:49)
	at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:256)
	at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:238)
	at sun.reflect.annotation.AnnotationParser.parseParameterAnnotations2(AnnotationParser.java:196)
	at sun.reflect.annotation.AnnotationParser.parseParameterAnnotations(AnnotationParser.java:173)
Caused by: java.lang.annotation.AnnotationFormatError: java.lang.IllegalArgumentException: Wrong type at constant pool index
	at sun.reflect.annotation.AnnotationParser.parseParameterAnnotations(AnnotationParser.java:179)
	at java.lang.reflect.Executable.parseParameterAnnotations(Executable.java:91)
	at java.lang.reflect.Executable.sharedGetParameterAnnotations(Executable.java:566)
	at java.lang.reflect.Method.getParameterAnnotations(Method.java:660)
	at org.springframework.core.MethodParameter.getParameterAnnotations(MethodParameter.java:635)
	at org.springframework.web.method.HandlerMethod$HandlerMethodParameter.getParameterAnnotations(HandlerMethod.java:541)
	at org.springframework.core.MethodParameter.getParameterAnnotation(MethodParameter.java:668)
	at org.springframework.core.MethodParameter.hasParameterAnnotation(MethodParameter.java:683)
	at org.springframework.web.method.annotation.ModelFactory.findSessionAttributeArguments(ModelFactory.java:182)
	at org.springframework.web.method.annotation.ModelFactory.initModel(ModelFactory.java:114)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:872)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)

class-dump.zip
class-dump.zip

[pshipton]

Issue for #13071 (comment)

@tajila fyi

[bvsvas]

Observation: This error comes when we re-transform controller class with method parameters having annotation. E.g.: Rest Controller methods with @RequestBody annotation, see example class CarsController in the sample project openj9-constant-pool-agent/spring-tomcat-http-app.

This issue comes only in IBM J9 JVM, not seen with HotSpot VM.

Here is the application and java agent to reproduce this issue: https://gitlab.com/bvsvas/openj9-constant-pool-agent, follow the instructions given in the README of this project.

[vashisthaa]

Dear team, I see this has been pushed to Jan refresh, Is this issue not going to be part of Oct refresh?

[tajila]

Dear team, I see this has been pushed to Jan refresh, Is this issue not going to be part of Oct refresh?

No we wont be able to get to it in time

[tajila]

@theresa-m Please take a look at this

[theresa-m]

This is caused by the same issue described in #10026.

There are a few places in the openjdk class libraries that should call public ConstantPool getConstantPool(Object internalConstantPool); including Executable.java to resolve this issue.

I don't think there's a way of doing this without making changes to class libraries in the extensions repo which isn't normally best practice in OpenJ9.

Another way could be to replace https://github.com/ibmruntimes/openj9-openjdk-jdk8/blob/openj9/jdk/src/share/classes/java/lang/reflect/Method.java#L639 with a call to com.ibm.oti.reflect.AnnotationParser.parseParameterAnnotations(Method).

@tajila can you please provide some feedback on these solutions?

[tajila]

Another way could be to replace https://github.com/ibmruntimes/openj9-openjdk-jdk8/blob/openj9/jdk/src/share/classes/java/lang/reflect/Method.java#L639 with a call to com.ibm.oti.reflect.AnnotationParser.parseParameterAnnotations(Method).

Not in favour of this one as we lose out on having the annotation cache in the byte array to speed things up.

I think we will need to do similar changes to #10026 in the extensions code.

[vashisthaa]

@tajila - I was about to echo the same sentiment that the similar changes ( as was done #10026 ) are needed, this time to ensure that the method parameter annotations are also taken care of in addition to other annotations.

[theresa-m]

Thanks @tajila. I'll open a pr for review sometime today to hopefully get this in before the code split.