Do not require the usage of opens in a modular app when using records.

[vab2048]

Is your feature request related to a problem? Please describe.

When using records in a Java modular app you must add opens declarations to the module-info.java for the package which contains the records you want to be serializable/deserializable:

opens some.package to com.fasterxml.jackson.databind;

If you do not do this, when it comes to using an object mapper you will fail to deserialize with the error:

 Caused by Failed to call `setAccess()` on Field ...... module <module-name> does not "opens <module-name>" to module com.fasterxml.jackson.databind

Describe the solution you'd like

Since it is a record I would like for there to be no need to provide 'opens' clauses in the module-info.java file.

I do not know if this is actually possible.

[cowtowncoder]

This problem might be a side-effect of some other issues wrt handling of Records, and if so, could be indirectly solved.

Does this problem occur with any Record type, even ones with no annotations, deserialize using basic default ObjectMapper, using method readValue()?

[robertvazan]

I can confirm that Jackson 2.13.1 is unable to deserialize public records without opening the package.

[robertvazan]

My setup:

ObjectMapper mapper = new ObjectMapper(new CBORFactory());
mapper.readValue(serialized, MyRecord.class)

[cowtowncoder]

Ok but here's my question: wouldn't "opens" always be required for Reflection to work both for:

1. Introspecting methods and fields that infer properties for POJOs and Records?
2. Calling constructor (for deserialization) and getter methods (for serialization)

and if so access must be given, does not exist by default.

[GedMarc]

@robertvazan use AccessType=Property instead of field ;)

This behavior is correct for strict encapsulation, no bug.

[GedMarc]

Put this on your record ;)

@JsonAutoDetect(fieldVisibility = NONE,
                getterVisibility = ANY,
                setterVisibility = ANY)  ```

[GedMarc]

@cowtowncoder using ASM to proxy records into the current module and avoid opens clauses adds too much overhead, I believe the constraints of strict encapsulation must be adhered to, there is a lot of risk in trying to avoid java locking people out of invalid operations

[cowtowncoder]

Ok I am just not 100% sure I understand what exactly is allowed and what is not, by default. If introspection of public accessors (field, methods), reflective access are ok, then that'd be fine.
And we could conceivably avoid using other access. But I have never been quite sure exactly what the limits are.

As to Records tho, with 2.13 at least, no access should be made for fields... I wish I had more time to dig in this as there is no need by Jackson to use fields in case of records. But I didn't think they'd be accessed with latest versions.

[vab2048]

@GedMarc

@cowtowncoder using ASM to proxy records into the current module and avoid opens clauses adds too much overhead, I believe the constraints of strict encapsulation must be adhered to, there is a lot of risk in trying to avoid java locking people out of invalid operations

I had asked a slightly unrelated question on the amber mailing list a while back about records and got some clarification:

Records are named tuples, they are defined only by their components, in a transparent manner i.e. no encapsulation. From a tuple, you can access to the value of each component and from all component values, you can create a tuple. The idea is that, in a method, if you are able to see a record, you can create it. Thus the canonical constructor has the same visibility as the record itself.

Key point: The idea is that, in a method, if you are able to see a record, you can create it

Based on this I don't think there is much risk in future java versions locking people out from serialising/deserialising public records which are in packages which are explicitly exported in the module-info.java.

---

@cowtowncoder

Ok but here's my question: wouldn't "opens" always be required for Reflection to work both for:

1. Introspecting methods and fields that infer properties for POJOs and Records?

2. Calling constructor (for deserialization) and getter methods (for serialization)

and if so access must be given, does not exist by default.

This is the crux of the matter... can someone please answer definitively?
Ideally we would just require the record to be public and in a package which is exported. Then in the spirit of records being "dumb data carriers" we would be able to serialise/deserialise without needing any annotations.

[robertvazan]

@GedMarc I don't want to add annotations to the records themselves, because they are part of public API, but I have tried the following on ObjectMapper with no success:

	private static final ObjectMapper mapper = new ObjectMapper(new CBORFactory())
		.setVisibility(PropertyAccessor.FIELD, Visibility.NONE)
		.setVisibility(PropertyAccessor.GETTER, Visibility.PUBLIC_ONLY)
		.setVisibility(PropertyAccessor.SETTER, Visibility.PUBLIC_ONLY)
		.setVisibility(PropertyAccessor.CREATOR, Visibility.PUBLIC_ONLY);

I have also tried Visibility.ANY instead of Visibility.PUBLIC_ONLY. I always get InaccessibleObjectException unless I open the package.

These are public records in exported package, so reflection should work on them according to setAccessible documentation:

This method may be used by a caller in class C to enable access to a member of declaring class D if any of the following hold:

• ...
• The member is public and D is public in a package that the module containing D exports to at least the module containing C.

[GedMarc]

@vab2048 Noted - Yes if the package is exported and not sealed, the API is public and should be accessible across the module-loader,

I notice two things here, the first is the CBOR factory, So I'm checking the order of the calling for that particular addon, if field access isn't required, checking the field access is not necessary and that might be triggering the invalid access, the other is if the property accessor is being applied in the factory

@cowtowncoder did say earlier records shouldn't do field access at all, I think he's coming up with a plan around that, but the cbor factory is very important to know in debugging this, thanks!

[cowtowncoder]

If there was a way to write a unit test (for maybe jackson-jdk11-compat-test or... ?) that shows the problem, I could try digging deeper. But unfortunately I think this is difficult to do. I'd really need to know where access is attempted: although ideally there should be none, it is quite difficult to prove other than having piece of code that shows unintended access.

My main suspicion would be that code that does property discovery will attempt to force access, perhaps as a "backup" setter via field.

If I did have time to proceed with Property Introspection rewrite (long-time plan), this could help here too but... as of now, my time to work on that is rather limited, unfortunately.

[robertvazan]

@cowtowncoder Why do you think the test would be hard to write? All you need is a tiny Java 16+ project with module-info.java, serializable record under src in exported package, and unit test attempting to serialize the record.

[cowtowncoder]

Sorry, I meant to suggest that getting suitable information out on offending access seemed non-trivial. Not triggering of the issue. Well, that, and then running it from IDE -- I have had some problems with module-info settings wrt test code (esp. with Eclipse).

But I'd be happy to find my concerns unfounded. :)

[robertvazan]

@cowtowncoder Well, exception is trivial to obtain:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Failed to call `setAccess()` on Field 'width' due to `java.lang.reflect.InaccessibleObjectException`, problem: Unable to make field private final int mypkg.MyClass.width accessible: module mypkg does not "opens mypkg" to module com.fasterxml.jackson.databind
 at [Source: (byte[])[1306040 bytes]; byte offset: #0]
	at [email protected]/com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at [email protected]/com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1904)
	at [email protected]/com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:268)
	at [email protected]/com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
	at [email protected]/com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
	at [email protected]/com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:642)
	at [email protected]/com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4805)
	at [email protected]/com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4675)
	at [email protected]/com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3690)

But I notice that Jackson fails to include the original exception in exception chain. I set breakpoint on IllegalArgumentException and debugger landed here:

ClassUtil.checkAndFixAccess(Member, boolean) line: 1009	
FieldProperty.fixAccess(DeserializationConfig) line: 104	
BeanDeserializerBuilder._fixAccess(Collection<SettableBeanProperty>) line: 522	
BeanDeserializerBuilder.build() line: 373	
BeanDeserializerFactory.buildBeanDeserializer(DeserializationContext, JavaType, BeanDescription) line: 294	
BeanDeserializerFactory.createBeanDeserializer(DeserializationContext, JavaType, BeanDescription) line: 150	
DeserializerCache._createDeserializer2(DeserializationContext, DeserializerFactory, JavaType, BeanDescription) line: 415	
DeserializerCache._createDeserializer(DeserializationContext, DeserializerFactory, JavaType) line: 350	
DeserializerCache._createAndCache2(DeserializationContext, DeserializerFactory, JavaType) line: 264	
DeserializerCache._createAndCacheValueDeserializer(DeserializationContext, DeserializerFactory, JavaType) line: 244	
DeserializerCache.findValueDeserializer(DeserializationContext, DeserializerFactory, JavaType) line: 142	
DefaultDeserializationContext$Impl(DeserializationContext).findRootValueDeserializer(JavaType) line: 642	
ObjectMapper._findRootDeserializer(DeserializationContext, JavaType) line: 4805	
ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 4675	
ObjectMapper.readValue(byte[], Class<T>) line: 3690	

Member being accessed is field rather than corresponding getter method, which is also apparent from the exception message. Here I got lost. I have no idea what makes Jackson use fields or getters.

I tried to add @JsonAutoDetect directly on the record as suggested by @GedMarc, but Jackson keeps accessing fields instead of getters.

[GedMarc]

@cowtowncoder Why do you think the test would be hard to write? All you need is a tiny Java 16+ project with module-info.java, serializable record under src in exported package, and unit test attempting to serialize the record.

You cannot test module-info viability in any unit test unfortunately, right now the only real way is to create a JLink project and monitor the outputs, what may be working for you in class path mode, will not as a native runtime. The results from unit tests are not on a modular environment but still run in class path and you are getting away with a lot of incorrect implementations which only appear when turning it off.

Not an easy task by any means :)

[cowtowncoder]

@robertvazan Thanks, that is useful -- in earlier cases I have only seen warnings wrt access; those do not have stack traces.
But this is more actionable, given that I can quite easily add chaining to except during troubleshooting.

This does leave the issue @GedMarc mentions, but I think that as long as it need not be unit from jackson-databind itself but an external project it may be possible. I'm fine with ability to produce stack trace being separate from simple regression test to verify fix (if and when we have it).

Now as to fields, setters: since Jackson's Record support is based on earlier POJO access, it does consider Fields as potential secondary (or tertiary) way to set property values during deserialization -- that is, each logical property can have one or more modifiers. For Records we do not really need that but I suspect that reuse of earlier functionality leaves some of these access changes in place (i.e. code does not have special handling for Record type in relevant places).
Challenge, then, would be to add more contextual handling to avoid calls to force access in this case where there is a Creator method (constructor); or, even prevent collection of Fields altogether.
In fact that'd probably be best since there really is no need for Field access for serialization or deserialization, for Record types.
To do that it is necessary to dig into where special Record handling actually starts... looks like mostly it'd be in BasicDeserializerFactory. But where we could actually more easily block Field introspection would be .... POJOPropertiesCollector, I think.

[robertvazan]

@GedMarc I am observing these exceptions in unit tests of the project that defines the records, so unit tests definitely do use modules. The only gotcha I can see is that automatic modules don't work with workspace dependency resolution (in Eclipse at least), so locally modified Jackson has to be installed in local maven repo before it can be tested. I am not sure about the MRJAR module-info.java generated by moditect plugin. That one might work with workspace dependencies too.

[GedMarc]

naw @robertvazan it doesn't quite work like that :)

There are a lot of restrictions put in place on the module-info that take place outside of running on a classpath (are you running jdk11^ apps on a JDK?) - if you are using jar files, you are on a classpath - with the -m flag or not, With the phasing out of this execution mode, the restrictions are being brought in slowly into classpath mode to assist with people moving over
One of these restrictions which you mention above is Automatic-Module-Name, the specific error is "Automatic module cannot be used with jlink" - this change is slated for JDK19/20 to disallow automatic module naming in classpath mode, so best to step away from that now, another example is classpath mode allowing encapsulation passthroughs, where from JDK 16, the restriction is moved from module to classpath which you note above, but for those using modules from JDK 9/11 this has been the default ever since the JDK release (9)

Also take note of how the service loader changes as well between the two operational modes

We have a jdk11-compat-test which generates the final artifact and performs all necessary validations on the module-info so it works on module "mode", which I am updating, I'm just going through a job change right now so it's going a little slow

The other thing we need to check on is sealed packages (package-info) and sealed classes, and it's effect on the library as well, especially with records -

I hope this clears a few things up, and maybe highlights a misstep in some of the assumptions?

[vab2048]

@GedMarc Thanks for stating all these upcoming issues (and the sealed class issue). It is interesting to me as an observer and user of Jackson to see how it will evolve.

@cowtowncoder I don't have the faintest clue of how to write a unit test for Jackson but I made this repo here where you can reference the problem with an illustrative integration test. Hope it helps.

[cowtowncoder]

@vab2048 thank you! Np wrt unit tests part (first things first). I hope to find time to look into this issue in near future (there are a few others on my longish todo list) as it seems possible we might have relatively easy improvements to make. But need to make sure there's a way to verify results with reproduction.

[cowtowncoder]

@vab2048 On sample repo: I may have missed it, but which JDK is required for it? JDK 17? And is the command to use

./gradlew test

or something else?

With JDK 17 I get failure like so:

tatu@ts-2020 jackson-records-modules-bug % ./gradlew test         
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details

> Task :test FAILED
Error occurred during initialization of boot layer
java.lang.module.FindException: Module org.apiguardian.api not found, required by org.junit.platform.launcher
Error occurred during initialization of boot layer
java.lang.module.FindException: Module org.apiguardian.api not found, required by org.junit.platform.launcher
Process 'Gradle Test Executor 2' finished with non-zero exit value 1
org.gradle.process.internal.ExecException: Process 'Gradle Test Executor 2' finished with non-zero ex

[vab2048]

@cowtowncoder apologies - I was running the test directly in IntelliJ 2021.3.1 (using the JUnit launcher) and it was displaying fine like so:

I have now since updated the build file so gradlew.bat test works. The reason for the issue you faced is Gradle bundles their own old version of Jupiter (see here) - which I have now overridden in the build.gradle file.

Hope this helps.

[cowtowncoder]

Thank you! It does; test now runs, fails, and results show the stack trace.

[cowtowncoder]

Interestingly enough, skipping collection of Fields for Record types, in POJOPropertiesCollector would avoid the exception. Unfortunately I think it would actually break some pre-JDK17 usage wherein use of fields for Records actually compensates for another flaw in Record handling (Constructor properties not bound to other properties).
There are 3 new test failures, mostly related to Naming Strategy handling.

It could even be that the annotations added in Record declaration are associated with underlying fields (I don't remember if this is the case or not). If so, Fields need to be collected first for the annotations that are then associated with getters and Constructor.

So I'll have to think of a less direct route; collecting Fields but preventing their use as Mutators.

[yihtserns]

@vab2048 disabling this should fix your issue: http://fasterxml.github.io/jackson-databind/javadoc/2.13/com/fasterxml/jackson/databind/MapperFeature.html#ALLOW_FINAL_FIELDS_AS_MUTATORS.

[cowtowncoder]

Ok, with changes from #3736 (via #3737) -- which should fully remove Field discovery for Records -- so I think this would be resolved for 2.15.0.

However, it'd be great if someone could actually verify this (and maybe include instructions here?) so I won't close this if there are remaining issues.

[Blacklands]

@cowtowncoder Just tested this in a project with a record, using 2.15.0-SNAPSHOT of jackson-databind, and on Java 19. Seems to work fine! You just need to exports the package to Jackson (com.fasterxml.jackson.databind module), opens is not necessary.

[cowtowncoder]

Ok. Closing this then, thank you @Blacklands.