Fix IllegalAccessException being thrown on an attempt to retrieve ser…

[sandwwraith]

…ializer for some private implementation classes from stdlib.

Fixes #2449

[pdvrieze]

I've gone in and added some tests. Using modifiers fixes the original problem, but also addresses some other object cases (tests added).

[pdvrieze]

There are two additional tests that should be added here. This adds two additional test cases (both fail with the suggested implementation):

• Getting a serializer from an anonymous object (canonicalName returns null) throws an NPE
• Getting a serializer from a local named object throws because it is not built in

Suggested change

	@Test
	fun testTopLevelMaps() {
	// Returning null here is a deliberate choice: map constructor functions may return different specialized
	// implementations (e.g., kotlin.collections.EmptyMap or java.util.Collections.SingletonMap)
	// that may or may not be generic. Since we generally cannot return a generic serializer using Java class only,
	// all attempts to get map serializer using only .javaClass should return null.
	assertNull(serializerOrNull(emptyMap<String, String>().javaClass))
	private object namedObject
	@Test
	fun testTopLevelMaps() {
	// Returning null here is a deliberate choice: map constructor functions may return different specialized
	// implementations (e.g., kotlin.collections.EmptyMap or java.util.Collections.SingletonMap)
	// that may or may not be generic. Since we generally cannot return a generic serializer using Java class only,
	// all attempts to get map serializer using only .javaClass should return null.
	assertNull(serializerOrNull(emptyMap<String, String>().javaClass))
	assertNull(serializerOrNull(object {}.javaClass))
	assertNull(serializerOrNull(namedObject.javaClass))

[sandwwraith]

Thanks for pointing out the problem with the anonymous class! Adding a safe call to canonicalName is enough to fix it, though.

[pdvrieze]

The problem is that it is not accessible (IllegalAccessException), if it is not accessible then it is also not an object. See the suggested additions to the test suite (JavaCollectionsTest) that are not fixed by the solution.

Suggested change

	if (this.canonicalName.let { it.startsWith("java.") || it.startsWith("kotlin.") }) return null
	if (!Modifier.isPublic(this.modifiers)) return null

[sandwwraith]

This check will not work correctly for protected situation like this one:

open class A {
  @Serializable protected object B {}
}

Regarding @Serializable private object X — you're right; this code will throw an IllegalAccessError now. However, it will indicate problems with users' code instead of masking it behind the null. Also, we support @Serializable private class X even if it is in a different package. So, probably, it should be supported instead of returning null. Seems like it is out of the scope of this PR anyways.

[pdvrieze]

Maybe the "best" way would be to just catch the IllegalAccessException as it is otherwise really hard to know what is accessible.

[sandwwraith]

It's an adequate approach (although we lose information that is contained in exception message). However, I should first check if we can support @Serializable private object X properly.

[pdvrieze]

INSTANCE could also not be public, and therefore the type would not be an object (small chance, but add a check for it being public).

Suggested change

	val field =
	declaredFields.singleOrNull { it.name == "INSTANCE" && it.type == this && Modifier.isStatic(it.modifiers) }
	val field = declaredFields.singleOrNull {
	it.name == "INSTANCE" &&
	it.type == this &&
	Modifier.isStatic(it.modifiers) &&
	Modifier.isPublic(it.modifiers)
	} ?: return null

[sandwwraith]

Can you provide an example? IIRC Kotlin doesn't produce INSTANCE outside of Kotlin objects and it's hard to produce such code by yourself

[pdvrieze]

Basically if someone were to have a Java based singleton such as:

class BreakThings {
    private static BreakThings INSTANCE = BreakThings();

    public static BreakThings getInstance() { return INSTANCE; }
}

[sandwwraith]

I think it's ok to fail or have unspecified behavior when asked for serializer for some java class