Handle empty String as null

[hfhbd]

What is your use-case and why do you need this feature?
Unfortunately, not every API sends valid JSON. Sometimes, it sends an empty string instead null. To use the nullable type, I use a workaround based on the private serialized string and typed getter/setter. A custom serializer for classes, containing many members, is more complex than this workaround.

It would be nice, if the serialization framework could handle this workaround, as a custom setting.

JsonConfiguration(handleEmptyStringAsNull = true
The default value should be false, to enforce strict mode.

Describe the solution you'd like

import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonConfiguration
import org.junit.Test
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Assertions.assertNull

class EmptyStringSerializerTest {
    private val json = Json(JsonConfiguration.Stable)

    /**
     * Normal, expected usage
     */
    @Serializable
    data class Valid(var content: Long?)

    @Test
    fun expected() {
        val json = Json(JsonConfiguration(handleEmptyStringAsNull = true)) //TODO: New setting to convert empty strings to nullable, instead throwing, default = false (strict)
        val invalideString = """{"content":""}"""
        val failing = json.parse(Valid.serializer(), invalideString) // java.lang.NumberFormatException: For input string: ""
        assertNull(failing.content)
    }

    /**
     * Workaround to eliminate the usage of a custom serializer, which is not nice with a class containing many members
     */
    @Serializable
    data class Invalide(@SerialName("content") private var value: String) {
        var content: Long? get() = value.toLongOrNull()
            set(newValue) {
                this.value = (newValue ?: "").toString()
            }
    }

    /**
     * Just testing the workaround
     */
    @Test
    fun customGetterSetterUsedByInvalide() {
        val invalide = Invalide("foo")
        invalide.content = null
        val empty = json.stringify(Invalide.serializer(), invalide)
        assertEquals("""{"content":""}""", empty)
        val invalideParsed = json.parse(Invalide.serializer(), empty)
        assertNull(invalideParsed.content)

        invalide.content = 42
        val contentString = json.stringify(Invalide.serializer(), invalide)
        assertEquals("""{"content":"42"}""", contentString)
        val contentSet = json.parse(Invalide.serializer(), contentString)
        assertEquals(invalide, contentSet)
    }


    /**
     * Working test of the valid, normal class
     */
    @Test
    fun valid() {
        val valid = Valid(null)
        val empty = json.stringify(Valid.serializer(), valid)
        assertEquals("""{"content":null}""", empty)
        val validParsed = json.parse(Valid.serializer(), empty)
        assertNull(validParsed.content)

        valid.content = 42
        val contentLong = json.stringify(Valid.serializer(), valid)
        assertEquals("""{"content":42}""", contentLong)
        val contentSet = json.parse(Valid.serializer(), contentLong)
        assertEquals(valid, contentSet)
    }
}

[pdvrieze]

@hfhbd There is also the option to use a custom serializer that does this. It allows much more fine-grained control.

[hfhbd]

@hfhbd There is also the option to use a custom serializer that does this.

I know, but writing a custom serializer for classes containing 50 or more attributes is a little bit "annoying". Its not possible to get the generated KSerializer from the compiler as kotlin source code, is it? This would eliminate the manual boilerplate code.

[pdvrieze]

@hfhbd I don't mean a custom serializer for the class that owns the string. I mean a custom serializer for (nullable) strings. Something like:

class ABigClassWithManyProperties {
  @SerializeWith(NullableStringSerializer::class)
  var nullableStringPropertly: String? = null
}

class NullableStringSerializer: KSerializer<String?> {
  private val delegate = NullableSerialzier(StringSerializer())

  fun serialize(encoder: Encoder, value: String?) {
    if (value?.isEmpty()==true) {
      delegate.serialize(encoder,null)
    } else {
      delegate.serialize(encoder,value)
    }
  }

  fun deserialize(decoder:Decoder): String? {
    val str = delegate.deserialize(decoder)
    if (str?.isEmpty()==true) return null
    return str
  }
}

[hfhbd]

@pdvrieze
Okay, but the class NullableStringSerializer cannot be generic, can it?

[pdvrieze]

@hfhbd I am not 100% what you mean. However you can add the @Serializewith annotation to any String parameter you want. It should also be possible to do this at file level. You are merely configuring the serialisation framework to use a different serializer for strings.

[hfhbd]

The api sends "" instead null for every type, not only Strings. So your NullableStringSerializer has to be generic, which would be impossible...

class NullableStringSerializer<In : Any>(private val delegate: KSerializer<In>) : KSerializer<In?> {

    override fun serialize(encoder: Encoder, obj: In?) {
        delegate.nullable.serialize(encoder, obj)
    }

    override fun deserialize(decoder: Decoder): In? {
        val jsonStringOfIn = decoder.decodeString() //wrong methods, does not return a object as string
        if (jsonStringOfIn.isEmpty()) return null
        return delegate.nullable.parse(jsonStringOfIn) // there is no fun parse(String): T in KSerializer
    }

    override val descriptor = StringDescriptor.withName("NullableSerializerOf${delegate.descriptor}")
}

How to call this in @Serializeable(with = NullableStringSerializer<In>::class)?

[LDuncAndroid]

Does NullableSerialzier still exist?

I notice it's been replaced by SerialDescriptor.nullable

I've been trying to implement similar to what @pdvrieze implemented here but every time I construct with a null it's never deserialized as 0.

@Serializable
data class ClassWithNullableInt(@Serializable(with = NullableIntSerializer::class) val value: Int?)

class NullableIntSerializer : KSerializer<Int?> {

    override val descriptor: SerialDescriptor = PrimitiveDescriptor("NullableIntSerializer", PrimitiveKind.INT).nullable
    private val delegate = Int.serializer().nullable

    override fun serialize(encoder: Encoder, value: Int?) {
        if (value == null) {
            delegate.serialize(encoder, 0)
        } else {
            delegate.serialize(encoder, value)
        }
    }

    override fun deserialize(decoder: Decoder): Int? {
        return delegate.deserialize(decoder) ?: 0
    }
}

@UnstableDefault
@ImplicitReflectionSerializer
class NullableIntSerializerTests : FunSpec({
    val json = Json(JsonConfiguration.Default)

    test("null value should deserialize as equal to 0") {
        val nullInt = ClassWithNullableInt(null)
        val zeroInt = ClassWithNullableInt(0)

        val jsonStringNullInt = json.stringify(ClassWithNullableInt.serializer(), nullInt)

        jsonStringNullInt shouldBe """{"value":0}"""

        val zeroInt2 = json.parse(Int.serializer().nullable, jsonStringNullInt)

        zeroInt2 shouldBe zeroInt
    }

    test("0 value should deserialize as equal to 0") {
        val zeroInt = ClassWithNullableInt(0)
        val jsonStringZeroInt = json.stringify(ClassWithNullableInt.serializer(), zeroInt)

        jsonStringZeroInt shouldBe """{"value":0}"""

        val zeroInt2 = json.parse(ClassWithNullableInt.serializer(), jsonStringZeroInt)

        zeroInt2 shouldBe zeroInt
    }

})

[pdvrieze]

You now just use the .nullable extension property instead of NullableSerializer.

[ddelegator]

@hfhbd I'm running into the same problem (with ServiceNow API). Could you share a solution if you found one?

[hfhbd]

@ddelegator I used my workaround

/**
 * Workaround to eliminate the usage of a custom serializer, which is not nice with a class containing many members
*/
@Serializable
data class Invalide(@SerialName("content") private var value: String) {
    var content: Long? 
        get() = value.toLongOrNull()
        set(newValue) {
            this.value = (newValue ?: "").toString()
        }
}

[DRSchlaubi]

Another Problem is that e.g. the Discord API sometimes just returns an empty string as the entire response and currently there is no way of parsing that response properly as the KSerializer API doesn't give you the information whether the input actually contains any content or not.

The way we solved this before is just catching the error any decode method would throw when trying to decode an empty string,
but this now causes the lexer.expectEof() line to fail after upgrading to 1.2.0

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.base/java.lang.StringLatin1.charAt(StringLatin1.java:48)
	at java.base/java.lang.String.charAt(String.java:711)
	at kotlinx.serialization.json.internal.JsonLexer.consumeNextToken(JsonLexer.kt:234)
	at kotlinx.serialization.json.internal.JsonLexer.expectEof(JsonLexer.kt:128)
	at kotlinx.serialization.json.Json.decodeFromString(Json.kt:96)

[Dominaezzz]

@DRSchlaubi you can always do (decoder as JsonDecoder).decodeJson() and inspect the json tree.

[DRSchlaubi]

@DRSchlaubi you can always do (decoder as JsonDecoder).decodeJson() and inspect the json tree.

I can't here because the exception gets thrown outside the serializer

[pdvrieze]

@DRSchlaubi This could be handled by the format, by special casing a lack of input (if given a nullable serializer - generated serializers are not nullable, but can be wrapped to be nullable), but it is perhaps something to do as a wrapper (or a wrapper format).

[evilcat-x]

I have a question related to this.
Since this feature is unavailable, we need use a custom serializer to achieve the result if it is on a class.

I have an endpoint that returns either [] a list / an empty list / a list of objects, however it cloud also return "" with EOF directly.

If we have the empty as null, then I can just decode it as List<Class>? but currently this doesn't work; I am trying to understand how I would use a custom serializer to achieve mapping a "" to a null or []

[sandwwraith]

@evilcat-x Would https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/json.md#json-transformations help?

[evilcat-x]

@sandwwraith Thanks for the direction, I was manually setup a Customer Serializer to do it, but this seems indeed the functionality I am looking for to map an empty JSON as in empty str to an empty list.