Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.lang.ArrayIndexOutOfBoundsException in JsonStringBuilder 1.2.0 #1460

Closed
ToxicMushroom opened this issue May 7, 2021 · 9 comments
Closed

Comments

@ToxicMushroom
Copy link

Describe the bug
java.lang.ArrayIndexOutOfBoundsException when using Json.encodeToString() with some input, idk what exactly is causing it
Stacktrace:

java.lang.ArrayIndexOutOfBoundsException: Index 473 out of bounds for length 473
	at kotlinx.serialization.json.internal.JsonStringBuilder.appendStringSlowPath(JsonStringBuilder.kt:84)
	at kotlinx.serialization.json.internal.JsonStringBuilder.appendQuoted(JsonStringBuilder.kt:61)
	at kotlinx.serialization.json.internal.Composer.printQuoted(Composers.kt:42)
	at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeString(StreamingJsonEncoder.kt:203)
	at kotlinx.serialization.internal.StringSerializer.serialize(Primitives.kt:139)
	at kotlinx.serialization.internal.StringSerializer.serialize(Primitives.kt:136)
	at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeSerializableValue(StreamingJsonEncoder.kt:211)
	at kotlinx.serialization.json.Json.encodeToString(Json.kt:80)
	at io.ktor.client.features.json.serializer.KotlinxSerializer.writeContent$ktor_client_serialization(KotlinxSerializer.kt:30)
	at io.ktor.client.features.json.serializer.KotlinxSerializer.write(KotlinxSerializer.kt:26)
	at io.ktor.client.features.json.JsonFeature$Feature$install$1.invokeSuspend(JsonFeature.kt:149)
	at io.ktor.client.features.json.JsonFeature$Feature$install$1.invoke(JsonFeature.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:246)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:116)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:126)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:113)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:246)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:116)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:246)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:116)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:136)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:79)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:191)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:43)
	at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:58)
	at me.melijn.melijnbot.internals.services.twitter.TwitterService.postNewTweets(TwitterService.kt:396)
	at me.melijn.melijnbot.internals.services.twitter.TwitterService.access$postNewTweets(TwitterService.kt:26)
	at me.melijn.melijnbot.internals.services.twitter.TwitterService$service$1.invokeSuspend(TwitterService.kt:45)
	at _COROUTINE._BOUNDARY._(CoroutineDebugging.kt:46)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:113)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:191)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:43)
	at me.melijn.melijnbot.internals.services.twitter.TwitterService.postNewTweets(TwitterService.kt:396)
	at me.melijn.melijnbot.internals.services.twitter.TwitterService$service$1.invokeSuspend(TwitterService.kt:45)
	at me.melijn.melijnbot.internals.threading.RunnableTask$run$1.invokeSuspend(Task.kt:140)
	at me.melijn.melijnbot.internals.threading.TaskManager$async$1$1.invokeSuspend(TaskManager.kt:29)
	at me.melijn.melijnbot.internals.threading.Task.run(Task.kt:12)
	at me.melijn.melijnbot.internals.threading.TaskManager$async$1.invokeSuspend(TaskManager.kt:30)

To Reproduce

import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json

Json.encodeToString("""{"avatar_url":"https://cdn.discordapp.com/avatars/384333349063491584/8adca1bddf8c5c46c7deed3edbd80d60.png","embeds":[{"color":1741274,"author":{"icon_url":"https://pbs.twimg.com/profile_images/1381321181719109633/4bpPMaer_normal.jpg","name":"Merlijn replied:","url":"https://twitter.com/@PixelHamster/status/1390719238155952129"},"description":"[@shroomizu](https://twitter.com/shroomizu) time for a pro controller","type":"rich","timestamp":"2021-05-07T17:24:39Z"}],"username":"Merijn"}""")

Expected behavior
Returns the String that was passed to the encodeToString() fucntion

Environment

  • Kotlin version: 1.5.0
  • Library version: 1.2.0
  • Kotlin platforms: JVM
  • Gradle version: 7.0
  • Other relevant context JRE 15.0.1
@ToxicMushroom ToxicMushroom changed the title java.lang.ArrayIndexOutOfBoundsException in JsonStringBuilder java.lang.ArrayIndexOutOfBoundsException in JsonStringBuilder 1.2.0 May 7, 2021
@serebit
Copy link

serebit commented May 8, 2021

I'm seeing this too. Reverted to v1.1.0 to work around it. Additionally seeing the occasional incorrectly encoded JSON string with a double quote missing.

@ileasile
Copy link
Member

ileasile commented May 9, 2021

Also reproduces for me on serialization of a long stacktrace. Seems rather critical to me stopping from updating actually

@Qubitium
Copy link

This is critical level bug and we have seen this in the wild with 1.2.0.

@nicholaslythall
Copy link

I'm getting the same issue.

To reproduce

import kotlinx.serialization.builtins.serializer
import kotlinx.serialization.json.Json

val query = """query AllAvailableProductsAndUpdatesQuery(${"$"}cursor: String) {
    availableProducts: products(first: 250) {
        edges {
            node {
                id
            }
        }
    }
    upsertedProducts: products(first: 250, sortKey: UPDATED_AT, after: ${"$"}cursor) {
        edges {
            cursor
            node {
                ...product
            }
        }
    }
}

fragment product on Product {
    id
    title
    handle
    descriptionHtml
    productType
    tags
    createdAt
    updatedAt
    variants(first: 10) {
        edges {
            node {
                ...productVariant
            }
        }
    }
    images(first: 10) {
        edges {
            node {
                ...image
            }
        }
    }
    nutritionCalories: metafield(namespace: "nutrition", key: "calories") {
        ...metafield
    }
    nutritionProtein: metafield(namespace: "nutrition", key: "protein") {
        ...metafield
    }
    nutritionCarbohydrates: metafield(namespace: "nutrition", key: "carbs") {
        ...metafield
    }
    nutritionFat: metafield(namespace: "nutrition", key: "fat") {
        ...metafield
    }
}

fragment productVariant on ProductVariant {
    id
    title
    priceV2 {
        ...money
    }
    compareAtPriceV2 {
        ...money
    }
    availableForSale
}

fragment image on Image {
    originalSrc
}

fragment metafield on Metafield {
    value
}

fragment money on MoneyV2 {
    amount
}"""

Json.encodeToString(String.serializer(), query)

An interesting discovery is that if I try and encode a longer string before encoding query everything works fine:

Json.encodeToString(String.serializer(), " ".repeat(1555))

I binary searched to get 1555 for the minimum required length for encoding the query to work.
1555 is exactly query.length + query.count { it == '"' } + query.count { it == '\n' }

I'm guessing there's some counting error when trying to encode escaped characters.

@JBeet
Copy link

JBeet commented May 10, 2021

Duplicate with #1441

@ToxicMushroom
Copy link
Author

Duplicate with #1441

It appears I am blind, I remember searching the exception tho, sorry for the duplicate

@ToxicMushroom
Copy link
Author

oh I didnt mean to close it

@ToxicMushroom ToxicMushroom reopened this May 10, 2021
@IRus
Copy link

IRus commented May 10, 2021

Reproducer:

@Serializable
data class MessageDTO(
    val id: String,
    val text: String,
    val created: Long,
    val tgId: String,
)
    @Test
    fun srl() {
        val messageDTO = MessageDTO(
            id = "mg-28",
            text = """Поздравляем с наступающим 2019 годом — и объявляем о возможности присоединиться к нашей команде.

2018 стал удачным годом для Telegram. Несмотря на попытки блокировки в авторитарных странах, за это время аудитория Telegram увеличилась на треть, финансовые ресурсы — более чем в 10 раз. 

В новом 2019 году Telegram расширяет звездную команду разработчиков. Заявки от кандидатов принимаем на @jobs_bot

Желаем счастья и свободы в новом году!""",
            created = 0,
            tgId = "123"
        )
        
        Json.encodeToString(MessageDTO.serializer(), messageDTO)
    }
Index 463 out of bounds for length 463
java.lang.ArrayIndexOutOfBoundsException: Index 463 out of bounds for length 463
	at kotlinx.serialization.json.internal.JsonStringBuilder.appendStringSlowPath(JsonStringBuilder.kt:99)
	at kotlinx.serialization.json.internal.JsonStringBuilder.appendQuoted(JsonStringBuilder.kt:61)
	at kotlinx.serialization.json.internal.Composer.printQuoted(Composers.kt:42)
	at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeString(StreamingJsonEncoder.kt:203)
	at kotlinx.serialization.encoding.AbstractEncoder.encodeStringElement(AbstractEncoder.kt:65)
	at io.heapy.tgto.dao.MessageDTO$$serializer.serialize(XdMessageDao.kt:12)
	at io.heapy.tgto.dao.MessageDTO$$serializer.serialize(XdMessageDao.kt:12)
	at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeSerializableValue(StreamingJsonEncoder.kt:211)
	at kotlinx.serialization.json.Json.encodeToString(Json.kt:80)
	at io.heapy.tgto.dao.MapDbMessageDaoTest.srl(MapDbMessageDaoTest.kt:100)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	...
	at java.base/java.lang.Thread.run(Thread.java:829)

@sandwwraith
Copy link
Member

I'll close this as a duplicate. We'll fix this bug and release 1.2.1 soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants