allow encoding literal JSON values #2041

aSemy · 2022-09-22T21:13:11Z

This PR provides a new function for encoding raw JSON content, without quoting it as a string. This allows for encoding JSON numbers of any size or precision, so BigDecimal and BigInteger can be supported.

Link to view the updated documentation

fix: #1051
fix: #1405

JsonUnquotedLiteral() is a new function that allows creating literal JSON content.
I added val coerceToInlineType to JsonLiteral, so that JsonUnquotedLiteral could use encodeInline()
I defined val jsonUnquotedLiteralDescriptor as a 'marker', for use with encodeInline()
ComposerForUnquotedLiterals (based on ComposerForUnsignedNumbers) will 'override' the encoder when a JsonLiteral has the jsonUnquotedLiteralDescriptor marker, and will encode the content as a string without surrounding quotes.

~~I'm not sure how to proceed with decoding literal numbers.~~ I just realised, there's no need for special implementation - primitives are all decoded the same.

aSemy · 2022-09-22T21:53:26Z

I think more tests and specialist knowledge is needed, but the approach is ready for review. I'm really keen to provide a way of supporting BigDecimal, as it's been a blocker in a couple of my projects for a long time.

YanishR

When do you guys foresee there will be progress on this? I have a use case for it in my project

sandwwraith · 2022-10-03T12:33:21Z

Hi, thanks for your PR! I think I have time for it on this week, stay tuned

formats/json/commonMain/src/kotlinx/serialization/json/internal/Composers.kt

formats/json-tests/jvmTest/src/kotlinx/serialization/BigDecimalTest.kt

formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt

formats/json/commonMain/src/kotlinx/serialization/json/internal/StreamingJsonEncoder.kt

formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt

sandwwraith · 2022-10-06T16:03:17Z

I think we should have additional design discussion on a general approach (using encodeInline and JsonPrimitive(isString = true instead of a separate JsonEncoder.encodeUnquoted function); I'll keep you posted

formats/json-tests/jvmTest/src/kotlinx/serialization/BigDecimalTest.kt

sandwwraith

After a discussion, the following points were established:

Function should accept arbitrary strings and do not validate them with regex. This will allow users to do various json-embeddings, support non-base10 numbers, different notations, etc and will also simplify an invariant for users and save us from potential problems with regex correctness and performance.
Function should accept a String? and return JsonNull if a null was passed to it, for consistency with JsonPrimitive overloads.
However, if string literal "null" is passed (i.e. value == "null"), function should throw IllegalArgumentException. This is motivated by the fact that when we serialize such a literal, if effective becomes indistinguishable from JsonNull and can't be deserialized back to JsonLiteral. We cannot know for sure if this was a user intent to do so or bug in validation. Better to prohibit such cases. IAE message can sound like this: "It is impossible to create a literal unquoted value of 'null'. If you want to create json null literal, use JsonNull object, otherwise, use quoted literal."

Summarising this all up, I suggest to name the new function 'JsonUnquotedPrimitive'. What do you think?

...n-tests/commonTest/src/kotlinx/serialization/json/serializers/JsonPrimitiveSerializerTest.kt

formats/json-tests/jvmTest/src/kotlinx/serialization/BigDecimalTest.kt

formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt

formats/json-tests/commonTest/src/kotlinx/serialization/json/serializers/JsonRawElementTest.kt

sandwwraith · 2022-10-13T13:02:00Z

Also, I've just noted that the PR is opened against master branch, while all new PRs with code should be opened against dev. Can you change the branch please?

aSemy · 2022-10-13T13:16:08Z

I've updated the PR to point at dev, and merged dev into this branch, but it looks like dev is missing commits from master. Can master be merged into dev?

sandwwraith · 2022-10-13T14:00:02Z

dev is missing commits from master. Can master be merged into dev?

That's a normal situation. We only merge master into dev before the actual release

…ammar updates

…ment.kt Co-authored-by: Leonid Startsev <[email protected]>

sandwwraith

Great job, thanks for implementing this important feature!

aSemy · 2022-10-21T14:29:59Z

Thanks! And thanks to you and @qwwdfsad for reviewing.

Do you know when this will be available? Does KxS have snapshot releases that I can use?

sandwwraith · 2022-10-21T14:31:31Z

No, we don't usually publish snapshots. The next release is going to be around Kotlin 1.8.0 release, in a month or two

This PR provides a new function for encoding raw JSON content, without quoting it as a string. This allows for encoding JSON numbers of any size or precision, so BigDecimal and BigInteger can be supported. Fixes Kotlin#1051 Fixes Kotlin#1405 The implementation is similar to how unsigned numbers are handled. JsonUnquotedLiteral() is a new function that allows creating literal JSON content. Added val coerceToInlineType to JsonLiteral, so that JsonUnquotedLiteral could use encodeInline() Defined val jsonUnquotedLiteralDescriptor as a 'marker', for use with encodeInline() ComposerForUnquotedLiterals (based on ComposerForUnsignedNumbers) will 'override' the encoder when a JsonLiteral has the jsonUnquotedLiteralDescriptor marker, and will encode the content as a string without surrounding quotes.

aSemy marked this pull request as ready for review September 22, 2022 21:51

aSemy mentioned this pull request Sep 30, 2022

How to serialize BigInteger and BigDouble? #1051

Closed

YanishR reviewed Oct 3, 2022

View reviewed changes

sandwwraith requested changes Oct 6, 2022

View reviewed changes

aSemy commented Oct 6, 2022

View reviewed changes

formats/json-tests/jvmTest/src/kotlinx/serialization/BigDecimalTest.kt Outdated Show resolved Hide resolved

sandwwraith requested changes Oct 12, 2022

View reviewed changes

aSemy changed the title ~~allow encoding literal JSON numbers~~ allow encoding literal JSON values Oct 12, 2022

aSemy commented Oct 12, 2022

View reviewed changes

formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt Outdated Show resolved Hide resolved

aSemy commented Oct 12, 2022

View reviewed changes

formats/json-tests/commonTest/src/kotlinx/serialization/json/serializers/JsonRawElementTest.kt Outdated Show resolved Hide resolved

aSemy changed the base branch from master to dev October 13, 2022 13:03

implement JsonRawElement, for encoding literal JSON strings

ff8fd99

aSemy force-pushed the feat/1051-big_decimals branch from 2d1e859 to ff8fd99 Compare October 13, 2022 14:35

aSemy added 4 commits October 13, 2022 16:38

update name of inlineRawJsonElementEncoder

ace3db0

Merge remote-tracking branch 'origin/dev' into feat/1051-big_decimals

ac474ed

Merge remote-tracking branch 'origin/dev' into feat/1051-big_decimals

92683c8

rename JsonRawElement -> JsonUnquotedLiteral

83043d4

aSemy mentioned this pull request Oct 18, 2022

Support Number jsons ionspin/kotlin-multiplatform-bignum#219

Closed

aSemy added 7 commits October 19, 2022 08:56

finish renaming JsonRawElement -> JsonUnquotedLiteral

462af90

update docs for JsonUnquotedLiteral

b79528d

update docs for JsonUnquotedLiteral

c437fc4

Merge remote-tracking branch 'origin/dev' into feat/1051-big_decimals

28c318f

add knit docs for JsonUnquotedLiteral

74a91a1

grammar/editing knit docs

816f50e

additional Knit test for BigDecimal decoding, and minor formatting/gr…

e33fbe9

…ammar updates

aSemy and others added 14 commits October 20, 2022 16:58

Update formats/json/commonMain/src/kotlinx/serialization/json/JsonEle…

e430234

…ment.kt Co-authored-by: Leonid Startsev <[email protected]>

Update formats/json/commonMain/src/kotlinx/serialization/json/JsonEle…

c5e40dc

…ment.kt Co-authored-by: Leonid Startsev <[email protected]>

Update formats/json/commonMain/src/kotlinx/serialization/json/JsonEle…

374fcc0

…ment.kt Co-authored-by: Leonid Startsev <[email protected]>

Update formats/json/commonMain/src/kotlinx/serialization/json/JsonEle…

be8a5d0

…ment.kt Co-authored-by: Leonid Startsev <[email protected]>

move note regarding "null"

1374e85

move non-parmeterized assert out of parametrizedTest {}

1a19d10

update 'knit'

61a7c42

bump knit version

d75ea07

rm unnecessary comma

5a8c304

fix link

de14566

fix TOC link (temp fix)

a03fc10

add note about creating a serializer

32f6d64

gramma fix, and fix link

74a5e09

fix link

4c2872b

sandwwraith mentioned this pull request Oct 21, 2022

Expose an API to read primitive values of JSON keys as-is as a string #1557

Closed

sandwwraith approved these changes Oct 21, 2022

View reviewed changes

sandwwraith merged commit 46a5ff6 into Kotlin:dev Oct 21, 2022

aSemy deleted the feat/1051-big_decimals branch May 9, 2023 15:47

aSemy mentioned this pull request Jun 18, 2023

Allow working with JSON5 without @Serializable classes xn32/json5k#2

Open

This was referenced Nov 1, 2023

Option to show small Double values not in exponential format in JSON #1262

Closed

Can not search substrings when JSON is nested #1415

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow encoding literal JSON values #2041

allow encoding literal JSON values #2041

aSemy commented Sep 22, 2022 •

edited

Loading

aSemy commented Sep 22, 2022

YanishR left a comment

sandwwraith commented Oct 3, 2022

sandwwraith commented Oct 6, 2022

sandwwraith left a comment

sandwwraith commented Oct 13, 2022

aSemy commented Oct 13, 2022

sandwwraith commented Oct 13, 2022

sandwwraith left a comment

aSemy commented Oct 21, 2022

sandwwraith commented Oct 21, 2022

allow encoding literal JSON values #2041

allow encoding literal JSON values #2041

Conversation

aSemy commented Sep 22, 2022 • edited Loading

aSemy commented Sep 22, 2022

YanishR left a comment

Choose a reason for hiding this comment

sandwwraith commented Oct 3, 2022

sandwwraith commented Oct 6, 2022

sandwwraith left a comment

Choose a reason for hiding this comment

sandwwraith commented Oct 13, 2022

aSemy commented Oct 13, 2022

sandwwraith commented Oct 13, 2022

sandwwraith left a comment

Choose a reason for hiding this comment

aSemy commented Oct 21, 2022

sandwwraith commented Oct 21, 2022

aSemy commented Sep 22, 2022 •

edited

Loading