Skip to content

Commit

Permalink
docs for protobuf oneof
Browse files Browse the repository at this point in the history
  • Loading branch information
xiaozhikang0916 committed Jan 20, 2024
1 parent a69c227 commit e5e703e
Show file tree
Hide file tree
Showing 13 changed files with 313 additions and 217 deletions.
80 changes: 70 additions & 10 deletions docs/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ stable, these are currently experimental features of Kotlin Serialization.
* [Integer types](#integer-types)
* [Lists as repeated fields](#lists-as-repeated-fields)
* [Packed fields](#packed-fields)
* [Oneof field (experimental)](#oneof-field-experimental)
* [ProtoBuf schema generator (experimental)](#protobuf-schema-generator-experimental)
* [Properties (experimental)](#properties-experimental)
* [Custom formats (experimental)](#custom-formats-experimental)
Expand Down Expand Up @@ -435,6 +436,65 @@ Per the standard packed fields can only be used on primitive numeric types. The
Per the [format description](https://developers.google.com/protocol-buffers/docs/encoding#packed) the parser ignores
the annotation, but rather reads list in either packed or repeated format.

### Oneof field (experimental)

Kotlin Serialization `ProtoBuf` format supports [oneof](https://protobuf.dev/programming-guides/proto2/#oneof) fields
base on the [Polymorphism](polymorphism.md).

You can declare a property of your class to be `oneof` by following the contracts:

* Declare an interface, or abstract class, in represent of the `oneof` group.
* Declare the property with the type added above, annotated with `@ProtoOneOf(ids)`
with all possible proto numbers, not `@ProtoNumber`.
* Declare subclasses from the type with **only one property** each per the oneof group elements.
* Annotated the subclasses with `@ProtoNumber` on the class declaration, not the property,
per the oneof group elements and `@ProtoOneOf(ids)` above.

<!--- INCLUDE
import kotlinx.serialization.*
import kotlinx.serialization.protobuf.*
-->

```kotlin
@Serializable
data class Data(
@ProtoNumber(1) val name: String,
@ProtoOneOf(2, 3) val phone: IPhoneType,
)
@Serializable sealed interface IPhoneType
@Serializable @ProtoNumber(2) @JvmInline value class HomePhone(val number: String): IPhoneType
@Serializable @ProtoNumber(3) data class WorkPhone(val number: String): IPhoneType

fun main() {
val dataTom = Data("Tom", HomePhone("123"))
val stringTom = ProtoBuf.encodeToHexString(dataTom)
val dataJerry = Data("Jerry", WorkPhone("789"))
val stringJerry = ProtoBuf.encodeToHexString(dataJerry)
println(stringTom)
println(stringJerry)
println(ProtoBuf.decodeFromHexString<Data>(stringTom))
println(ProtoBuf.decodeFromHexString<Data>(stringJerry))
}
```

> You can get the full code [here](../guide/example/example-formats-08.kt).
```text
0a03546f6d1203313233
0a054a657272791a03373839
Data(name=Tom, phone=HomePhone(number=123))
Data(name=Jerry, phone=WorkPhone(number=789))
```

<!--- TEST -->

In [ProtoBuf diagnostic mode](https://protogen.marcgravell.com/decode) the first 2 lines on output is equivalent to

```
Field #1: 0A String Length = 3, Hex = 03, UTF8 = "Tom" Field #2: 12 String Length = 3, Hex = 03, UTF8 = "123"
Field #1: 0A String Length = 5, Hex = 05, UTF8 = "Jerry" Field #3: 1A String Length = 3, Hex = 03, UTF8 = "789"
```

### ProtoBuf schema generator (experimental)

As mentioned above, when working with protocol buffers you usually use a ".proto" file and a code generator for your
Expand Down Expand Up @@ -467,15 +527,15 @@ fun main() {
println(schemas)
}
```
> You can get the full code [here](../guide/example/example-formats-08.kt).
> You can get the full code [here](../guide/example/example-formats-09.kt).
Which would output as follows.

```text
syntax = "proto2";
// serial name 'example.exampleFormats08.SampleData'
// serial name 'example.exampleFormats09.SampleData'
message SampleData {
required int64 amount = 1;
optional string description = 2;
Expand Down Expand Up @@ -519,7 +579,7 @@ fun main() {
}
```

> You can get the full code [here](../guide/example/example-formats-09.kt).
> You can get the full code [here](../guide/example/example-formats-10.kt).
The resulting map has dot-separated keys representing keys of the nested objects.

Expand Down Expand Up @@ -599,7 +659,7 @@ fun main() {
}
```

> You can get the full code [here](../guide/example/example-formats-10.kt).
> You can get the full code [here](../guide/example/example-formats-11.kt).
As a result, we got all the primitive values in our object graph visited and put into a list
in _serial_ order.
Expand Down Expand Up @@ -701,7 +761,7 @@ fun main() {
}
```

> You can get the full code [here](../guide/example/example-formats-11.kt).
> You can get the full code [here](../guide/example/example-formats-12.kt).
Now we can convert a list of primitives back to an object tree.

Expand Down Expand Up @@ -792,7 +852,7 @@ fun main() {
}
-->

> You can get the full code [here](../guide/example/example-formats-12.kt).
> You can get the full code [here](../guide/example/example-formats-13.kt).
<!--- TEST
[kotlinx.serialization, kotlin, 9000]
Expand Down Expand Up @@ -899,7 +959,7 @@ fun main() {
}
```

> You can get the full code [here](../guide/example/example-formats-13.kt).
> You can get the full code [here](../guide/example/example-formats-14.kt).
We see the size of the list added to the result, letting the decoder know where to stop.

Expand Down Expand Up @@ -1011,7 +1071,7 @@ fun main() {

```

> You can get the full code [here](../guide/example/example-formats-14.kt).
> You can get the full code [here](../guide/example/example-formats-15.kt).
In the output we see how not-null`!!` and `NULL` marks are used.

Expand Down Expand Up @@ -1139,7 +1199,7 @@ fun main() {
}
```
> You can get the full code [here](../guide/example/example-formats-15.kt).
> You can get the full code [here](../guide/example/example-formats-16.kt).
As we can see, the result is a dense binary format that only contains the data that is being serialized.
It can be easily tweaked for any kind of domain-specific compact encoding.
Expand Down Expand Up @@ -1333,7 +1393,7 @@ fun main() {
}
```
> You can get the full code [here](../guide/example/example-formats-16.kt).
> You can get the full code [here](../guide/example/example-formats-17.kt).
As we can see, our custom byte array format is being used, with the compact encoding of its size in one byte.

Expand Down
1 change: 1 addition & 0 deletions docs/serialization-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ Once the project is set up, we can start serializing some classes.
* <a name='integer-types'></a>[Integer types](formats.md#integer-types)
* <a name='lists-as-repeated-fields'></a>[Lists as repeated fields](formats.md#lists-as-repeated-fields)
* <a name='packed-fields'></a>[Packed fields](formats.md#packed-fields)
* <a name='oneof-field-experimental'></a>[Oneof field (experimental)](formats.md#oneof-field-experimental)
* <a name='protobuf-schema-generator-experimental'></a>[ProtoBuf schema generator (experimental)](formats.md#protobuf-schema-generator-experimental)
* <a name='properties-experimental'></a>[Properties (experimental)](formats.md#properties-experimental)
* <a name='custom-formats-experimental'></a>[Custom formats (experimental)](formats.md#custom-formats-experimental)
Expand Down
23 changes: 15 additions & 8 deletions guide/example/example-formats-08.kt
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,23 @@ package example.exampleFormats08

import kotlinx.serialization.*
import kotlinx.serialization.protobuf.*
import kotlinx.serialization.protobuf.schema.ProtoBufSchemaGenerator

@Serializable
data class SampleData(
val amount: Long,
val description: String?,
val department: String = "QA"
data class Data(
@ProtoNumber(1) val name: String,
@ProtoOneOf(2, 3) val phone: IPhoneType,
)
@Serializable sealed interface IPhoneType
@Serializable @ProtoNumber(2) @JvmInline value class HomePhone(val number: String): IPhoneType
@Serializable @ProtoNumber(3) data class WorkPhone(val number: String): IPhoneType

fun main() {
val descriptors = listOf(SampleData.serializer().descriptor)
val schemas = ProtoBufSchemaGenerator.generateSchemaText(descriptors)
println(schemas)
val dataTom = Data("Tom", HomePhone("123"))
val stringTom = ProtoBuf.encodeToHexString(dataTom)
val dataJerry = Data("Jerry", WorkPhone("789"))
val stringJerry = ProtoBuf.encodeToHexString(dataJerry)
println(stringTom)
println(stringJerry)
println(ProtoBuf.decodeFromHexString<Data>(stringTom))
println(ProtoBuf.decodeFromHexString<Data>(stringJerry))
}
20 changes: 10 additions & 10 deletions guide/example/example-formats-09.kt
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@
package example.exampleFormats09

import kotlinx.serialization.*
import kotlinx.serialization.properties.Properties // todo: remove when no longer needed
import kotlinx.serialization.properties.*
import kotlinx.serialization.protobuf.*
import kotlinx.serialization.protobuf.schema.ProtoBufSchemaGenerator

@Serializable
class Project(val name: String, val owner: User)

@Serializable
class User(val name: String)

data class SampleData(
val amount: Long,
val description: String?,
val department: String = "QA"
)
fun main() {
val data = Project("kotlinx.serialization", User("kotlin"))
val map = Properties.encodeToMap(data)
map.forEach { (k, v) -> println("$k = $v") }
val descriptors = listOf(SampleData.serializer().descriptor)
val schemas = ProtoBufSchemaGenerator.generateSchemaText(descriptors)
println(schemas)
}
32 changes: 7 additions & 25 deletions guide/example/example-formats-10.kt
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,17 @@
package example.exampleFormats10

import kotlinx.serialization.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*
import kotlinx.serialization.modules.*

class ListEncoder : AbstractEncoder() {
val list = mutableListOf<Any>()

override val serializersModule: SerializersModule = EmptySerializersModule()

override fun encodeValue(value: Any) {
list.add(value)
}
}

fun <T> encodeToList(serializer: SerializationStrategy<T>, value: T): List<Any> {
val encoder = ListEncoder()
encoder.encodeSerializableValue(serializer, value)
return encoder.list
}

inline fun <reified T> encodeToList(value: T) = encodeToList(serializer(), value)
import kotlinx.serialization.properties.Properties // todo: remove when no longer needed
import kotlinx.serialization.properties.*

@Serializable
data class Project(val name: String, val owner: User, val votes: Int)
class Project(val name: String, val owner: User)

@Serializable
data class User(val name: String)
class User(val name: String)

fun main() {
val data = Project("kotlinx.serialization", User("kotlin"), 9000)
println(encodeToList(data))
val data = Project("kotlinx.serialization", User("kotlin"))
val map = Properties.encodeToMap(data)
map.forEach { (k, v) -> println("$k = $v") }
}
28 changes: 1 addition & 27 deletions guide/example/example-formats-11.kt
Original file line number Diff line number Diff line change
Expand Up @@ -24,29 +24,6 @@ fun <T> encodeToList(serializer: SerializationStrategy<T>, value: T): List<Any>

inline fun <reified T> encodeToList(value: T) = encodeToList(serializer(), value)

class ListDecoder(val list: ArrayDeque<Any>) : AbstractDecoder() {
private var elementIndex = 0

override val serializersModule: SerializersModule = EmptySerializersModule()

override fun decodeValue(): Any = list.removeFirst()

override fun decodeElementIndex(descriptor: SerialDescriptor): Int {
if (elementIndex == descriptor.elementsCount) return CompositeDecoder.DECODE_DONE
return elementIndex++
}

override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder =
ListDecoder(list)
}

fun <T> decodeFromList(list: List<Any>, deserializer: DeserializationStrategy<T>): T {
val decoder = ListDecoder(ArrayDeque(list))
return decoder.decodeSerializableValue(deserializer)
}

inline fun <reified T> decodeFromList(list: List<Any>): T = decodeFromList(list, serializer())

@Serializable
data class Project(val name: String, val owner: User, val votes: Int)

Expand All @@ -55,8 +32,5 @@ data class User(val name: String)

fun main() {
val data = Project("kotlinx.serialization", User("kotlin"), 9000)
val list = encodeToList(data)
println(list)
val obj = decodeFromList<Project>(list)
println(obj)
println(encodeToList(data))
}
6 changes: 2 additions & 4 deletions guide/example/example-formats-12.kt
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,8 @@ class ListDecoder(val list: ArrayDeque<Any>) : AbstractDecoder() {
}

override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder =
ListDecoder(list)

override fun decodeSequentially(): Boolean = true
}
ListDecoder(list)
}

fun <T> decodeFromList(list: List<Any>, deserializer: DeserializationStrategy<T>): T {
val decoder = ListDecoder(ArrayDeque(list))
Expand Down
24 changes: 8 additions & 16 deletions guide/example/example-formats-13.kt
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,7 @@ class ListEncoder : AbstractEncoder() {

override fun encodeValue(value: Any) {
list.add(value)
}

override fun beginCollection(descriptor: SerialDescriptor, collectionSize: Int): CompositeEncoder {
encodeInt(collectionSize)
return this
}
}
}

fun <T> encodeToList(serializer: SerializationStrategy<T>, value: T): List<Any> {
Expand All @@ -29,26 +24,23 @@ fun <T> encodeToList(serializer: SerializationStrategy<T>, value: T): List<Any>

inline fun <reified T> encodeToList(value: T) = encodeToList(serializer(), value)

class ListDecoder(val list: ArrayDeque<Any>, var elementsCount: Int = 0) : AbstractDecoder() {
class ListDecoder(val list: ArrayDeque<Any>) : AbstractDecoder() {
private var elementIndex = 0

override val serializersModule: SerializersModule = EmptySerializersModule()

override fun decodeValue(): Any = list.removeFirst()

override fun decodeElementIndex(descriptor: SerialDescriptor): Int {
if (elementIndex == elementsCount) return CompositeDecoder.DECODE_DONE
if (elementIndex == descriptor.elementsCount) return CompositeDecoder.DECODE_DONE
return elementIndex++
}

override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder =
ListDecoder(list, descriptor.elementsCount)
ListDecoder(list)

override fun decodeSequentially(): Boolean = true

override fun decodeCollectionSize(descriptor: SerialDescriptor): Int =
decodeInt().also { elementsCount = it }
}
}

fun <T> decodeFromList(list: List<Any>, deserializer: DeserializationStrategy<T>): T {
val decoder = ListDecoder(ArrayDeque(list))
Expand All @@ -58,13 +50,13 @@ fun <T> decodeFromList(list: List<Any>, deserializer: DeserializationStrategy<T>
inline fun <reified T> decodeFromList(list: List<Any>): T = decodeFromList(list, serializer())

@Serializable
data class Project(val name: String, val owners: List<User>, val votes: Int)
data class Project(val name: String, val owner: User, val votes: Int)

@Serializable
data class User(val name: String)

fun main() {
val data = Project("kotlinx.serialization", listOf(User("kotlin"), User("jetbrains")), 9000)
val data = Project("kotlinx.serialization", User("kotlin"), 9000)
val list = encodeToList(data)
println(list)
val obj = decodeFromList<Project>(list)
Expand Down
Loading

0 comments on commit e5e703e

Please sign in to comment.