From fa88b49d883936e3622b1e581319bc85bfbd6584 Mon Sep 17 00:00:00 2001 From: Ashhar Hasan Date: Fri, 14 Apr 2023 14:57:38 +0530 Subject: [PATCH] Document Protobuf oneof support in Kafka connector --- docs/src/main/sphinx/connector/kafka.rst | 42 ++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/src/main/sphinx/connector/kafka.rst b/docs/src/main/sphinx/connector/kafka.rst index c4f7e848714..12f698f8b15 100644 --- a/docs/src/main/sphinx/connector/kafka.rst +++ b/docs/src/main/sphinx/connector/kafka.rst @@ -418,6 +418,7 @@ table description supplier are: * New tables can be defined without a cluster restart. * Schema updates are detected automatically. * There is no need to define tables manually. +* Some Protobuf specific types like ``oneof`` are supported and mapped to JSON. Set ``kafka.table-description-supplier`` to ``CONFLUENT`` to use the schema registry. You must also configure the additional properties in the following table: @@ -479,6 +480,45 @@ used to resolve the subject name via the topic name. Note that a case insensitive match must be done, as identifiers cannot contain upper case characters. +Protobuf-specific type handling in Confluent table description supplier +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +When using the Confluent table description supplier, the following Protobuf +specific types are supported in addition to the :ref:`normally supported types +`: + +oneof ++++++ + +Protobuf schemas containing ``oneof`` fields are mapped to a ``JSON`` field in +Trino. + +For example, given the following Protobuf schema: + +.. code-block:: text + + syntax = "proto3"; + + message schema { + oneof test_oneof_column { + string string_column = 1; + uint32 integer_column = 2; + uint64 long_column = 3; + double double_column = 4; + float float_column = 5; + bool boolean_column = 6; + } + } + +The corresponding Trino row is a ``JSON`` field ``test_oneof_column`` +containing a JSON object with a single key. The value of the key matches +the name of the ``oneof`` type that is present. + +In the above example, if the Protobuf message has the +``test_oneof_column`` containing ``string_column`` set to a value ``Trino`` +then the corresponding Trino row includes a column named +``test_oneof_column`` with the value ``JSON '{"string_column": "Trino"}'``. + .. _kafka-sql-inserts: Kafka inserts @@ -1350,6 +1390,8 @@ The schema evolution behavior is as follows: If the type coercion is supported by Avro, then the conversion happens. An error is thrown for incompatible types. +.. _kafka-protobuf-decoding: + Protobuf decoder """"""""""""""""