Skip to content

Commit

Permalink
Document Protobuf oneof support in Kafka connector
Browse files Browse the repository at this point in the history
  • Loading branch information
hashhar committed Apr 23, 2023
1 parent 02cc332 commit d6f3759
Showing 1 changed file with 42 additions and 0 deletions.
42 changes: 42 additions & 0 deletions docs/src/main/sphinx/connector/kafka.rst
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,7 @@ table description supplier are:
* New tables can be defined without a cluster restart.
* Schema updates are detected automatically.
* There is no need to define tables manually.
* Some Protobuf specific types like ``oneof`` are supported and mapped to JSON.

Set ``kafka.table-description-supplier`` to ``CONFLUENT`` to use the
schema registry. You must also configure the additional properties in the following table:
Expand Down Expand Up @@ -479,6 +480,45 @@ used to resolve the subject name via the topic name. Note that a case
insensitive match must be done, as identifiers cannot contain upper case
characters.

Protobuf-specific type handling in Confluent table description supplier
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

When using the Confluent table description supplier, the following Protobuf
specific types are supported in addition to the :ref:`normally supported types
<kafka-protobuf-decoding>`:

oneof
+++++

Protobuf schemas containing ``oneof`` fields are mapped to a ``JSON`` field in
Trino.

For example, given the following Protobuf schema:

.. code-block:: text
syntax = "proto3";
message schema {
oneof test_oneof_column {
string string_column = 1;
uint32 integer_column = 2;
uint64 long_column = 3;
double double_column = 4;
float float_column = 5;
bool boolean_column = 6;
}
}
The corresponding Trino row is a ``JSON`` field ``test_oneof_column``
containing a JSON object with a single key. The value of the key matches
the name of the ``oneof`` type that is present.

In the above example, if the Protobuf message has the
``test_oneof_column`` containing ``string_column`` set to a value ``Trino``
then the corresponding Trino row includes a column named
``test_oneof_column`` with the value ``JSON '{"string_column": "Trino"}'``.

.. _kafka-sql-inserts:

Kafka inserts
Expand Down Expand Up @@ -1350,6 +1390,8 @@ The schema evolution behavior is as follows:
If the type coercion is supported by Avro, then the conversion happens. An
error is thrown for incompatible types.

.. _kafka-protobuf-decoding:

Protobuf decoder
""""""""""""""""

Expand Down

0 comments on commit d6f3759

Please sign in to comment.