doc/user: document configurable group and transactional IDs for Kafka

This commit adds documentation for the features added in MaterializeInc#23792. See that PR for details.
benesch · Dec 10, 2023 · 614c0ab · 614c0ab
1 parent 9da5968
commit 614c0ab
Show file tree

Hide file tree

Showing 2 changed files with 18 additions and 19 deletions.
diff --git a/doc/user/content/sql/create-sink/kafka.md b/doc/user/content/sql/create-sink/kafka.md
@@ -56,9 +56,12 @@ _item&lowbar;name_ | The name of the source, table or materialized view you want
 
 ### `CONNECTION` options
 
-Field                | Value  | Description
----------------------|--------|------------
-`TOPIC`              | `text` | The prefix used to generate the Kafka topic name to create and write to.
+Field                      | Value  | Description
+---------------------------|--------|------------
+`TOPIC`                    | `text` | The name of the Kafka topic to write to.
+`TRANSACTIONAL ID`         | `text` | Sets the transactional ID to use when producing to the Kafka topic.<br>Default: `materialize-{REGION ID}-{CONNECTION ID}-{SINK ID}`
+`PROGRESS GROUP ID PREFIX` | `text` | Sets the prefix of the consumer group ID to use when reading from the progress topic.<br>Default: `materialize-{REGION ID}-{CONNECTION ID}-{SINK ID}`
+
 
 ### CSR `CONNECTION` options
 
@@ -367,9 +370,9 @@ to perform the following operations on the following resources:
 Operation type  | Resource type    | Resource name
 ----------------|------------------|--------------
 Read, Write     | Topic            | Consult `mz_kafka_connections.sink_progress_topic` for the sink's connection
-Write           | Topic            | The specified `TOPIC` option
-Write           | Transactional ID | `mz-producer-{SINK ID}-*`
-Read            | Group            | `materialize-bootstrap-sink-{SINK ID}`
+Write           | Topic            | The specified [`TOPIC` option](#connection-options)
+Write           | Transactional ID | The specified [`TRANSACTIONAL ID` option](#connection-options)
+Read            | Group            | All group IDs beginning with the specified [`PROGRESS GROUP ID PREFIX` option](#connection-options)
 
 When using [automatic topic creation](#automatic-topic-creation), Materialize
 additionally requires access to the following operations:

diff --git a/doc/user/content/sql/create-source/kafka.md b/doc/user/content/sql/create-source/kafka.md
@@ -49,7 +49,7 @@ The same syntax, supported formats and features can be used to connect to a [Red
 Field                                | Value     | Description
 -------------------------------------|-----------|-------------------------------------
 `TOPIC`                              | `text`    | The Kafka topic you want to subscribe to.
-`GROUP ID PREFIX`                    | `text`    | The prefix of the consumer group ID to use. Materialize may append a suffix to the provided prefix. See [Monitoring consumer lag](#monitoring-consumer-lag).<br>Default: `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}`
+`GROUP ID PREFIX`                    | `text`    | Sets the prefix of the consumer group ID. See [Monitoring consumer lag](#monitoring-consumer-lag).<br>Default: `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}`
 
 ### `WITH` options
 
@@ -362,15 +362,17 @@ provided solely for the benefit of Kafka monitoring tools.
 {{< /note >}}
 
 Committed offsets are associated with a consumer group specific to the source.
-The ID of the consumer group has a prefix with the following format:
+The ID of the consumer group consists of the prefix configured with the [`GROUP
+ID PREFIX` option](#connection-options) followed by a Materialize-generated
+suffix.
 
 You should not make assumptions about the number of consumer groups that
 Materialize will use to consume from a given source. The only guarantee is that
-the ID of each consumer group will begin with the above prefix.
+the ID of each consumer group will begin with the configured prefix.
 
-The rendered consumer group ID prefix for each Kafka source in the system is
-available in the `group_id_prefix` column of the [`mz_kafka_sources`] table. To
-look up the `group_id_prefix` for a source by name, use:
+The consumer group ID prefix for each Kafka source in the system is available in
+the `group_id_prefix` column of the [`mz_kafka_sources`] table. To look up the
+`group_id_prefix` for a source by name, use:
 
 ```sql
 SELECT group_id_prefix
@@ -387,13 +389,7 @@ to perform the following operations on the following resources:
 Operation type | Resource type    | Resource name
 ---------------|------------------|--------------
 Read           | Topic            | The specified `TOPIC` option
-
-To allow Materialize to [commit offsets](#monitoring-consumer-lag) to the Kafka
-broker, Materialize additionally requires access to the following operations:
-
-Operation type | Resource type    | Resource name
----------------|------------------|--------------
-Read           | Group            | `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}*`
+Read           | Group            | All group IDs starting with the specified [`GROUP ID PREFIX` option](#connection-options)
 
 ## Examples