diff --git a/doc/user/content/sql/create-sink/kafka.md b/doc/user/content/sql/create-sink/kafka.md index aac98df3bcbcd..e3b9f39be6f14 100644 --- a/doc/user/content/sql/create-sink/kafka.md +++ b/doc/user/content/sql/create-sink/kafka.md @@ -56,9 +56,12 @@ _item_name_ | The name of the source, table or materialized view you want ### `CONNECTION` options -Field | Value | Description ----------------------|--------|------------ -`TOPIC` | `text` | The prefix used to generate the Kafka topic name to create and write to. +Field | Value | Description +---------------------------|--------|------------ +`TOPIC` | `text` | The name of the Kafka topic to write to. +`TRANSACTIONAL ID` | `text` | Sets the transactional ID to use when producing to the Kafka topic.
Default: `materialize-{REGION ID}-{CONNECTION ID}-{SINK ID}` +`PROGRESS GROUP ID PREFIX` | `text` | Sets the prefix of the consumer group ID to use when reading from the progress topic.
Default: `materialize-{REGION ID}-{CONNECTION ID}-{SINK ID}` + ### CSR `CONNECTION` options @@ -367,9 +370,9 @@ to perform the following operations on the following resources: Operation type | Resource type | Resource name ----------------|------------------|-------------- Read, Write | Topic | Consult `mz_kafka_connections.sink_progress_topic` for the sink's connection -Write | Topic | The specified `TOPIC` option -Write | Transactional ID | `mz-producer-{SINK ID}-*` -Read | Group | `materialize-bootstrap-sink-{SINK ID}` +Write | Topic | The specified [`TOPIC` option](#connection-options) +Write | Transactional ID | The specified [`TRANSACTIONAL ID` option](#connection-options) +Read | Group | All group IDs beginning with the specified [`PROGRESS GROUP ID PREFIX` option](#connection-options) When using [automatic topic creation](#automatic-topic-creation), Materialize additionally requires access to the following operations: diff --git a/doc/user/content/sql/create-source/kafka.md b/doc/user/content/sql/create-source/kafka.md index 908656e5fe9ea..1e9625709d878 100644 --- a/doc/user/content/sql/create-source/kafka.md +++ b/doc/user/content/sql/create-source/kafka.md @@ -49,7 +49,7 @@ The same syntax, supported formats and features can be used to connect to a [Red Field | Value | Description -------------------------------------|-----------|------------------------------------- `TOPIC` | `text` | The Kafka topic you want to subscribe to. -`GROUP ID PREFIX` | `text` | The prefix of the consumer group ID to use. Materialize may append a suffix to the provided prefix. See [Monitoring consumer lag](#monitoring-consumer-lag).
Default: `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}` +`GROUP ID PREFIX` | `text` | Sets the prefix of the consumer group ID. See [Monitoring consumer lag](#monitoring-consumer-lag).
Default: `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}` ### `WITH` options @@ -362,15 +362,17 @@ provided solely for the benefit of Kafka monitoring tools. {{< /note >}} Committed offsets are associated with a consumer group specific to the source. -The ID of the consumer group has a prefix with the following format: +The ID of the consumer group consists of the prefix configured with the [`GROUP +ID PREFIX` option](#connection-options) followed by a Materialize-generated +suffix. You should not make assumptions about the number of consumer groups that Materialize will use to consume from a given source. The only guarantee is that -the ID of each consumer group will begin with the above prefix. +the ID of each consumer group will begin with the configured prefix. -The rendered consumer group ID prefix for each Kafka source in the system is -available in the `group_id_prefix` column of the [`mz_kafka_sources`] table. To -look up the `group_id_prefix` for a source by name, use: +The consumer group ID prefix for each Kafka source in the system is available in +the `group_id_prefix` column of the [`mz_kafka_sources`] table. To look up the +`group_id_prefix` for a source by name, use: ```sql SELECT group_id_prefix @@ -387,13 +389,7 @@ to perform the following operations on the following resources: Operation type | Resource type | Resource name ---------------|------------------|-------------- Read | Topic | The specified `TOPIC` option - -To allow Materialize to [commit offsets](#monitoring-consumer-lag) to the Kafka -broker, Materialize additionally requires access to the following operations: - -Operation type | Resource type | Resource name ----------------|------------------|-------------- -Read | Group | `materialize-{REGION-ID}-{CONNECTION-ID}-{SOURCE_ID}*` +Read | Group | All group IDs starting with the specified [`GROUP ID PREFIX` option](#connection-options) ## Examples