Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: Update Debezium CDC guides to v2.x #22434

Merged
merged 5 commits into from
Nov 10, 2023
Merged

Doc: Update Debezium CDC guides to v2.x #22434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3f211ce
Doc: Update Debezium CDC guides to v2.x
bobbyiliev Oct 17, 2023
85cd3ac
Doc: Update MySQL Debezium CDC guides to v2.x
bobbyiliev Oct 17, 2023
d6b2cd4
Update doc/user/content/ingest-data/cdc-mysql.md
bobbyiliev Nov 10, 2023
e4ebb03
Update doc/user/content/ingest-data/cdc-mysql.md
bobbyiliev Nov 10, 2023
615ed5d
Add link to the debezium avro docs
bobbyiliev Nov 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 52 additions & 2 deletions doc/user/content/ingest-data/cdc-mysql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ connectors, so you first need to define a MySQL connector configuration and then
If you deploy the MySQL Debezium connector in [Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mysql-source-cdc-debezium.html), you **must** override the default value of `After-state only` to `false`.
{{</ warning >}}

{{< tabs >}}
{{< tab "Debezium 1.5+">}}

1. Create a connector configuration file and save it as `register-mysql.json`:

```json
Expand All @@ -78,6 +81,55 @@ If you deploy the MySQL Debezium connector in [Confluent Cloud](https://docs.con

You can read more about each configuration property in the [Debezium documentation](https://debezium.io/documentation/reference/connectors/mysql.html#mysql-connector-properties).

{{< /tab >}}
{{< tab "Debezium 2.0+">}}

1. From Debezium 2.0, Confluent Schema Registry (CSR) support is not bundled in Debezium containers. To enable CSR, you must install the following Confluent Avro converter JAR files into the Kafka Connect plugin directory (by default, `/kafka/connect`):

* `kafka-connect-avro-converter`
* `kafka-connect-avro-data`
* `kafka-avro-serializer`
* `kafka-schema-serializer`
* `kafka-schema-registry-client`
* `common-config`
* `common-utils`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we point to the Debezium documentation here for more details?


You can read more about this in the [Debezium documentation](https://debezium.io/documentation/reference/stable/configuration/avro.html#deploying-confluent-schema-registry-with-debezium-containers).

1. Create a connector configuration file and save it as `register-mysql.json`:

```json
{
"name": "your-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "user",
"database.password": "mysqlpwd",
"database.server.id":"223344",
"topic.prefix": "dbserver1",
"database.include.list": "db1",
"database.history.kafka.topic":"dbserver1.history",
"database.history.kafka.bootstrap.servers":"kafka:9092",
"schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
"schema.history.internal.kafka.topic": "dbserver1.internal.history",
"table.include.list": "table1",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://<scheme-registry>:8081",
"value.converter.schema.registry.url": "http://<scheme-registry>:8081",
"include.schema.changes": false
}
}
```

You can read more about each configuration property in the [Debezium documentation](https://debezium.io/documentation/reference/2.4/connectors/mysql.html). By default, the connector writes events for each table to a Kafka topic named `serverName.databaseName.tableName`.

{{< /tab >}}
{{< /tabs >}}

1. Start the Debezium MySQL connector using the configuration file:

```bash
Expand All @@ -101,8 +153,6 @@ If you deploy the MySQL Debezium connector in [Confluent Cloud](https://docs.con
--topic dbserver1.db1.table1
```

**Note:** By default, the connector writes events for each table to a Kafka topic named `serverName.databaseName.tableName`.

### Create a source

{{< debezium-json >}}
Expand Down
50 changes: 50 additions & 0 deletions doc/user/content/ingest-data/cdc-postgres-kafka-debezium.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,9 @@ Debezium is deployed as a set of Kafka Connect-compatible connectors, so you fir
If you deploy the PostgreSQL Debezium connector in [Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mysql-source-cdc-debezium.html), you **must** override the default value of `After-state only` to `false`.
{{</ warning >}}

{{< tabs >}}
{{< tab "Debezium 1.5+">}}

1. Create a connector configuration file and save it as `register-postgres.json`:

```json
Expand Down Expand Up @@ -159,6 +162,53 @@ If you deploy the PostgreSQL Debezium connector in [Confluent Cloud](https://doc
You can read more about each configuration property in the [Debezium documentation](https://debezium.io/documentation/reference/1.6/connectors/postgresql.html#postgresql-connector-properties). By default, the connector writes events for each table to a Kafka topic named `serverName.schemaName.tableName`.


{{< /tab >}}
{{< tab "Debezium 2.0+">}}

1. Beginning with Debezium 2.0.0, Confluent Schema Registry support is not included in the Debezium containers. To enable the Confluent Schema Registry for a Debezium container, install the following Confluent Avro converter JAR files into the Connect plugin directory:

* `kafka-connect-avro-converter`
* `kafka-connect-avro-data`
* `kafka-avro-serializer`
* `kafka-schema-serializer`
* `kafka-schema-registry-client`
* `common-config`
* `common-utils`

You can read more about this in the [Debezium documentation](https://debezium.io/documentation/reference/stable/configuration/avro.html#deploying-confluent-schema-registry-with-debezium-containers).

1. Create a connector configuration file and save it as `register-postgres.json`:

```json
{
"name": "your-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"plugin.name":"pgoutput",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "postgres",
"topic.prefix": "pg_repl",
"schema.include.list": "public",
"table.include.list": "public.table1",
"publication.autocreate.mode":"filtered",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://<scheme-registry>:8081",
"value.converter.schema.registry.url": "http://<scheme-registry>:8081",
"value.converter.schemas.enable": false
}
}
```

You can read more about each configuration property in the [Debezium documentation](https://debezium.io/documentation/reference/2.4/connectors/postgresql.html#postgresql-connector-properties). By default, the connector writes events for each table to a Kafka topic named `serverName.schemaName.tableName`.

{{< /tab >}}
{{< /tabs >}}

1. Start the Debezium Postgres connector using the configuration file:

```bash
Expand Down