Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL source resource #486

Merged
merged 11 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ services:
container_name: mysql
image: mysql:8.0
volumes:
- ./integration/mysql/mysql.cnf:/etc/mysql/conf.d/mysql.cnf
- ./integration/mysql:/docker-entrypoint-initdb.d
ports:
- 3306:3306
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_aws_privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ resource "materialize_connection_aws_privatelink" "example_privatelink_connectio
### Optional

- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.

### Read-Only

Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_confluent_schema_registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ resource "materialize_connection_confluent_schema_registry" "example_confluent_s

- `aws_privatelink` (Block List, Max: 1) The AWS PrivateLink configuration for the Confluent Schema Registry. (see [below for nested schema](#nestedblock--aws_privatelink))
- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `password` (Block List, Max: 1) The password for the Confluent Schema Registry. (see [below for nested schema](#nestedblock--password))
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.
- `ssh_tunnel` (Block List, Max: 1) The SSH tunnel configuration for the Confluent Schema Registry. (see [below for nested schema](#nestedblock--ssh_tunnel))
- `ssl_certificate` (Block List, Max: 1) The client certificate for the Confluent Schema Registry.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate))
- `ssl_certificate_authority` (Block List, Max: 1) The CA certificate for the Confluent Schema Registry.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate_authority))
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,15 +82,15 @@ resource "materialize_connection_kafka" "example_kafka_connection_multiple_broke

- `aws_privatelink` (Block List, Max: 1) AWS PrivateLink configuration. Conflicts with `kafka_broker`. (see [below for nested schema](#nestedblock--aws_privatelink))
- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `kafka_broker` (Block List) The Kafka broker's configuration. (see [below for nested schema](#nestedblock--kafka_broker))
- `ownership_role` (String) The owernship role of the object.
- `progress_topic` (String) The name of a topic that Kafka sinks can use to track internal consistency metadata.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `sasl_mechanisms` (String) The SASL mechanism for the Kafka broker.
- `sasl_password` (Block List, Max: 1) The SASL password for the Kafka broker. (see [below for nested schema](#nestedblock--sasl_password))
- `sasl_username` (Block List, Max: 1) The SASL username for the Kafka broker.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--sasl_username))
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.
- `security_protocol` (String) The security protocol to use: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`.
- `ssh_tunnel` (Block List, Max: 1) The default SSH tunnel configuration for the Kafka brokers. (see [below for nested schema](#nestedblock--ssh_tunnel))
- `ssl_certificate` (Block List, Max: 1) The client certificate for the Kafka broker.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate))
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,12 @@ resource "materialize_connection_mysql" "example_mysql_connection" {

- `aws_privatelink` (Block List, Max: 1) The AWS PrivateLink configuration for the MySQL database. (see [below for nested schema](#nestedblock--aws_privatelink))
- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `password` (Block List, Max: 1) The MySQL database password. (see [below for nested schema](#nestedblock--password))
- `port` (Number) The MySQL database port.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.
- `ssh_tunnel` (Block List, Max: 1) The SSH tunnel configuration for the MySQL database. (see [below for nested schema](#nestedblock--ssh_tunnel))
- `ssl_certificate` (Block List, Max: 1) The client certificate for the MySQL database.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate))
- `ssl_certificate_authority` (Block List, Max: 1) The CA certificate for the MySQL database.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate_authority))
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,12 @@ resource "materialize_connection_postgres" "example_postgres_connection" {

- `aws_privatelink` (Block List, Max: 1) The AWS PrivateLink configuration for the Postgres database. (see [below for nested schema](#nestedblock--aws_privatelink))
- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `password` (Block List, Max: 1) The Postgres database password. (see [below for nested schema](#nestedblock--password))
- `port` (Number) The Postgres database port.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.
- `ssh_tunnel` (Block List, Max: 1) The SSH tunnel configuration for the Postgres database. (see [below for nested schema](#nestedblock--ssh_tunnel))
- `ssl_certificate` (Block List, Max: 1) The client certificate for the Postgres database.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate))
- `ssl_certificate_authority` (Block List, Max: 1) The CA certificate for the Postgres database.. Can be supplied as either free text using `text` or reference to a secret object using `secret`. (see [below for nested schema](#nestedblock--ssl_certificate_authority))
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/connection_ssh_tunnel.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,10 @@ resource "materialize_connection_ssh_tunnel" "example_ssh_connection" {
### Optional

- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the connection database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the connection database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the connection schema. Defaults to `public`.
- `schema_name` (String) The identifier for the connection schema in Materialize. Defaults to `public`.

### Read-Only

Expand Down
4 changes: 2 additions & 2 deletions docs/resources/materialized_view.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ resource "materialize_materialized_view" "simple_materialized_view" {
### Optional

- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the materialized view database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the materialized view database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `not_null_assertion` (List of String) **Private Preview** A list of columns for which to create non-null assertions.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the materialized view schema. Defaults to `public`.
- `schema_name` (String) The identifier for the materialized view schema in Materialize. Defaults to `public`.

### Read-Only

Expand Down
2 changes: 1 addition & 1 deletion docs/resources/schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ resource "materialize_schema" "example_schema" {
### Optional

- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the schema database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the schema database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.

Expand Down
4 changes: 2 additions & 2 deletions docs/resources/secret.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ resource "materialize_secret" "example_secret" {
### Optional

- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the secret database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the secret database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the secret schema. Defaults to `public`.
- `schema_name` (String) The identifier for the secret schema in Materialize. Defaults to `public`.

### Read-Only

Expand Down
7 changes: 3 additions & 4 deletions docs/resources/sink_kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,7 @@ resource "materialize_sink_kafka" "example_sink_kafka" {
# FROM schema.table
# INTO KAFKA CONNECTION "kafka_connection" (TOPIC 'test_avro_topic')
# FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION "database"."schema"."csr_connection"
# ENVELOPE UPSERT
# WITH (SIZE = '3xsmall');
# ENVELOPE UPSERT;
```

<!-- schema generated by tfplugindocs -->
Expand All @@ -61,14 +60,14 @@ resource "materialize_sink_kafka" "example_sink_kafka" {
- `cluster_name` (String) The cluster to maintain this sink.
- `comment` (String) **Public Preview** Comment on an object in the database.
- `compression_type` (String) The type of compression to apply to messages before they are sent to Kafka.
- `database_name` (String) The identifier for the sink database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the sink database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `envelope` (Block List, Max: 1) How to interpret records (e.g. Debezium, Upsert). (see [below for nested schema](#nestedblock--envelope))
- `format` (Block List, Max: 1) How to decode raw bytes from different formats into data structures it can understand at runtime. (see [below for nested schema](#nestedblock--format))
- `key` (List of String) An optional list of columns to use for the Kafka key. If unspecified, the Kafka key is left unset.
- `key_not_enforced` (Boolean) Disable Materialize's validation of the key's uniqueness.
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the sink schema. Defaults to `public`.
- `schema_name` (String) The identifier for the sink schema in Materialize. Defaults to `public`.
- `snapshot` (Boolean) Whether to emit the consolidated results of the query before the sink was created at the start of the sink.

### Read-Only
Expand Down
7 changes: 3 additions & 4 deletions docs/resources/source_kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,7 @@ resource "materialize_source_kafka" "example_source_kafka" {
# CREATE SOURCE kafka_metadata
# FROM KAFKA CONNECTION "database"."schema"."kafka_connection" (TOPIC 'data')
# FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION "database"."schema"."csr_connection"
# ENVELOPE NONE
# WITH (SIZE = '3xsmall');
# ENVELOPE NONE;
```

<!-- schema generated by tfplugindocs -->
Expand All @@ -56,7 +55,7 @@ resource "materialize_source_kafka" "example_source_kafka" {

- `cluster_name` (String) The cluster to maintain this source.
- `comment` (String) **Public Preview** Comment on an object in the database.
- `database_name` (String) The identifier for the source database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the source database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `envelope` (Block List, Max: 1) How Materialize should interpret records (e.g. append-only, upsert).. (see [below for nested schema](#nestedblock--envelope))
- `expose_progress` (Block List, Max: 1) The name of the progress subsource for the source. If this is not specified, the subsource will be named `<src_name>_progress`. (see [below for nested schema](#nestedblock--expose_progress))
- `format` (Block List, Max: 1) How to decode raw bytes from different formats into data structures Materialize can understand at runtime. (see [below for nested schema](#nestedblock--format))
Expand All @@ -73,7 +72,7 @@ resource "materialize_source_kafka" "example_source_kafka" {
- `key_format` (Block List, Max: 1) Set the key format explicitly. (see [below for nested schema](#nestedblock--key_format))
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the source schema. Defaults to `public`.
- `schema_name` (String) The identifier for the source schema in Materialize. Defaults to `public`.
- `start_offset` (List of Number) Read partitions from the specified offset.
- `start_timestamp` (Number) Use the specified value to set `START OFFSET` based on the Kafka timestamp.
- `value_format` (Block List, Max: 1) Set the value format explicitly. (see [below for nested schema](#nestedblock--value_format))
Expand Down
7 changes: 3 additions & 4 deletions docs/resources/source_load_generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,7 @@ resource "materialize_source_load_generator" "example_source_load_generator" {

# CREATE SOURCE schema.source_load_generator
# FROM LOAD GENERATOR COUNTER
# (TICK INTERVAL '500ms' SCALE FACTOR 0.01)
# WITH (SIZE = '3xsmall');
# (TICK INTERVAL '500ms' SCALE FACTOR 0.01);
```

<!-- schema generated by tfplugindocs -->
Expand All @@ -46,12 +45,12 @@ resource "materialize_source_load_generator" "example_source_load_generator" {
- `cluster_name` (String) The cluster to maintain this source.
- `comment` (String) **Public Preview** Comment on an object in the database.
- `counter_options` (Block List, Max: 1) Counter Options. (see [below for nested schema](#nestedblock--counter_options))
- `database_name` (String) The identifier for the source database. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `database_name` (String) The identifier for the source database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `expose_progress` (Block List, Max: 1) The name of the progress subsource for the source. If this is not specified, the subsource will be named `<src_name>_progress`. (see [below for nested schema](#nestedblock--expose_progress))
- `marketing_options` (Block List, Max: 1) Marketing Options. (see [below for nested schema](#nestedblock--marketing_options))
- `ownership_role` (String) The owernship role of the object.
- `region` (String) The region to use for the resource connection. If not set, the default region is used.
- `schema_name` (String) The identifier for the source schema. Defaults to `public`.
- `schema_name` (String) The identifier for the source schema in Materialize. Defaults to `public`.
- `tpch_options` (Block List, Max: 1) TPCH Options. (see [below for nested schema](#nestedblock--tpch_options))

### Read-Only
Expand Down
Loading
Loading