Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source versioning: Postgres, MySQL and Load generator #647

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
ce42144
Source versioning initial implementation
bobbyiliev Sep 3, 2024
decb60f
Use source table instead of table from source
bobbyiliev Sep 3, 2024
e4b5883
MySQL source: separate for tables and all tables
bobbyiliev Sep 3, 2024
d8612c1
Loadgen source: add all tables bool attr
bobbyiliev Sep 3, 2024
cb3b49a
Add tests
bobbyiliev Sep 4, 2024
718bf71
Add more tests for mysql and loadgen
bobbyiliev Sep 4, 2024
eb36ba3
Add ignore columns for MySQL
bobbyiliev Sep 4, 2024
bf5196b
Add source_id logic
bobbyiliev Sep 9, 2024
6864079
Add source table migration guide
bobbyiliev Sep 9, 2024
99d2f62
Add deprecated message
bobbyiliev Sep 9, 2024
74593f3
Ignore text columns for load gen source tables
bobbyiliev Sep 11, 2024
f8fc659
Refactor source table for individual sources
bobbyiliev Sep 13, 2024
a4d0eba
Add datasource
bobbyiliev Sep 13, 2024
191649c
Format examples
bobbyiliev Sep 13, 2024
dd1f993
Add Kafka source table resource
bobbyiliev Sep 16, 2024
6ab37e5
Review updates
bobbyiliev Sep 16, 2024
82ee6b8
Update guide migration guide
bobbyiliev Sep 16, 2024
bb48f92
Update guide migration guide
bobbyiliev Sep 16, 2024
ec8d61a
Add import examples for Kafka source tables
bobbyiliev Sep 17, 2024
a9017a0
Add upstream mysql and postgres table names
bobbyiliev Sep 20, 2024
640996f
Fix unit tests
bobbyiliev Sep 20, 2024
c34e8ac
Add Kafka upstream references
bobbyiliev Sep 20, 2024
bb2f61c
Add integration tests
bobbyiliev Sep 24, 2024
6330f06
Fix failing test
bobbyiliev Sep 24, 2024
6ecc0b4
Extend data source to include upstream names
bobbyiliev Sep 24, 2024
bc2298d
Small updates
bobbyiliev Sep 27, 2024
12f00aa
Switch back to latest image
bobbyiliev Sep 30, 2024
0358960
FromAsCasing: 'as' and 'FROM' keywords' casing do not match
bobbyiliev Oct 6, 2024
a624bfc
Add source reference data source
bobbyiliev Oct 6, 2024
2e2ec9a
Add source reference data source example
bobbyiliev Oct 6, 2024
89ebfb3
First round of the initial PR change requests
bobbyiliev Oct 14, 2024
29d8b48
Fix failing tests
bobbyiliev Oct 14, 2024
e8e322f
Fix failing tests
bobbyiliev Oct 14, 2024
cab7a27
Second round of the initial PR change requests
bobbyiliev Oct 15, 2024
3312ebd
Add unit tests to data source source reference
bobbyiliev Oct 15, 2024
4c3d251
PR change requests
bobbyiliev Oct 21, 2024
620e020
Fix failing tests
bobbyiliev Oct 21, 2024
abd069c
Fix failing tests
bobbyiliev Oct 21, 2024
5d442dd
Add alter source refresh to data source
bobbyiliev Oct 28, 2024
e57d5f0
Add new kafk acolumns from mz_kafka_source_tables
bobbyiliev Oct 28, 2024
7d57eb5
Fix failing MockSourceTableKafkaScan test
bobbyiliev Oct 28, 2024
d3c6ed2
Remove confusing line from migration guide
bobbyiliev Oct 29, 2024
cd69c11
Remove a left behind comment
bobbyiliev Nov 18, 2024
e3bea80
explicitly enable create table from source as --all-features is not w…
bobbyiliev Nov 18, 2024
44ae47d
Terraform Docs
bobbyiliev Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ services:
- --system-parameter-default=max_clusters=100
- --system-parameter-default=max_sources=100
- --system-parameter-default=max_aws_privatelink_connections=10
- --system-parameter-default=enable_create_table_from_source=on
- --all-features
environment:
MZ_NO_TELEMETRY: 1
Expand Down Expand Up @@ -48,6 +49,7 @@ services:
- --system-parameter-default=max_sources=100
- --system-parameter-default=max_aws_privatelink_connections=10
- --system-parameter-default=transaction_isolation=serializable
- --system-parameter-default=enable_create_table_from_source=on
- --all-features
environment:
MZ_NO_TELEMETRY: 1
Expand Down
53 changes: 53 additions & 0 deletions docs/data-sources/source_reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
# generated by https://github.com/hashicorp/terraform-plugin-docs
page_title: "materialize_source_reference Data Source - terraform-provider-materialize"
subcategory: ""
description: |-
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we populate this to explain that these are 'available' source references? Such that these expose all the possible upstream references that this source can create a table for, not necessarily all the references it is already ingesting

The materialize_source_reference data source retrieves a list of available upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables.
---

# materialize_source_reference (Data Source)

The `materialize_source_reference` data source retrieves a list of *available* upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables.

## Example Usage

```terraform
data "materialize_source_reference" "source_references" {
source_id = materialize_source_mysql.test.id
}

output "source_references" {
value = data.materialize_source_reference.my_source_references.references
}
```

<!-- schema generated by tfplugindocs -->
## Schema

### Required

- `source_id` (String) The ID of the source to get references for

### Optional

- `region` (String) The region in which the resource is located.

### Read-Only

- `id` (String) The ID of this resource.
- `references` (List of Object) The source references (see [below for nested schema](#nestedatt--references))

<a id="nestedatt--references"></a>
### Nested Schema for `references`

Read-Only:

- `columns` (List of String)
- `name` (String)
- `namespace` (String)
- `source_database_name` (String)
- `source_name` (String)
- `source_schema_name` (String)
- `source_type` (String)
- `updated_at` (String)
65 changes: 65 additions & 0 deletions docs/data-sources/source_table.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
# generated by https://github.com/hashicorp/terraform-plugin-docs
page_title: "materialize_source_table Data Source - terraform-provider-materialize"
subcategory: ""
description: |-

---

# materialize_source_table (Data Source)



## Example Usage

```terraform
data "materialize_source_table" "all" {}

data "materialize_source_table" "materialize" {
database_name = "materialize"
}

data "materialize_source_table" "materialize_schema" {
database_name = "materialize"
schema_name = "schema"
}
```

<!-- schema generated by tfplugindocs -->
## Schema

### Optional

- `database_name` (String) Limit tables to a specific database
- `region` (String) The region in which the resource is located.
- `schema_name` (String) Limit tables to a specific schema within a specific database

### Read-Only

- `id` (String) The ID of this resource.
- `tables` (List of Object) The source tables in the account (see [below for nested schema](#nestedatt--tables))

<a id="nestedatt--tables"></a>
### Nested Schema for `tables`

Read-Only:

- `comment` (String)
- `database_name` (String)
- `id` (String)
- `name` (String)
- `owner_name` (String)
- `schema_name` (String)
- `source` (List of Object) (see [below for nested schema](#nestedobjatt--tables--source))
- `source_type` (String)
- `upstream_name` (String)
- `upstream_schema_name` (String)

<a id="nestedobjatt--tables--source"></a>
### Nested Schema for `tables.source`

Read-Only:

- `database_name` (String)
- `name` (String)
- `schema_name` (String)
13 changes: 13 additions & 0 deletions docs/data-sources/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,20 @@ description: |-



## Example Usage

```terraform
data "materialize_table" "all" {}

data "materialize_table" "materialize" {
database_name = "materialize"
}

data "materialize_table" "materialize_schema" {
database_name = "materialize"
schema_name = "schema"
}
```

<!-- schema generated by tfplugindocs -->
## Schema
Expand Down
Loading