-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Source versioning: Postgres, MySQL and Load generator #647
Open
bobbyiliev
wants to merge
45
commits into
main
Choose a base branch
from
source-versioning
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
ce42144
Source versioning initial implementation
bobbyiliev decb60f
Use source table instead of table from source
bobbyiliev e4b5883
MySQL source: separate for tables and all tables
bobbyiliev d8612c1
Loadgen source: add all tables bool attr
bobbyiliev cb3b49a
Add tests
bobbyiliev 718bf71
Add more tests for mysql and loadgen
bobbyiliev eb36ba3
Add ignore columns for MySQL
bobbyiliev bf5196b
Add source_id logic
bobbyiliev 6864079
Add source table migration guide
bobbyiliev 99d2f62
Add deprecated message
bobbyiliev 74593f3
Ignore text columns for load gen source tables
bobbyiliev f8fc659
Refactor source table for individual sources
bobbyiliev a4d0eba
Add datasource
bobbyiliev 191649c
Format examples
bobbyiliev dd1f993
Add Kafka source table resource
bobbyiliev 6ab37e5
Review updates
bobbyiliev 82ee6b8
Update guide migration guide
bobbyiliev bb48f92
Update guide migration guide
bobbyiliev ec8d61a
Add import examples for Kafka source tables
bobbyiliev a9017a0
Add upstream mysql and postgres table names
bobbyiliev 640996f
Fix unit tests
bobbyiliev c34e8ac
Add Kafka upstream references
bobbyiliev bb2f61c
Add integration tests
bobbyiliev 6330f06
Fix failing test
bobbyiliev 6ecc0b4
Extend data source to include upstream names
bobbyiliev bc2298d
Small updates
bobbyiliev 12f00aa
Switch back to latest image
bobbyiliev 0358960
FromAsCasing: 'as' and 'FROM' keywords' casing do not match
bobbyiliev a624bfc
Add source reference data source
bobbyiliev 2e2ec9a
Add source reference data source example
bobbyiliev 89ebfb3
First round of the initial PR change requests
bobbyiliev 29d8b48
Fix failing tests
bobbyiliev e8e322f
Fix failing tests
bobbyiliev cab7a27
Second round of the initial PR change requests
bobbyiliev 3312ebd
Add unit tests to data source source reference
bobbyiliev 4c3d251
PR change requests
bobbyiliev 620e020
Fix failing tests
bobbyiliev abd069c
Fix failing tests
bobbyiliev 5d442dd
Add alter source refresh to data source
bobbyiliev e57d5f0
Add new kafk acolumns from mz_kafka_source_tables
bobbyiliev 7d57eb5
Fix failing MockSourceTableKafkaScan test
bobbyiliev d3c6ed2
Remove confusing line from migration guide
bobbyiliev cd69c11
Remove a left behind comment
bobbyiliev e3bea80
explicitly enable create table from source as --all-features is not w…
bobbyiliev 44ae47d
Terraform Docs
bobbyiliev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
--- | ||
# generated by https://github.com/hashicorp/terraform-plugin-docs | ||
page_title: "materialize_source_reference Data Source - terraform-provider-materialize" | ||
subcategory: "" | ||
description: |- | ||
The materialize_source_reference data source retrieves a list of available upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables. | ||
--- | ||
|
||
# materialize_source_reference (Data Source) | ||
|
||
The `materialize_source_reference` data source retrieves a list of *available* upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables. | ||
|
||
## Example Usage | ||
|
||
```terraform | ||
data "materialize_source_reference" "source_references" { | ||
source_id = materialize_source_mysql.test.id | ||
} | ||
|
||
output "source_references" { | ||
value = data.materialize_source_reference.my_source_references.references | ||
} | ||
``` | ||
|
||
<!-- schema generated by tfplugindocs --> | ||
## Schema | ||
|
||
### Required | ||
|
||
- `source_id` (String) The ID of the source to get references for | ||
|
||
### Optional | ||
|
||
- `region` (String) The region in which the resource is located. | ||
|
||
### Read-Only | ||
|
||
- `id` (String) The ID of this resource. | ||
- `references` (List of Object) The source references (see [below for nested schema](#nestedatt--references)) | ||
|
||
<a id="nestedatt--references"></a> | ||
### Nested Schema for `references` | ||
|
||
Read-Only: | ||
|
||
- `columns` (List of String) | ||
- `name` (String) | ||
- `namespace` (String) | ||
- `source_database_name` (String) | ||
- `source_name` (String) | ||
- `source_schema_name` (String) | ||
- `source_type` (String) | ||
- `updated_at` (String) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
# generated by https://github.com/hashicorp/terraform-plugin-docs | ||
page_title: "materialize_source_table Data Source - terraform-provider-materialize" | ||
subcategory: "" | ||
description: |- | ||
|
||
--- | ||
|
||
# materialize_source_table (Data Source) | ||
|
||
|
||
|
||
## Example Usage | ||
|
||
```terraform | ||
data "materialize_source_table" "all" {} | ||
|
||
data "materialize_source_table" "materialize" { | ||
database_name = "materialize" | ||
} | ||
|
||
data "materialize_source_table" "materialize_schema" { | ||
database_name = "materialize" | ||
schema_name = "schema" | ||
} | ||
``` | ||
|
||
<!-- schema generated by tfplugindocs --> | ||
## Schema | ||
|
||
### Optional | ||
|
||
- `database_name` (String) Limit tables to a specific database | ||
- `region` (String) The region in which the resource is located. | ||
- `schema_name` (String) Limit tables to a specific schema within a specific database | ||
|
||
### Read-Only | ||
|
||
- `id` (String) The ID of this resource. | ||
- `tables` (List of Object) The source tables in the account (see [below for nested schema](#nestedatt--tables)) | ||
|
||
<a id="nestedatt--tables"></a> | ||
### Nested Schema for `tables` | ||
|
||
Read-Only: | ||
|
||
- `comment` (String) | ||
- `database_name` (String) | ||
- `id` (String) | ||
- `name` (String) | ||
- `owner_name` (String) | ||
- `schema_name` (String) | ||
- `source` (List of Object) (see [below for nested schema](#nestedobjatt--tables--source)) | ||
- `source_type` (String) | ||
- `upstream_name` (String) | ||
- `upstream_schema_name` (String) | ||
|
||
<a id="nestedobjatt--tables--source"></a> | ||
### Nested Schema for `tables.source` | ||
|
||
Read-Only: | ||
|
||
- `database_name` (String) | ||
- `name` (String) | ||
- `schema_name` (String) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we populate this to explain that these are 'available' source references? Such that these expose all the possible upstream references that this source can create a table for, not necessarily all the references it is already ingesting