Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'entity level' support for delta updates and targetSchema to UI for materializations #1135

Open
travjenkins opened this issue Jun 4, 2024 · 1 comment
Assignees
Labels
create Creating an entity edit Editing an entity enhancement New feature or request

Comments

@travjenkins
Copy link
Member

travjenkins commented Jun 4, 2024

Current Ask

  1. When a user adds a sourceCapture we will then need to show them a setting to control the deltaUpdates flag. This is setting ON the sourceCapture setting itself is really controlling if fields added has the binding deltaUpdates defaulted.
  2. When adding a manual binding we will need to get the deltaUpdate and targetSchema settings for the bindings.

We will only default new bindings when selecting a Source Capture

deltaUpdates: true, then new bindings that are added automatically by sourceCapture, are added with delta updates set to true.
targetSchema: "fromSource", then new bindings that are added automatically by sourceCapture, are added with their schema set to the 2nd-to-last part of the collection name
    (the alternative value of targetSchema is leaveEmpty which leaves the schema empty for that binding, thus defaulting to the global schema of the materialization task (current behavior)

Previous Ask

We want to start defaulting delta_updates to true for materializations. While doing this work we also want to expose this setting in the form (somewhere) for the user to easily edit.

Docs:
https://docs.estuary.dev/reference/Connectors/materialization-connectors/BigQuery/#delta-updates

@travjenkins travjenkins added the enhancement New feature or request label Jun 4, 2024
@travjenkins travjenkins added edit Editing an entity create Creating an entity labels Aug 14, 2024
@travjenkins travjenkins changed the title Set delta updates as a default option on the materialization level Add support for delta updates and targetSchema to UI for materializations Oct 1, 2024
@travjenkins travjenkins self-assigned this Oct 10, 2024
@travjenkins
Copy link
Member Author

Requires estuary/flow#1760

@travjenkins travjenkins changed the title Add support for delta updates and targetSchema to UI for materializations Add 'entity level' support for delta updates and targetSchema to UI for materializations Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
create Creating an entity edit Editing an entity enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant