Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new columns added when running upsert causes error #103

Open
loveeklund-osttra opened this issue Oct 4, 2024 · 0 comments
Open

new columns added when running upsert causes error #103

loveeklund-osttra opened this issue Oct 4, 2024 · 0 comments

Comments

@loveeklund-osttra
Copy link
Contributor

if a new column is added to a stream that has upsert, it will cause an error because the merge statement will do something along the lines of target.<new_column> = source.<new_column> and fail on <new_column> not found inside target at [1:416].

I think this is because the update_schema function https://github.com/z3z1ma/target-bigquery/blob/9d1d0b08606a716a5a36f53b3388cbd6055535a8/target_bigquery/core.py#L587C1-L601C14 only does a resolve and then updates the schema if it is different.

I created a PR here that could solve it by getting the actual target tables schema and comparing it in update_schema.
See PR here #102

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant