Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use bigquery_client for gcs_stage methods #70

Merged
merged 1 commit into from
Oct 27, 2023

Conversation

tomasvotava
Copy link
Contributor

When using gcs_stage method, self.client attribute of BaseBigQuerySink is actually a storage.Client instance, not bigquery.Client instance. This makes the clean_up fail if overwrite or upsert is used, because storage.Client has no query
method.

The solution here was to create a bigquery.Client using the existing factory when needed.

The downside of this is that the bq client may actually be needed elsewhere, also it could be reused but instead
we create a new instance for each table.

I'll let you consider whether to pull or close and consider this a hotfix.

Closes #69, IMO.

@tomasvotava
Copy link
Contributor Author

I believe the checks are failing because of secrets access. If that's not the case, let me know and I'll fix things.

@z3z1ma
Copy link
Owner

z3z1ma commented Oct 27, 2023

makes sense, lgtm!

@z3z1ma z3z1ma merged commit d9904b8 into z3z1ma:main Oct 27, 2023
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upsert does not work with gcs_stage method
2 participants