Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BigQuery configuration with parent-project-id #23041

Merged
merged 3 commits into from
Aug 29, 2024

Conversation

pajaks
Copy link
Member

@pajaks pajaks commented Aug 14, 2024

Description

If bigquery.parent-project-id was specified in configuration it was set as projectId in BigQueryOptions.
For queries without TableId or DatasetId specified BigQueryOptions was used resulting in using wrong projectId.

This problem is visible when using PTF with parent-proejct-id set, resulting in permission problem like:

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
POST https://bigquery.googleapis.com/bigquery/v2/projects/parent-project-id/jobs?prettyPrint=false
{
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "message": "Access Denied: Table parent-project-id:tpch.nation: User does not have permission to query table parent-project-id:tpch.nation, or perhaps it does not exist.",
      "reason": "accessDenied"
    }
  ],
  "message": "Access Denied: Table parent-project-id:tpch.nation: User does not have permission to query table parent-project-id:tpch.nation, or perhaps it does not exist.",
  "status": "PERMISSION_DENIED"
}

Billing project id should be set using `setQuotaProjectId' according to https://cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.ServiceOptions#com_google_cloud_ServiceOptions_getQuotaProjectId__

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# BigQuery
* Fix failures with PTF queries when parent-project-id is defined. ({issue}`23041`)

@cla-bot cla-bot bot added the cla-signed label Aug 14, 2024
@pajaks pajaks changed the title Fix BigQuery configuration with parentProjectId Fix BigQuery configuration with parent-project-id Aug 14, 2024
@github-actions github-actions bot added the bigquery BigQuery connector label Aug 14, 2024
@pajaks pajaks force-pushed the pajaks/bigquery_parent_fix branch from 6240fb8 to fcdb2cc Compare August 16, 2024 10:28
@pajaks pajaks force-pushed the pajaks/bigquery_parent_fix branch from fcdb2cc to 1698700 Compare August 19, 2024 08:08
@findinpath
Copy link
Contributor

# BigQuery
* Fix configruation with parent-project-id. ({issue}`23041`)

nit: typo in configruation

@pajaks pajaks force-pushed the pajaks/bigquery_parent_fix branch from 1698700 to 327d4ee Compare August 26, 2024 08:11
@pajaks pajaks requested review from SemionPar and ebyhr August 26, 2024 08:12
@pajaks pajaks marked this pull request as ready for review August 26, 2024 08:12
Copy link
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash 4th and 3rd commits into one.

@ebyhr
Copy link
Member

ebyhr commented Aug 26, 2024

/test-with-secrets sha=56ec6f35fda5235416262aab2120a25bdaf304ca

Copy link

github-actions bot commented Aug 26, 2024

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/10567518374

@pajaks
Copy link
Member Author

pajaks commented Aug 28, 2024

Big Query tests with secretes passed in mirror PR #23141
Other tests failures not related to changes

Copy link
Member

@hashhar hashhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be very helpful if we can have a short comment somewhere explaining the differences between project id, parent project id and how those are related to the quota/billing project.

Code itself seems fine but the semantics are unclear to me.

@pajaks pajaks force-pushed the pajaks/bigquery_parent_fix branch from b1c326c to 0c6611b Compare August 29, 2024 09:34
@pajaks
Copy link
Member Author

pajaks commented Aug 29, 2024

It would be very helpful if we can have a short comment somewhere explaining the differences between project id, parent project id and how those are related to the quota/billing project.

Quota project id is different name for parent project id, both indicates project used for quota and billing purposes.
Added comment in code.

@pajaks pajaks requested a review from hashhar August 29, 2024 09:35
@pajaks pajaks force-pushed the pajaks/bigquery_parent_fix branch from 0c6611b to 3f41661 Compare August 29, 2024 10:43
@ebyhr ebyhr force-pushed the pajaks/bigquery_parent_fix branch from 3f41661 to a076b11 Compare August 29, 2024 22:14
@ebyhr ebyhr merged commit 74cce25 into trinodb:master Aug 29, 2024
4 of 14 checks passed
@github-actions github-actions bot added this to the 456 milestone Aug 29, 2024
@ebyhr
Copy link
Member

ebyhr commented Aug 29, 2024

Fix configuration with parent-project-id

@pajaks Thanks for writing the release note entry. I think users can't understand what was the issue from this sentence. Could you update it? We usually use "Fix failure when ..." for bug fix entries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery BigQuery connector cla-signed
Development

Successfully merging this pull request may close these issues.

5 participants