Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HJ-116 - Fix BigQuery partitioning queries to properly support multiple identity clauses #5432

Merged
merged 18 commits into from
Oct 31, 2024
Merged

HJ-116 - Fix BigQuery partitioning queries to properly support multiple identity clauses #5432

merged 18 commits into from
Oct 31, 2024

Conversation

andres-torres-marroquin
Copy link
Contributor

@andres-torres-marroquin andres-torres-marroquin commented Oct 29, 2024 •
edited
Loading

Fix BigQuery partitioning support bug - WHERE queries are not properly generated

Closes #HJ-116

Description Of Changes

Update BigQuery to use parenthesis on the WHERE clause for SELECT queries.

Code Changes

  • Update BigQueryQueryConfig.get_formatted_query_string to surround the WHERE clause contents in parenthesis always.

Steps to Confirm

  • Run the test suite.

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation:
    • documentation complete, PR opened in fidesdocs
    • documentation issue created in fidesdocs
    • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md
  • For API changes, the Postman collection has been updated
  • If there are any database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!

@vercel Vercel
Copy link

vercel bot commented Oct 29, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Oct 31, 2024 7:17pm

@cypress Cypress
Copy link

cypress bot commented Oct 29, 2024 •
edited
Loading

fides    Run #10721

Run Properties:  status check passed Passed #10721  •  git commit 8a1be3eeab ℹ️: Merge 59f86bed0706f1885c8f70ed7dd0adf19191d45a into 008cf6818d94915fbdf3a41138ff...
Project fides
Branch Review refs/pull/5432/merge
Run status status check passed Passed #10721
Run duration 00m 37s
Commit git commit 8a1be3eeab ℹ️: Merge 59f86bed0706f1885c8f70ed7dd0adf19191d45a into 008cf6818d94915fbdf3a41138ff...
Committer Andres Torres
View all properties for this run ↗︎
Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

@codecov Codecov
Copy link

codecov bot commented Oct 29, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.47%. Comparing base (4dd5bff) to head (c50b73d).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/fides/api/service/connectors/query_config.py 40.00% 3 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #5432   +/-   ##
=======================================
  Coverage   85.47%   85.47%           
=======================================
  Files         384      384           
  Lines       24118    24118           
  Branches     2624     2624           
=======================================
  Hits        20614    20614           
  Misses       2950     2950           
  Partials      554      554

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andres-torres-marroquin andres-torres-marroquin marked this pull request as ready for review October 29, 2024 17:28
@andres-torres-marroquin andres-torres-marroquin added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Oct 29, 2024
NevilleS
NevilleS requested changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@NevilleS NevilleS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andres-torres-marroquin I don't see any tests here that also exercise the partitioning clauses. I'd like to be extra confident that we've tested how those get merged into the rest

@andres-torres-marroquin
Copy link
Contributor Author

andres-torres-marroquin commented Oct 29, 2024 •
edited
Loading

@andres-torres-marroquin I don't see any tests here that also exercise the partitioning clauses. I'd like to be extra confident that we've tested how those get merged into the rest

@NevilleS Actually this test uses the partitioning clause, do you want me to add one that uses more than one identifying field? or a more explicit check?

@NevilleS
Copy link
Contributor

@andres-torres-marroquin I don't see any tests here that also exercise the partitioning clauses. I'd like to be extra confident that we've tested how those get merged into the rest

@NevilleS Actually this test uses the partitioning clause, do you want me to add one that uses more than one identifying field? or a more explicit check?

Yes, I'd like to ensure there is an explicit test that shows the query we generate when there is a partitioning clause and multiple identities.

NevilleS
NevilleS approved these changes Oct 31, 2024
View reviewed changes
Copy link
Contributor

@NevilleS NevilleS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. The test feels awkward but that's non-blocking- maybe you can simplify that assertion...?

tests/ops/service/connectors/test_bigquery_connector.py Outdated Show resolved Hide resolved
tests/ops/service/connectors/test_bigquery_connector.py Outdated Show resolved Hide resolved
tests/fixtures/bigquery_fixtures.py Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
@andres-torres-marroquin andres-torres-marroquin changed the title HJ-116 - Fix BigQuery partitioning support bug - WHERE queries are not properly generated HJ-116 - Fix BigQuery partitioning queries to properly support multiple identity clauses Oct 31, 2024
@andres-torres-marroquin andres-torres-marroquin merged commit b0f976a into main Oct 31, 2024
34 of 40 checks passed
@andres-torres-marroquin andres-torres-marroquin deleted the andres/HJ-116 branch October 31, 2024 19:52
andres-torres-marroquin added a commit that referenced this pull request Oct 31, 2024
@andres-torres-marroquin @NevilleS
HJ-116 - Fix BigQuery partitioning queries to properly support mult…
34e00b1 
…iple identity clauses (#5432)

Co-authored-by: Neville Samuell <[email protected]>
@cypress Cypress
Copy link

cypress bot commented Oct 31, 2024

fides    Run #10722

Run Properties:  status check passed Passed #10722  •  git commit b0f976a11b: HJ-116 - Fix BigQuery `partitioning` queries to properly support multiple identi...
Project fides
Branch Review main
Run status status check passed Passed #10722
Run duration 00m 36s
Commit git commit b0f976a11b: HJ-116 - Fix BigQuery `partitioning` queries to properly support multiple identi...
Committer Andres Torres
View all properties for this run ↗︎
Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

@andres-torres-marroquin andres-torres-marroquin mentioned this pull request Oct 31, 2024
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thingscouldbeworse thingscouldbeworse thingscouldbeworse left review comments

@tvandort tvandort tvandort approved these changes

@NevilleS NevilleS NevilleS approved these changes

Assignees
No one assigned
Labels
run unsafe ci checks Runs fides-related CI checks that require sensitive credentials
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@andres-torres-marroquin @NevilleS @thingscouldbeworse @tvandort