Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Grant SQL Macros #5369

Merged
merged 37 commits into from
Jul 11, 2022
Merged

Add Grant SQL Macros #5369

merged 37 commits into from
Jul 11, 2022

Conversation

McKnight-42
Copy link
Contributor

@McKnight-42 McKnight-42 commented Jun 13, 2022

Resolves #5263
Resolves #5329

Description:
Allow users to define grants as a reasonable default in dbt_project.yml and within each model sql or yml file as a config.

Checklist

@cla-bot cla-bot bot added the cla:yes label Jun 13, 2022
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

@McKnight-42 McKnight-42 changed the title init push or ct-660 work Add Grant SQL Macros Jun 13, 2022
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start on this @McKnight-42! Thanks for opening the draft PR so early, so that we can keep pushing forward on this.

In my comments, I tried to provide relevant information that will help us decide if we're implementing the right standard, by initializing this default behavior in dbt-core. Much of the database-specific nuance will come out in the per-adapter implementations, but we always need to first make sure that we're putting the right core constructs in place.

@McKnight-42 McKnight-42 self-assigned this Jun 14, 2022
@McKnight-42 McKnight-42 requested a review from jtcohen6 June 14, 2022 21:19
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice progress!

…t_config and curreent_grants to see what needs to be revoked, I know if we assign a role, and a model becomes dependent on it we can't drop the role now still not seeing the diff appear in log
@McKnight-42
Copy link
Contributor Author

McKnight-42 commented Jun 22, 2022

NOTE: create postgres version of get_show_grant sql https://linuxhint.com/check-postgres-user-privileges/

… errors hopefully will clear up as we add the other postgres versions of macros and isn't a psycopg2 issue as indicated by searching
@McKnight-42 McKnight-42 requested a review from jtcohen6 June 27, 2022 22:10
{%- macro postgres__get_show_grant_sql(relation) -%}
select grantee, privilege_type
from information_schema.role_table_grants
where grantor = current_role
Copy link
Contributor

@jtcohen6 jtcohen6 Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a TODO, just an open question: Should we / do we need to limit ourselves to just the grants provided by the current_role?

To copy-paste what I just said in dbt-snowflake:

The original motivation for this was: If someone else has granted a privilege on this model, and then dbt tries to revoke it, the revocation may fail. But if dbt's user/role owns the table, it should be able to revoke any access on it... right?

By removing this logic, we'd be making a strong claim, that ALL the grants applied on this object, MUST be applied by dbt, in the grants config. I think that might be a strong opinion we want to hold!

No action needed right now. Just wanted to leave the comment for future reference.

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jul 7, 2022

@McKnight-42 I've gone back through the threads above, and I think all the blocking TODOs have been resolved! Nice work!

The tests are still failing, and the fixes for those tests are in #5447. Let's either:

  • pull the fixes into this branch, so we can merge this PR into main
  • wait to merge CT-808 grant adapter tests #5447 (into this branch), before merging anything into main

McKnight-42 and others added 2 commits July 7, 2022 14:57
Comment on lines 670 to 681
dict_diff = {}
dict_a = {k.lower(): v for k, v in dict_a.items()}
dict_b = {k.lower(): v for k, v in dict_b.items()}
for k in dict_a:
if k in dict_b:
a_lowered = map(lambda x: x.lower(), dict_a[k])
b_lowered = map(lambda x: x.lower(), dict_b[k])
diff = list(set(a_lowered) - set(b_lowered))
if diff:
dict_diff.update({k: diff})
else:
dict_diff.update({k: dict_a[k]})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have we tested the performance on this when the grants dict config is huge (and very different from current grants)? might not suck once, but might if they have this same complicated diff on hundreds of models?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code can + should absolutely be refactored for more Pythonic comprehension + better performance / lower memory footprint. I'm not the right person to do that :)

hundreds of models?

FWIW this code will be executed in parallel --threads, assuming the user is making use of them

@@ -20,6 +20,8 @@
-- BEGIN, in a separate transaction
{%- set preexisting_intermediate_relation = load_cached_relation(intermediate_relation)-%}
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not something needed to do before merging, but we added the same change pretty much for all materialization methods. Should we consider refactoring the code so we can only do it in one place?

McKnight-42 and others added 3 commits July 11, 2022 09:02
* Add tests for invalid user and invalid privilege

* Add more grant tests

* Macro touch ups

* Many more tests

* Allow easily replacing privilege names

* Keep adding tests

* Refactor macros to return lists, fix test

* Code checks

* Keep tweaking tests

* Revert cool grantees join bc Snowflake isnt happy

* Use Postgres/BQ as standard for standardize_grants_dict

* Code checks

* add missing replace

* small replace tweak,  add additional dict diffs

* All tests passing on BQ

* Add type cast to test_snapshot_grants

* Refactor for DRYer apply_grants macros

Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Emily Rockman <[email protected]>
Copy link
Contributor

@dbeatty10 dbeatty10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
Looks good to me. Only thing we should do first is apply the white space fixes.

My opinion is to ship this now so that we can unblock merging for everything else downstream.

Separately, we can break out any optimizations or refactoring separately to incorporate @nathaniel-may and @ChenyuLInx's ideas.

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jul 11, 2022

What Doug said! The recommended changes are good ones, and we can pick them up in subsequent refactors. Let's unblock the adapter PRs in the meantime.

@McKnight-42 McKnight-42 merged commit c25260e into main Jul 11, 2022
@McKnight-42 McKnight-42 deleted the ct-660-grant-sql branch July 11, 2022 16:58
agoblet pushed a commit to BigDataRepublic/dbt-core that referenced this pull request Sep 16, 2022
* init push or ct-660 work

* changes to default versions of get_show_grant_sql and get_grant_sql

* completing init default versions of all macros being called for look over and collaboration

* minor update to should_revoke

* post pairing push up (does have log statements to make sure we remove)

* minor spacing changes

* minor changes, and removal of logs so people can have clean grab of code

* minor changes to how get_revoke_sql works

* init attempt at applying apply_grants to all materialzations

* name change from recipents -> grantee

* minor changes

* working on making a context to handle the diff gathering between grant_config and curreent_grants to see what needs to be revoked, I know if we assign a role, and a model becomes dependent on it we can't drop the role now still not seeing the diff appear in log

* removing logs from most materializations to better track diff of grants generation logs

* starting to build out postgres get_show_grant_sql getting empty query errors hopefully will clear up as we add the other postgres versions of macros and isn't a psycopg2 issue as indicated by searching

* 6/27 eod update looking into diff_grants variable not getting passed into get_revoke_sql

* changes to loop cases

* changes after pairing meeting

* adding apply_grants to create_or_replace_view.sql

* models are building but testing out small issues around revoke statement never building

* postgrest must fixes from jeremy's feedback

* postgres minor change to standarize_grants_dict

* updating after pairing with dough and jeremey incorporating the new version of should revoke logic.

* adding  ref of diff_of_two_dicts to base keys ref

* change of method type for standardize_grants_dict

* minor update trying to fix unit test

* changes based on morning feedback

* change log message in default_apply_grants macro

* CT-808 grant adapter tests (dbt-labs#5447)

* Create initial test for grants

Co-authored-by: Jeremy Cohen <[email protected]>

* rename grant[privilege] -> grant_config[privilege]

* postgres macro rename to copy_grants

* CT-808 more grant adapter tests (dbt-labs#5452)

* Add tests for invalid user and invalid privilege

* Add more grant tests

* Macro touch ups

* Many more tests

* Allow easily replacing privilege names

* Keep adding tests

* Refactor macros to return lists, fix test

* Code checks

* Keep tweaking tests

* Revert cool grantees join bc Snowflake isnt happy

* Use Postgres/BQ as standard for standardize_grants_dict

* Code checks

* add missing replace

* small replace tweak,  add additional dict diffs

* All tests passing on BQ

* Add type cast to test_snapshot_grants

* Refactor for DRYer apply_grants macros

Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Emily Rockman <[email protected]>

* update to main, create changelog, whitespace fixes

Co-authored-by: Gerda Shank <[email protected]>
Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Emily Rockman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-719] Add Grants to Postgres Materializations [CT-660] [Feature] Add Grant SQL to Global Project
7 participants