Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a configuration option kms_key_name to dbt_project.yml for BigQ… #1851

Merged
merged 6 commits into from
Nov 8, 2019
Merged

Add a configuration option kms_key_name to dbt_project.yml for BigQ… #1851

merged 6 commits into from
Nov 8, 2019

Conversation

kconvey
Copy link
Contributor

@kconvey kconvey commented Oct 22, 2019 •
edited by drewbanin
Loading

…uery projects where a Cloud KMS key can be specified and used to encrypt models. The key is specified to bigquery in the ddl OPTIONS.

ex.

models:
  fishtown_analytics:
  kms_key_name: 'projects/PROJECT_ID/locations/global/keyRings/test/cryptoKeys/quickstart'
    
    my_model:
      kms_key_name: none   # disables kms encryption

Fixes #1829

@kconvey
Add a configuration option kms_key_name to dbt_project.yml for BigQ…
bcd9534 
…uery

projects where a Cloud KMS key can be specified and used to encrypt
models. The key is specified to bigquery in the ddl OPTIONS.
@cla-bot
Copy link

cla-bot bot commented Oct 22, 2019

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin.

CLA has not been signed by users: @kconvey

@kconvey
Copy link
Contributor Author

kconvey commented Oct 22, 2019

CLA should be signed now :)

@drewbanin drewbanin self-requested a review October 23, 2019 15:37
drewbanin
drewbanin requested changes Oct 23, 2019
View reviewed changes
Copy link
Contributor

@drewbanin drewbanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor change requested around some recently-updated code, but the core logic implemented here looks very good to me! :D

plugins/bigquery/dbt/include/bigquery/macros/adapters.sql
@@ -36,24 +36,25 @@
{% if temporary %}
{% do opts.update({'expiration_timestamp': 'TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)'}) %}
{% endif %}
{% if kms_key_name %}
{% do opts.update({'kms_key_name': "'" ~ kms_key_name ~ "'"}) %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a lot we can do to make these settings less onerous to manage, especially if we intend to add more of them in the future. For now though, this is a-ok with me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't say they're too bad to mess with at present, but maybe a pointer to how one might go about it would help.

plugins/bigquery/dbt/include/bigquery/macros/adapters.sql Outdated Show resolved Hide resolved
@drewbanin
Copy link
Contributor

drewbanin commented Oct 23, 2019 •
edited
Loading

Thanks for opening this PR @kconvey! I think the CLA should be in good shape now (@cla-bot check).

Have a look at my comment above and let me know if you have any questions :)

Also: please ignore the failing integration tests. I'll kick those off once the changes listed above are addressed.

@cla-bot cla-bot bot added the cla:yes label Oct 23, 2019
@cla-bot
Copy link

cla-bot bot commented Oct 23, 2019

The cla-bot has been summoned, and re-checked this pull request!

kconvey
kconvey commented Oct 24, 2019
View reviewed changes
Copy link
Contributor Author

@kconvey kconvey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hoping this looks better now. Thanks for reviewing @drewbanin !

@kconvey kconvey requested a review from drewbanin November 5, 2019 18:18
drewbanin
drewbanin approved these changes Nov 6, 2019
View reviewed changes
Copy link
Contributor

@drewbanin drewbanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me now, thanks @kconvey!

I just kicked off the test suite - I'll merge this once the tests pass :)

@kconvey kconvey mentioned this pull request Nov 6, 2019
Closed
@drewbanin
Copy link
Contributor

@kconvey looks like the tests here are failing because of an issue with the snowflake-connector-python library. Can you try pulling the latest dev/louisa-may-alcott into this branch and pushing up your changes?

@beckjake
Copy link
Contributor

beckjake commented Nov 8, 2019

This looks great! Thanks for your contribution @kconvey - I'll merge this now.

@beckjake beckjake merged commit 58b2955 into dbt-labs:dev/louisa-may-alcott Nov 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drewbanin drewbanin drewbanin approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support KMS encryption for BigQuery
3 participants
@kconvey @drewbanin @beckjake