Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use New Fideslang DSR-updated models [#1761] #2009

Merged
merged 21 commits into from
Dec 14, 2022
Merged

Use New Fideslang DSR-updated models [#1761] #2009

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5f1327a
Initial pass at converting project over to new fideslang
TheAndrewJackson Nov 22, 2022
1677721
Merge branch 'main' into ajackson_1761_use_new_fideslang_models
pattisdr Dec 2, 2022
0423669
Temporarily point fideslang to the add dsr constructs branch.
pattisdr Dec 8, 2022
ab8a9d5
Update check_fides_key to catch a ValidationError instead.
pattisdr Dec 8, 2022
a604e6a
Remove data_categories at object level. New validation on FidesLang …
pattisdr Dec 8, 2022
8117cd7
Migrate ctl_datasets table to rename fidesctl_meta field to fides_meta.
pattisdr Dec 8, 2022
39dd611
Rename fidesops_meta to fides_meta in datasets only (not saas) - to p…
pattisdr Dec 8, 2022
182b947
Get rid of FidesOpsKey in favor of fideslang FidesKey which has been …
pattisdr Dec 8, 2022
d842448
Rename Dataset to GraphDataset to avoid confusion with Fideslang Data…
pattisdr Dec 8, 2022
a4089ed
Merge branch 'main' into fides_1761_use_new_fideslang_models
pattisdr Dec 8, 2022
b3d85b4
Running static checks after merge.
pattisdr Dec 8, 2022
543ccf9
Update db_dataset to reflect Dataset fidesctl_meta -> fides_meta rename.
pattisdr Dec 8, 2022
099be49
Temporarily install git in docs Dockerfile to install a pinned versio…
pattisdr Dec 8, 2022
7b06e37
Update the other database dataset to have fides_meta instead of fides…
pattisdr Dec 8, 2022
992c1aa
Fix FidesKey error messages in tests.
pattisdr Dec 8, 2022
9655ef5
Adjust test now that dataset has been updated to have fides_meta.
pattisdr Dec 9, 2022
28c2192
Check if return annotation has the same value as List[Row], not wheth…
pattisdr Dec 9, 2022
ea83275
Merge main.
pattisdr Dec 9, 2022
e11b22d
Update changelog.
pattisdr Dec 9, 2022
f51f640
Revert import rename of Fideslang Dataset now that we've renamed the …
pattisdr Dec 9, 2022
c575177
- Adjust some docstrings and variable names related to the Dataset ->…
pattisdr Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .fides/db_dataset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ dataset:
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: fidesctl_meta
- name: fides_meta
seanpreston marked this conversation as resolved.
Show resolved Hide resolved
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
Expand Down
4 changes: 0 additions & 4 deletions .fides/redis_dataset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,11 @@ dataset:
fields:
- name: EN_ACCESS_GRAPH__<privacy_request_id>
description: This graph is summarized and sent to Fideslog to create high level insight into how graphs change between retries to inform future features.
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fidesops_meta:
data_type: object # Stores an encrypted representation of the fidesops graph that executes the privacy requests.
fields:
- name: <dataset_name>:<collection_name> # The current collection
data_categories:
- system.operations
Comment on lines -10 to -18
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dataset file reflects what we store in Redis. Now that we're sharing fideslang models between ops and ctl there's validation that doesn't allow data categories at the object field level - they should be on the nested fields instead.

data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fidesops_meta:
data_type: object
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ The types of changes are:

* Update sample project landing page copy to be version-agnostic [#1958](https://github.com/ethyca/fides/pull/1958)
* `get` and `ls` CLI commands now return valid `fides` object YAML [#1991](https://github.com/ethyca/fides/pull/1991)
* Remove several fidesops schemas for DSR's in favor of updated Fideslang schemas [#2009](https://github.com/ethyca/fides/pull/2009)

### Developer Experience

Expand Down
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ RUN apt-get update && \
g++ \
gnupg \
gcc \
git \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is temporary, will remove when fideslang has been updated.

&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

Expand Down
54 changes: 27 additions & 27 deletions data/dataset/bigquery_example_test_dataset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ dataset:
data_categories: [user.contact.address.street]
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I adjusted the fidesops_meta fields in the database datasets, but not the saas datasets to demonstrate both still work. Fideslang dataset schemas can take in both fides_meta and fidesops_meta but they are converted to fides_meta. We also don't want to break backwards compatibility with existing saas datasets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Should/can the saas templates be converted over to use fides_meta in a future PR down the line in the unified fides resources work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this comment, it sounded like we wanted to preserve the backwards compatibility of this for now https://github.com/ethyca/fideslang/issues/94#issuecomment-1341175140

primary_key: True
- name: state
data_categories: [user.contact.address.state]
Expand All @@ -24,7 +24,7 @@ dataset:
fields:
- name: address_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: address.id
Expand All @@ -33,54 +33,54 @@ dataset:
data_categories: [system.operations]
- name: email
data_categories: [user.contact.email]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
primary_key: True
- name: name
data_categories: [user.name]
fidesops_meta:
fides_meta:
data_type: string
length: 40

- name: employee
fields:
- name: address_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: address.id
direction: to
- name: email
data_categories: [user.contact.email]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
primary_key: True
- name: name
data_categories: [user.name]
fidesops_meta:
fides_meta:
data_type: string

- name: login
fields:
- name: customer_id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: time
data_categories: [user.sensor]
Expand All @@ -89,18 +89,18 @@ dataset:
fields:
- name: customer_id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: shipping_address_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: address.id
Expand All @@ -111,14 +111,14 @@ dataset:
fields:
- name: order_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: orders.id
direction: from
- name: product_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: product.id
Expand All @@ -130,7 +130,7 @@ dataset:
fields:
- name: billing_address_id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: address.id
Expand All @@ -141,14 +141,14 @@ dataset:
data_categories: [user.financial]
- name: customer_id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: name
data_categories: [user.financial]
Expand All @@ -159,7 +159,7 @@ dataset:
fields:
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: name
data_categories: [system.operations]
Expand All @@ -170,12 +170,12 @@ dataset:
fields:
- name: email
data_categories: [user.contact.email]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: month
data_categories: [system.operations]
Expand All @@ -190,26 +190,26 @@ dataset:
fields:
- name: alt_email
data_categories: [user.contact.email]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: closed
data_categories: [system.operations]
- name: email
data_categories: [system.operations]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: employee_id
data_categories: [user.unique_id]
fidesops_meta:
fides_meta:
references:
- dataset: bigquery_example_test_dataset
field: employee.id
direction: from
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: True
- name: opened
data_categories: [system.operations]
Expand All @@ -218,7 +218,7 @@ dataset:
fields:
- name: email
data_categories: [user.contact.email]
fidesops_meta:
fides_meta:
identity: email
data_type: string
- name: last_visit
Expand Down
16 changes: 8 additions & 8 deletions data/dataset/email_dataset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ dataset:
fields:
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: true
- name: customer_id
data_categories: [user]
fidesops_meta:
fides_meta:
references:
- dataset: postgres_example_test_dataset
field: customer.id
Expand All @@ -22,18 +22,18 @@ dataset:
fields:
- name: id
data_categories: [system.operations]
fidesops_meta:
fides_meta:
primary_key: true
- name: first_name
data_categories: [user.childrens]
- name: last_name
data_categories: [user.childrens]
- name: birthday
data_categories: [user.childrens]
fidesops_meta:
fides_meta:
data_type: string
- name: report_card
fidesops_meta:
fides_meta:
data_type: object
fields:
- name: grades
Expand All @@ -45,7 +45,7 @@ dataset:
- name: test_scores
data_categories: [ user.childrens ]
- name: parent_id
fidesops_meta:
fides_meta:
references:
- dataset: email_dataset
field: daycare_customer.id
Expand All @@ -54,11 +54,11 @@ dataset:
fields:
- name: id
data_categories: [ system.operations ]
fidesops_meta:
fides_meta:
primary_key: true
- name: payer_email
data_categories: [ user.contact.email ]
fidesops_meta:
fides_meta:
identity: email
- name: ccn
data_categories: [user.financial.account_number]
Loading