fides: add `System` model support for new tcf fields #4228

adamsachs · 2023-10-05T18:01:19Z

Closes #4227

Description Of Changes

Adds support in fides for the new fields on the System and PrivacyDeclaration model added to fideslang.

System.cookie_max_age_seconds
System.cookie_refresh
System.uses_cookies
System.uses_non_cookie_access
System.legitimate_interest_disclosure_url
PrivacyDeclaration.flexible_legal_basis_for_processing

Code Changes

update System db model
DB migration to update ctl_systems table
update system API tests to cover new fields
update TCF overlay generation to include and populate these fields in the vendor records
~~(maybe) update admin UI forms to expose these new fields?~~ split into admin UI: support new TCF related fields on System model #4248
~~(maybe) update TCF overlay UI to leverage these new fields?~~ split into TCF overlay FE: support new TCF related fields on System model #4255

Steps to Confirm

additional fields can't be set yet in the datamap UI - but i ensured they could be set manually via API using the following payload with a PUT /system

{"fides_key":"ts1","organization_fides_key":"default_organization","tags":[],"name":"ts1","description":"","registry_id":null,"meta":null,"fidesctl_meta":null,"system_type":"","data_responsibility_title":null,"egress":null,"ingress":null,"privacy_declarations":[{"name":"pd1", "flexible_legal_basis_for_profiling": true, "data_categories":["system.operations"],"data_use":"analytics.reporting.ad_performance","data_subjects":[],"features":[],"impact_assessment_location":"","retention_period":"","processes_special_category_data":false,"data_shared_with_third_parties":false,"customFieldValues":{},"cookies":[],"id":""}],"joint_controller":null,"third_country_transfers":null,"administrating_department":"","data_protection_impact_assessment":null,"vendor_id":null,"dataset_references":[],"processes_personal_data":true,"exempt_from_privacy_regulations":false,"reason_for_exemption":null,"uses_profiling":false,"legal_basis_for_profiling":[],"does_international_transfers":false,"legal_basis_for_transfers":[],"requires_data_protection_assessments":false,"dpa_location":null,"dpa_progress":null,"privacy_policy":null,"legal_name":"","legal_address":"","responsibility":[],"dpo":"","joint_controller_info":"","data_security_practices":"","cookie_max_age_seconds":355,"uses_cookies":true,"cookie_refresh":true,"uses_non_cookie_access":true,"legitimate_interest_disclosure_url":"http://testurl.com","created_at":"2023-10-11T16:15:54.925759+00:00","connection_configs":null,"data_stewards":[],"cookies":[]}

confirm that a sample TCF overlay has these fields populated

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

cypress · 2023-10-05T20:37:39Z

Passing run #4580 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `18144fa` into `7783dd5`...
Project: fides	Commit: `2cee94b174 ℹ️`
Status: Passed	Duration: 00:52 💡
Started: Oct 11, 2023 10:07 PM	Ended: Oct 11, 2023 10:08 PM

Review all test suite changes for PR #4228 ↗︎

src/fides/api/alembic/migrations/versions/81886da90395_add_legal_basis_dimension.py

codecov · 2023-10-11T17:11:24Z

Codecov Report

All modified lines are covered by tests ✅

❗ No coverage uploaded for pull request base (integration/compass-1.0.0@7783dd5). Click here to learn what that means.

Additional details and impacted files

@@                     Coverage Diff                      @@
##             integration/compass-1.0.0    #4228   +/-   ##
============================================================
  Coverage                             ?   87.74%           
============================================================
  Files                                ?      331           
  Lines                                ?    20895           
  Branches                             ?     2710           
============================================================
  Hits                                 ?    18334           
  Misses                               ?     2094           
  Partials                             ?      467

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

adamsachs · 2023-10-11T19:13:31Z

i've got a changelog commit queued up locally, just didn't want to push to re-trigger CI - writing this so i don't forget to push it up!

pattisdr · 2023-10-11T19:16:20Z

starting review -

pattisdr

Looks great @adamsachs. I know we're not ready to bump a fideslang requirement here, but did you test locally with a fideslang alpha release here to make sure this will work as expected?

src/fides/api/alembic/migrations/versions/81886da90395_add_legal_basis_dimension.py

tests/ctl/core/test_api.py

pattisdr · 2023-10-11T19:41:17Z

tests/ops/util/test_tcf_experience_contents.py

+            EmbeddedVendor(id="sendgrid", name="TCF System Test")
+        ]
+
+        assert tcf_contents.tcf_vendor_consents[0].id == "sendgrid"


This is something that will be affected with my PR: #4256, I stop making the vendor_id: "sendgrid". Just something to sort out on merge, for whoever merges second 😄

src/fides/api/util/tcf/tcf_experience_contents.py

adamsachs · 2023-10-11T20:08:39Z

Looks great @adamsachs. I know we're not ready to bump a fideslang requirement here, but did you test locally with a fideslang alpha release here to make sure this will work as expected?

yup! the base branch is pointed at a fideslang alpha tag, so this branch is too, it's just not showing up in the diff 👍

pattisdr · 2023-10-11T20:22:37Z

yup! the base branch is pointed at a fideslang alpha tag, so this branch is too, it's just not showing up in the diff 👍

Ah! 😅 missed that thank you

this also ensures that VendorRelationship records are created for ANY vendor record in the overlay

… declarations query instead of using the vendor_id to conditionally filter. - define the vendor_relationship_record early so we can assign its basic attributes in one place.

src/fides/api/util/tcf/tcf_experience_contents.py

pattisdr · 2023-10-11T22:03:43Z

OK I think this is ready when CI passes - I'm waiting on whether separate #4256 should be merged to main since these sets of changes need to be reconciled. Wanted to double check since I've missed some context around the integration branch the last couple of days.

src/fides/api/util/tcf/tcf_experience_contents.py

adamsachs · 2023-10-12T01:22:51Z

OK I think this is ready when CI passes - I'm waiting on whether separate #4256 should be merged to main since these sets of changes need to be reconciled. Wanted to double check since I've missed some context around the integration branch the last couple of days.

great, i'll go ahead and merge this one first since i think it's got more dependencies on it 🤷

we need to make both work, and i don't think the conflicts will be different no matter which we merge first...

adamsachs force-pushed the asachs/4227-addtl-tcf-fields-system branch 2 times, most recently from 433d4b7 to 2f2633c Compare October 5, 2023 20:23

adamsachs force-pushed the asachs/4227-addtl-tcf-fields-system branch from 2f2633c to 36ea973 Compare October 6, 2023 11:24

add System model support for new tcf fields

ce5c491

adamsachs force-pushed the asachs/4227-addtl-tcf-fields-system branch from 36ea973 to ce5c491 Compare October 11, 2023 16:22

adamsachs changed the base branch from main to integration/compass-1.0.0 October 11, 2023 16:30

adamsachs commented Oct 11, 2023

View reviewed changes

src/fides/api/alembic/migrations/versions/81886da90395_add_legal_basis_dimension.py Show resolved Hide resolved

adamsachs requested a review from pattisdr October 11, 2023 16:44

adamsachs mentioned this pull request Oct 11, 2023

TCF overlay FE: support new TCF related fields on System model #4255

Closed

support new system fields in tcf overlay generation

8f710e9

adamsachs marked this pull request as ready for review October 11, 2023 18:33

update changelog

4673b22

pattisdr approved these changes Oct 11, 2023

View reviewed changes

fix test syntax

bc944c2

adamsachs and others added 2 commits October 11, 2023 17:09

include basic TCF attributes on VendorRelationships

a1c9b75

this also ensures that VendorRelationship records are created for ANY vendor record in the overlay

Add every vendor to VendorRelationships if it turns up in the privacy…

18144fa

… declarations query instead of using the vendor_id to conditionally filter. - define the vendor_relationship_record early so we can assign its basic attributes in one place.

pattisdr reviewed Oct 11, 2023

View reviewed changes

src/fides/api/util/tcf/tcf_experience_contents.py Outdated Show resolved Hide resolved

adamsachs commented Oct 12, 2023

View reviewed changes

src/fides/api/util/tcf/tcf_experience_contents.py Show resolved Hide resolved

adamsachs merged commit 04fc4f8 into integration/compass-1.0.0 Oct 12, 2023
36 checks passed

adamsachs deleted the asachs/4227-addtl-tcf-fields-system branch October 12, 2023 01:23

This was referenced Oct 12, 2023

Backend: Assume Universal Vendor Ids #4256

Merged

Integration/compass 1.0.0 #4261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fides: add `System` model support for new tcf fields #4228

fides: add `System` model support for new tcf fields #4228

adamsachs commented Oct 5, 2023 •

edited

Loading

cypress bot commented Oct 5, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading

adamsachs commented Oct 11, 2023

pattisdr commented Oct 11, 2023

pattisdr left a comment

pattisdr Oct 11, 2023

adamsachs commented Oct 11, 2023

pattisdr commented Oct 11, 2023

pattisdr commented Oct 11, 2023

adamsachs commented Oct 12, 2023

fides: add System model support for new tcf fields #4228

fides: add System model support for new tcf fields #4228

Conversation

adamsachs commented Oct 5, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Oct 5, 2023 • edited Loading

Passing run #4580 ↗︎

Review all test suite changes for PR #4228 ↗︎

codecov bot commented Oct 11, 2023 • edited Loading

Codecov Report

adamsachs commented Oct 11, 2023

pattisdr commented Oct 11, 2023

pattisdr left a comment

Choose a reason for hiding this comment

pattisdr Oct 11, 2023

Choose a reason for hiding this comment

adamsachs commented Oct 11, 2023

pattisdr commented Oct 11, 2023

pattisdr commented Oct 11, 2023

adamsachs commented Oct 12, 2023

fides: add `System` model support for new tcf fields #4228

fides: add `System` model support for new tcf fields #4228

adamsachs commented Oct 5, 2023 •

edited

Loading

cypress bot commented Oct 5, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading