TCF Backend #3804

pattisdr · 2023-07-18T00:13:56Z

Closes #3817
Closes #3822
Closes #3876
Closes #3935
Closes https://github.com/ethyca/fidesplus/issues/1051

❗ Contains migration; bump downrev before merge

Description Of Changes

Adds the backend for being able to build a TCF overlay.

Importantly:

Adds a new type of experience: the TCF Overlay, that is only returned when the customer has TCF enabled and the user is in an EEA country.
The contents for the TCF overlay are generated at runtime by looking at what data uses are on current systems (and soon) what features are on systems, to return Purposes, Special Purposes, Vendors (where we have a vendor_id, in other words, we know what type of system we have), Features and Special Features, and Systems (no vendor id).
- There are no Privacy Notices associated with a TCF experience. You save preferences against TCF components directly.
- Data uses need to be an "exact match" unlike Privacy Notices where we allow hierarchy-matching on data use
Adds the ability to save preferences with respect to individual TCF purposes, special purposes, vendors, special features, and features, and systems.
Adds the ability to save when individual TCF components were served.
When building the TCF Experience, surfaces any previously-saved or previously-served details alongside the TCF data.
Relaxes constraints on PrivacyPreferenceHistory, ServedNoticeHistory, CurrentPrivacyPreference, and LastServedNotice to use these tables to store when preferences were saved and served against not just notices, but TCF components as well.

Code Changes

Steps to Confirm

You need to be coming in from a region in the EEA like France
Also, tcf_enabled needs to be set to True: make a PATCH to /consent-settings and set tcf_enabled: true in the request body
You need a system with at least one data use that belongs to a GVL purpose or special purpose. For example, data_use: "essential.fraud_detection" maps to Special Purpose #1
That data use also needs to have a legal basis for processing of "Consent" or "Legitimate interests"
If you want a system to show up under "tcf_vendors", add a vendor_id to the system in the backend. Otherwise it will show up under "tcf_systems"

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

- Privacy Notices are not needed for the TCF Experience. Instead, the content is generated at runtime by looking at what TCF data uses are on current systems. - Add a new experience type: the TCF overlay. - Expands the abilities to retrieve and save consent preferences with respect to "vendors", "data_uses" and "features", not just "privacy notices". - Relaxes constraints on PrivacyPreferenceHistory, ServedNoticeHistory, CurrentPrivacyPreference, and LastServedNotice to use these tables to store when preferences were saved and served against not just notices, but vendors, data uses, and features.

…s/vendors. Improve logic around loading tcf data uses.

… need to track consent mechanism, enforcement level, or has_gpc_flag. - Start drying up code related to saving privacy preferences and unit testing w.r.t data use, vendor, or feature.

cypress · 2023-07-18T23:06:27Z

Passing run #4242 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `ff1e9c9` into `b9efa45`...
Project: fides	Commit: `2013be19ab ℹ️`
Status: Passed	Duration: 01:12 💡
Started: Sep 20, 2023 2:04 PM	Ended: Sep 20, 2023 2:06 PM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

…notices were served for various identities. Refactor methods now that we're sharing a lot of code here.

…s on getting settings.

- Split TCF schemas into their own file - Add test verifying that we can't create PrivacyPreferenceHistory records against multiple preference types. - Rename schemas for saving/serializing that notices were served to be more generic. - Write unit test for saving that tcf items were served.

src/fides/api/api/v1/endpoints/consent_settings_endpoints.py

…with many data uses, rather than a "data use" directly, as a purpose can be associated with many data uses. - Temporarily pin to a fideslang commit on a branch and pull in the tcf purpose mapping from there.

…thod. - Fix bug with saving current privacy preferences with respect to a special purpose and special feature.

- Fix duplicate region in EEA_COUNTRIES - Add test for ConsentSetting model - Add test for PrivacyExperience.get_should_show_banner in the case where it is a TCF overlay - I return True here - Add test for PrivacyExperience.get_related_privacy_notices for TCF overlay - exits early with an empty list - not relevant here! - Add method tests for cache_saved_and_served_on_consent_record for special purpose, notice, and vendor - Add docstrings to TCF schemas and move TCFExperienceContents out of the file for schemas that are returned via the API - Add test for PrivacyPrefenceHistory.preference_type property - Add some checks on PrivacyPreferenceHistory and CurrentPrivacyPreference records on create - Rename PreferenceType enum and preference_type property to be more generic since they're also shared with served notices - Docstrings for schemas - Updated to TCF_COMPONENT_MAPPING and TCF_PREFERENCES_FIELD_MAPPING - Add test for privacy_notice_id - that it works w/ TCF too

- Added more fields to the vendor record - "has_notices" flag now also takes into account if TCF content exists too. - Tried to take care of areas accessing info off of the privacy notice history - "relevant_systems" definition has changed, and also now factors in TCF components. - Fix bug where wrong field is returned in historical report - Add validation on purposes and special purposes and duplicates for both saving preferences and serving TCF components - Surface tcf version in consent report

- Fix bug related to Consent Settings scope name, copy/paste error - Fix test that relaxes returning systems as related even if they do not have system wide enforcement

…ng) work with TCF without error. - Don't return TCF data in the privacy center unless TCF is turned on. (hopefully the FE is just looking at privacy preferences saved against notices).

pattisdr · 2023-09-05T02:41:06Z

@adamsachs hoping you could review TCF backend this week! I've added vendors, features/special features, and did a sweep to see what remaining items needed to be addressed regarding code sharing with notices/experiences and the new tcf components. I'll annotate the PR with areas to focus on tomorrow 9/5

requirements.txt

src/fides/api/alembic/migrations/versions/66df7d9b8103_add_tcf_columns.py

src/fides/api/api/v1/endpoints/privacy_experience_endpoints.py

src/fides/api/models/privacy_experience.py

src/fides/data/privacy_notices/privacy_experience_config_defaults.yml

src/fides/api/util/consent_util.py

src/fides/api/schemas/privacy_preference.py

src/fides/api/schemas/privacy_experience.py

src/fides/api/api/v1/endpoints/consent_settings_endpoints.py

pattisdr · 2023-09-08T20:07:41Z

TODO:

Revert making consent settings public Dynamically send TCF bundle based on location #3957
-Accommodate case insensitive regions (will likely be just getting up to date w/ main)

adamsachs

@pattisdr high level this is looking great! nothing stands out to me as particularly problematic.

i still haven't been able to get down to the level of detail i'd like to in a few areas, namely around the TCF overlay rendering logic. i do think that's worth a thorough review, i just didn't get the time for it today. i'll look to circle back to that on monday, but in the mean time wanted to leave you with some of the comments i had in the meantime!

src/fides/api/api/v1/endpoints/consent_settings_endpoints.py

src/fides/api/schemas/consent_settings.py

src/fides/api/api/v1/endpoints/privacy_experience_endpoints.py

src/fides/api/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/alembic/migrations/versions/66df7d9b8103_add_tcf_columns.py

src/fides/api/util/tcf_util.py

adamsachs

OK! i took a deeper dive into the TCF overlay population logic and it looks great 👍 really nice job taking a lot of care to get that right, there's a lot packed in there.

i just had some minor comments that may help to improve readability, though by and large this is quite readable for the complexity involved.

i'd also like to get your thoughts on the broader comment i have about a future refactor, it'd be good to align on how we see this shaping up for future iterations!

src/fides/api/util/tcf_util.py

tests/ops/util/test_tcf_util.py

# Conflicts: # requirements.txt Pin fideslang to TCF work branched off of 1.4.6. Bump downrev of tcf columns.

- Removing auto-generated messaging template commits from my already-large migration. - Revert consent settings endpoint to being public again - Add an alias to for has_notices query param. - Try to better clarify when saving TCF preferences that nothing is saved if preferences are an empty list. - Fix mypy region issue - Add code comments to automatically generated migration that adds a lot of new columns - Get rid of _clone_top_level_record_then_add_legal_bases function - Add assert_length_of_tcf_sections helper, and add an auto use fixture for clearing the cache - "..ad_serving" is not a fideslang data use

src/fides/api/util/tcf_util.py

# Conflicts: # requirements.txt # tests/fixtures/application_fixtures.py

- Update features to v3 - Get rid of lru_cache. - Rearrange where we're extending legal bases

adamsachs

i think we're ready to go! amazing work, this is a milestone PR. really exciting to have this coming into main! 🙌

thanks for bearing with me on some of the nits 😄

pattisdr · 2023-09-18T22:38:46Z

great thanks for your comments @adamsachs plan to merge tomorrow after 2.20 is released

…ven (#3870) Co-authored-by: eastandwestwind <[email protected]> Co-authored-by: Adam Sachs <[email protected]>

pattisdr · 2023-09-20T13:45:40Z

Getting up to date with main again, error with migration heads -

Conflicts: Changelog, migrations.

pattisdr added 6 commits July 17, 2023 19:11

Revert cython changes.

7a06cd1

Merge branch 'main' into tcf_backend_poc

19df040

Address original failing tests.

ddc0a5d

DRY up code around getting saved and served notices/data uses/feature…

2c9a7a3

…s/vendors. Improve logic around loading tcf data uses.

Get rid of persisting tcf_details in a jsonb column now that we don't…

07eb2dd

… need to track consent mechanism, enforcement level, or has_gpc_flag. - Start drying up code related to saving privacy preferences and unit testing w.r.t data use, vendor, or feature.

Dry up user data collected for consent reporting. Dry up saving that …

59dbe0b

…notices were served for various identities. Refactor methods now that we're sharing a lot of code here.

pattisdr mentioned this pull request Jul 19, 2023

Connect TCF Backend to Return Proper Data #3822

Closed

pattisdr added 5 commits July 19, 2023 15:07

Address early linting issues.

452e9f3

Add simple tests for consent settings endpoints and adjust permission…

445aa4b

…s on getting settings.

Merge branch 'main' into tcf_backend_poc

ac051e9

Bump downrev.

f5f247b

pattisdr changed the title ~~TCF Backend POC~~ TCF Data Model and Endpoint Updates Jul 19, 2023

adamsachs reviewed Jul 20, 2023

View reviewed changes

src/fides/api/api/v1/endpoints/consent_settings_endpoints.py Show resolved Hide resolved

pattisdr added 6 commits July 21, 2023 15:52

Refactor to save preferences against a "purpose" which is associated …

b859941

…with many data uses, rather than a "data use" directly, as a purpose can be associated with many data uses. - Temporarily pin to a fideslang commit on a branch and pull in the tcf purpose mapping from there.

Add separate columns to track special purposes and special features.

050c651

Merge branch 'main' into tcf_backend_poc

fbe40e7

Dry up embed_experience_details using new get_related_tcf_contents me…

8ce5794

…thod. - Fix bug with saving current privacy preferences with respect to a special purpose and special feature.

Dry up some repeated code when saving preferences or notices served.

18b037e

allisonking mentioned this pull request Jul 27, 2023

Refactor overlay to allow components to not necessarily be notice driven #3870

Merged

9 tasks

pattisdr added 5 commits July 27, 2023 17:08

Refactor get_tcf_contents.

8b1aba7

Make progress towards getting tests passing:

c4f1fb3

- Fix bug related to Consent Settings scope name, copy/paste error - Fix test that relaxes returning systems as related even if they do not have system wide enforcement

Merge branch 'main' into tcf_backend_poc

18e3384

Add new dataset annotations and fix some issues with existing.

a55fe7d

pattisdr marked this pull request as ready for review July 28, 2023 14:23

pattisdr changed the title ~~TCF Data Model and Endpoint Updates~~ TCF Backend: Purposes, Special Purposes, and Vendors Jul 28, 2023

For completeness, verify that privacy center endpoints (getting, savi…

163815c

…ng) work with TCF without error. - Don't return TCF data in the privacy center unless TCF is turned on. (hopefully the FE is just looking at privacy preferences saved against notices).

pattisdr added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Sep 1, 2023

Merge main

50ab1bd

pattisdr requested a review from adamsachs September 5, 2023 02:38

pattisdr commented Sep 5, 2023

View reviewed changes

Merge branch 'main' into tcf_backend_poc

9137949

adamsachs reviewed Sep 8, 2023

View reviewed changes

adamsachs reviewed Sep 11, 2023

View reviewed changes

src/fides/api/util/tcf_util.py Show resolved Hide resolved

src/fides/api/util/tcf_util.py Outdated Show resolved Hide resolved

tests/ops/util/test_tcf_util.py Outdated Show resolved Hide resolved

tests/ops/util/test_tcf_util.py Show resolved Hide resolved

pattisdr added 2 commits September 12, 2023 15:16

Merge branch 'main' into tcf_backend_poc

39aecf0

# Conflicts: # requirements.txt Pin fideslang to TCF work branched off of 1.4.6. Bump downrev of tcf columns.

adamsachs reviewed Sep 13, 2023

View reviewed changes

src/fides/api/util/tcf_util.py Outdated Show resolved Hide resolved

adamsachs reviewed Sep 13, 2023

View reviewed changes

src/fides/api/util/tcf_util.py Outdated Show resolved Hide resolved

pattisdr added 5 commits September 14, 2023 18:10

Merge branch 'main' into tcf_backend_poc

9197765

# Conflicts: # requirements.txt # tests/fixtures/application_fixtures.py

Bump downrev

19cc0aa

Continue to respond to CR

d87fce5

- Update features to v3 - Get rid of lru_cache. - Rearrange where we're extending legal bases

Pylint

8ccea25

Bump fideslang requirement to 2.1.0

ca3928d

adamsachs approved these changes Sep 18, 2023

View reviewed changes

pattisdr added 3 commits September 18, 2023 11:58

Merge branch 'main' into tcf_backend_poc

1f6f061

Update changelog.

ca1d498

Merge branch 'main' into tcf_backend_poc

97c4507

pattisdr and others added 2 commits September 19, 2023 11:51

Merge branch 'main' into tcf_backend_poc

0523079

Refactor overlay to allow components to not necessarily be notice dri…

db4a7ed

…ven (#3870) Co-authored-by: eastandwestwind <[email protected]> Co-authored-by: Adam Sachs <[email protected]>

Merge main:

ff1e9c9

Conflicts: Changelog, migrations.

pattisdr merged commit e279130 into main Sep 20, 2023
46 of 50 checks passed

pattisdr deleted the tcf_backend_poc branch September 20, 2023 15:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TCF Backend #3804

TCF Backend #3804

pattisdr commented Jul 18, 2023 •

edited

Loading

cypress bot commented Jul 18, 2023 •

edited

Loading

pattisdr commented Sep 5, 2023

pattisdr commented Sep 8, 2023 •

edited

Loading

adamsachs left a comment

adamsachs left a comment

adamsachs left a comment

pattisdr commented Sep 18, 2023

pattisdr commented Sep 20, 2023

TCF Backend #3804

TCF Backend #3804

Conversation

pattisdr commented Jul 18, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Jul 18, 2023 • edited Loading

Passing run #4242 ↗︎

pattisdr commented Sep 5, 2023

pattisdr commented Sep 8, 2023 • edited Loading

adamsachs left a comment

Choose a reason for hiding this comment

adamsachs left a comment

Choose a reason for hiding this comment

adamsachs left a comment

Choose a reason for hiding this comment

pattisdr commented Sep 18, 2023

pattisdr commented Sep 20, 2023

pattisdr commented Jul 18, 2023 •

edited

Loading

cypress bot commented Jul 18, 2023 •

edited

Loading

pattisdr commented Sep 8, 2023 •

edited

Loading