Developer-Friendly TCF Experiences #4160

pattisdr · 2023-09-25T23:16:20Z

Description Of Changes

Adjust the response of GET Privacy Experience to make it more developer-friendly.

Code Changes

Adds minimal ability for encoding accept-all and reject-all TC strings in Python
A new ?include_meta query param on GET Privacy Experience adds a meta section to the TCF Experience with the following details:
- A version_hash which includes the most important details about the TCF experience. Comparing these hashes between subsequent fetches should determine if consent needs to be resurfaced for TCF
- An accept_all_tc_string - The TC string if the user were to opt-in to everything displayed in the TCF experience, dynamically populated by systems in the data map
- A reject_all_tc_string - The TC string if the user were to opt-out of everything displayed in the TCF experience, dynamically populated by systems in the data map
- Similar accept_all_tc_mobile_data and reject_all_tc_mobile_data for use in building mobile apps with the raw contents to build TC strings
Adds a new EEA region and creates a TCF Overlay Experience for that region. This is used for fetching a generic TCF experience that works with any region in the EEA.
Adds new iab-tcf requirement for decoding TC strings

Steps to Confirm

Enable TCF
Add systems with a data use in the GVL mapping and a legal basis of consent or legitimate interest
Make an API request with region=eea and include_meta=True

curl -X 'GET' \
  'http://localhost:8080/api/v1/privacy-experience?show_disabled=true&region=eea&component=overlay&has_notices=true&has_config=true&systems_applicable=false&include_meta=true&page=1&size=50' \
  -H 'accept: application/json'

Note new meta section, string will vary based on your applicable systems. Decode the TC strings here

{
   "meta":{
      "version_hash":"eaab1c195073",
  "accept_all_tc_string":"CPyqAcAPyqAcAAMABBENASEEAPLAAAAAAAAAABEAAAAA.IgoMV_H3_bX9v8Tq16ft0eYFX99j77uQxBhfJs-4FyDvW_JwVn2EyNAUytqYKmRIEuzZBIQFsHJnURVChQogVryHsYkGMhTEAJ-BkgHMRa2dYCFxvmYtjEQIZ5_p_czf52R_9_dn-3czjT9lnnzVtO2dlcKidS59tH_l_bRKb-7Ied-7W_4v09tfrkW_eTVu_9evvp9-u_s_v__934AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ",
      "accept_all_tc_mobile_data":{
         "IABTCF_CmpSdkID":12,
         "IABTCF_CmpSdkVersion":1,
         "IABTCF_PolicyVersion":4,
         "IABTCF_gdprApplies":1,
         "IABTCF_PublisherCC":"AA",
         "IABTCF_PurposeOneTreatment":0,
         "IABTCF_UseNonStandardTexts":0,
         "IABTCF_TCString":"CPyqAcAPyqAcAAMABBENASEEAPLAAAAAAAAAABEAAAAA.IgoMV_H3_bX9v8Tq16ft0eYFX99j77uQxBhfJs-4FyDvW_JwVn2EyNAUytqYKmRIEuzZBIQFsHJnURVChQogVryHsYkGMhTEAJ-BkgHMRa2dYCFxvmYtjEQIZ5_p_czf52R_9_dn-3czjT9lnnzVtO2dlcKidS59tH_l_bRKb-7Ied-7W_4v09tfrkW_eTVu_9evvp9-u_s_v__934AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ",
         "IABTCF_VendorConsents":"01",
         "IABTCF_VendorLegitimateInterests":"",
         "IABTCF_PurposeConsents":"111100101100000000000000",
         "IABTCF_PurposeLegitimateInterests":"000000000000000000000000",
         "IABTCF_SpecialFeaturesOptIns":"010000000000",
         "IABTCF_PublisherConsent":null,
         "IABTCF_PublisherLegitimateInterests":null,
         "IABTCF_PublisherCustomPurposesConsents":null,
         "IABTCF_PublisherCustomPurposesLegitimateInterests":null
      },
      "reject_all_tc_string":"CPyqAcAPyqAcAAMABBENASEAAAAAAAAAAAAAAAAAAAAA.IgoMV_H3_bX9v8Tq16ft0eYFX99j77uQxBhfJs-4FyDvW_JwVn2EyNAUytqYKmRIEuzZBIQFsHJnURVChQogVryHsYkGMhTEAJ-BkgHMRa2dYCFxvmYtjEQIZ5_p_czf52R_9_dn-3czjT9lnnzVtO2dlcKidS59tH_l_bRKb-7Ied-7W_4v09tfrkW_eTVu_9evvp9-u_s_v__934AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ",
      "reject_all_tc_mobile_data":{
         "IABTCF_CmpSdkID":12,
         "IABTCF_CmpSdkVersion":1,
         "IABTCF_PolicyVersion":4,
         "IABTCF_gdprApplies":1,
         "IABTCF_PublisherCC":"AA",
         "IABTCF_PurposeOneTreatment":0,
         "IABTCF_UseNonStandardTexts":0,
         "IABTCF_TCString":"CPyqAcAPyqAcAAMABBENASEAAAAAAAAAAAAAAAAAAAAA.IgoMV_H3_bX9v8Tq16ft0eYFX99j77uQxBhfJs-4FyDvW_JwVn2EyNAUytqYKmRIEuzZBIQFsHJnURVChQogVryHsYkGMhTEAJ-BkgHMRa2dYCFxvmYtjEQIZ5_p_czf52R_9_dn-3czjT9lnnzVtO2dlcKidS59tH_l_bRKb-7Ied-7W_4v09tfrkW_eTVu_9evvp9-u_s_v__934AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ",
         "IABTCF_VendorConsents":"",
         "IABTCF_VendorLegitimateInterests":"",
         "IABTCF_PurposeConsents":"000000000000000000000000",
         "IABTCF_PurposeLegitimateInterests":"000000000000000000000000",
         "IABTCF_SpecialFeaturesOptIns":"000000000000",
         "IABTCF_PublisherConsent":null,
         "IABTCF_PublisherLegitimateInterests":null,
         "IABTCF_PublisherCustomPurposesConsents":null,
         "IABTCF_PublisherCustomPurposesLegitimateInterests":null
      }
   }
}

Fetch the experience again and verify the version_hash doesn't change. Add a new vendor, update your data use on your existing system, or update your legal basis, and verify the version hash changes

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

…l core tc string.

- Fix vendor legitimate interests section - Fix padding bitstring with the appropriate number of bits

…rpose with a legal basis of consent. - Also add handling for a reject-all tc string.

…odel.

…erience contents so we don't have to build this again. - Add a first stab at the mobile data object schema - Add a tc_string key to the schema for saving privacy preferences.

…e for reject all.

…rs from the tc string - not the entire experience, not the experience config, etc.

…when GET PrivacyExperience is called without filters.

…r some of the methods to only build the tc models up front since they are used for the tc string and mobile data.

…thout knowing the location of a specific user.

…s vendor is not in the GVL

- Make TCMobileData format match casing. - Share number of bits definitions between similar locations when building the TC model and the TC Mobile data model - Add tests for data serialized by TCMobileData - Move TCFVersionHash to experience meta file where it's being used - Surface same TC string under mobile data and top level - Add tests for version hash model sorting ascending

cypress · 2023-09-25T23:36:19Z

Passing run #4422 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `d1a4adf` into `ddbae00`...
Project: fides	Commit: `3d37d06452 ℹ️`
Status: Passed	Duration: 00:49 💡
Started: Sep 29, 2023 5:29 PM	Ended: Sep 29, 2023 5:30 PM

Review all test suite changes for PR #4160 ↗︎

CHANGELOG.md

requirements.txt

src/fides/api/api/v1/endpoints/privacy_experience_endpoints.py

src/fides/api/schemas/privacy_experience.py

src/fides/api/util/tcf/experience_meta.py

src/fides/api/util/tcf/tc_mobile_data.py

src/fides/api/util/tcf/tc_model.py

src/fides/api/util/tcf/tc_string.py

src/fides/api/util/tcf/tc_mobile_data.py

src/fides/api/util/tcf/tc_string.py

pattisdr · 2023-09-26T16:02:29Z

Pulling in the GVL work!

# Conflicts: # CHANGELOG.md # src/fides/api/api/v1/endpoints/privacy_experience_endpoints.py # src/fides/api/models/privacy_experience.py # src/fides/api/schemas/privacy_experience.py # tests/ops/api/v1/endpoints/test_privacy_experience_endpoints.py

…l bases by comparing against the GVL.

…a test relying on the current ordering)

…f folder.

allisonking

amazing work @pattisdr !! left a few comments, but they're more me thinking out loud than any requested changes. for the most part this looks to match the FE well 👍

src/fides/api/models/privacy_notice.py

src/fides/api/util/tcf/tc_model.py

src/fides/api/util/tcf/tc_string.py

pattisdr · 2023-09-26T19:54:55Z

Thanks so much for reviewing @allisonking and for your initial pointers on this work to get it headed in the right direction. I'm going to add the change for the disclosed vendors, I agree with your interpretation 👍

codecov · 2023-09-26T20:33:50Z

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (ddbae00) 87.50% compared to head (d1a4adf) 87.67%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4160      +/-   ##
==========================================
+ Coverage   87.50%   87.67%   +0.16%     
==========================================
  Files         329      333       +4     
  Lines       20452    20734     +282     
  Branches     2656     2690      +34     
==========================================
+ Hits        17897    18178     +281     
  Misses       2093     2093              
- Partials      462      463       +1

Files	Coverage Δ
...i/api/v1/endpoints/privacy_experience_endpoints.py	`93.20% <100.00%> (+0.27%)`	⬆️
...es/api/api/v1/endpoints/served_notice_endpoints.py	`96.15% <100.00%> (ø)`
src/fides/api/models/privacy_experience.py	`98.59% <100.00%> (+0.06%)`	⬆️
src/fides/api/models/privacy_notice.py	`98.83% <ø> (ø)`
src/fides/api/models/privacy_preference.py	`96.89% <100.00%> (ø)`
src/fides/api/schemas/privacy_experience.py	`100.00% <100.00%> (ø)`
src/fides/api/schemas/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/util/consent_util.py	`98.87% <ø> (ø)`
src/fides/api/util/tcf/experience_meta.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/tc_mobile_data.py	`100.00% <100.00%> (ø)`
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…re both in the gvl, and appeared in the TCF experience. Previously, I was adding all the vendors in the gvl to match the library. - Add additional test asserting tc string with multiple systems configured.

# Conflicts: # CHANGELOG.md

allisonking

awesome work @pattisdr ! new TC string expert 🎉 I tested it out locally and things work really well 💯

src/fides/api/util/tcf/experience_meta.py

src/fides/api/util/tcf/tc_model.py

pattisdr · 2023-09-29T17:07:39Z

😄 Thank you so much for your review @allisonking!

pattisdr added 20 commits September 21, 2023 11:19

POC Commit - explore hashing the experience and building an accept al…

e354c2a

…l core tc string.

Add IAB TCF Python library for decoding.

4e671c5

- Fix vendor legitimate interests section - Fix padding bitstring with the appropriate number of bits

Let's go ahead and convert to snake casing now.

98aeb36

Vendor ids shouldn't show up in Vendor consents unless they have a pu…

006284f

…rpose with a legal basis of consent. - Also add handling for a reject-all tc string.

Add TCField to avoid adding specific encoding instructions to the TCM…

330832e

…odel.

Separate TC string and TC model into separate files.

15b2b27

Add date and disclosed vendor assertions.

0d95a80

Refactor methods that build TC model and string to take in the tc exp…

da2c3ad

…erience contents so we don't have to build this again. - Add a first stab at the mobile data object schema - Add a tc_string key to the schema for saving privacy preferences.

Reorganize so we return two tc data blocks, one for accept all and on…

e2c27e5

…e for reject all.

Refactor building a TCF experience hash to just focus on key indicato…

217b357

…rs from the tc string - not the entire experience, not the experience config, etc.

Reorganize schema to flatten one level for mobile data

285725e

Rearrange when get_tcf_contents is called, so it is only called once …

36f3348

…when GET PrivacyExperience is called without filters.

Split off building the experience meta into its own file, and refacto…

4425aa8

…r some of the methods to only build the tc models up front since they are used for the tc string and mobile data.

Merge branch 'main' into fidesplus_1102_PROD-1077_dev_friendly_exp

f21729a

Linting.

de1b621

Gate TCF meta being returned behind an include_meta query param.

30059c0

Add the "eea" region to be called to fetch TCF Overlay Experiences wi…

914f197

…thout knowing the location of a specific user.

Add assertions when system is included with relevant purposes, but it…

8b3e1bb

…s vendor is not in the GVL

Update changelog

51c7069

pattisdr commented Sep 26, 2023

View reviewed changes

src/fides/api/util/tcf/tc_mobile_data.py Show resolved Hide resolved

pattisdr commented Sep 26, 2023

View reviewed changes

src/fides/api/util/tcf/tc_string.py Show resolved Hide resolved

pattisdr requested review from adamsachs, allisonking and NevilleS September 26, 2023 00:15

pattisdr added 2 commits September 26, 2023 11:25

Add some additional validation to remove vendors with disallowed lega…

38751b1

…l bases by comparing against the GVL.

pattisdr added 2 commits September 26, 2023 14:21

Move the EEA to the bottom of the list of EEA countries (as there is …

544d0a9

…a test relying on the current ordering)

Move the file that builds the TCF Experience contents into the new tc…

cc6cf4c

…f folder.

allisonking reviewed Sep 26, 2023

View reviewed changes

src/fides/api/models/privacy_notice.py Show resolved Hide resolved

src/fides/api/util/tcf/tc_model.py Show resolved Hide resolved

src/fides/api/util/tcf/tc_string.py Show resolved Hide resolved

src/fides/api/util/tcf/tc_string.py Show resolved Hide resolved

Update definition of disclosed vendors to only include vendors that a…

00fa84c

…re both in the gvl, and appeared in the TCF experience. Previously, I was adding all the vendors in the gvl to match the library. - Add additional test asserting tc string with multiple systems configured.

allisonking mentioned this pull request Sep 26, 2023

Only list vendors in the data map as vendors_disclosed in the FE #4165

Closed

pattisdr requested a review from allisonking September 27, 2023 13:13

Merge branch 'main' into fidesplus_1102_PROD-1077_dev_friendly_exp

3d20685

# Conflicts: # CHANGELOG.md

allisonking approved these changes Sep 29, 2023

View reviewed changes

src/fides/api/util/tcf/experience_meta.py Show resolved Hide resolved

src/fides/api/util/tcf/tc_model.py Show resolved Hide resolved

Merge branch 'main' into fidesplus_1102_PROD-1077_dev_friendly_exp

d1a4adf

pattisdr merged commit 37f7708 into main Sep 29, 2023
38 of 39 checks passed

pattisdr deleted the fidesplus_1102_PROD-1077_dev_friendly_exp branch September 29, 2023 18:21

pattisdr added a commit that referenced this pull request Sep 29, 2023

Developer-Friendly TCF Experiences (#4160)

d6be45c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Developer-Friendly TCF Experiences #4160

Developer-Friendly TCF Experiences #4160

pattisdr commented Sep 25, 2023 •

edited by allisonking

Loading

cypress bot commented Sep 25, 2023 •

edited

Loading

pattisdr commented Sep 26, 2023

allisonking left a comment

pattisdr commented Sep 26, 2023

codecov bot commented Sep 26, 2023 •

edited

Loading

allisonking left a comment

pattisdr commented Sep 29, 2023

Developer-Friendly TCF Experiences #4160

Developer-Friendly TCF Experiences #4160

Conversation

pattisdr commented Sep 25, 2023 • edited by allisonking Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Sep 25, 2023 • edited Loading

Passing run #4422 ↗︎

Review all test suite changes for PR #4160 ↗︎

pattisdr commented Sep 26, 2023

allisonking left a comment

Choose a reason for hiding this comment

pattisdr commented Sep 26, 2023

codecov bot commented Sep 26, 2023 • edited Loading

Codecov Report

allisonking left a comment

Choose a reason for hiding this comment

pattisdr commented Sep 29, 2023

pattisdr commented Sep 25, 2023 •

edited by allisonking

Loading

cypress bot commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 26, 2023 •

edited

Loading