Developer Friendly Privacy Preferences #4221

pattisdr · 2023-10-03T21:59:45Z

Closes https://github.com/ethyca/fidesplus/issues/1115

❗ Dependent on Legal Basis Dimension refactor #4201

Description Of Changes

Adds the ability to take in a TC string and save privacy preferences on the backend

Code Changes

Adds the ability to take in a TC string and save privacy preferences. Only "purpose_consent_preferences", "purpose_legitimate_interests_preferences", "vendor_consent_preferences", "vendor_legitimate_interests_preferences", and "special_purpose_preferences" are saved from the string, as this is all that can be represented by the string.
Returns the decoded string in the response in mobile format

Steps to Confirm

Turn TCF on with an environment variable (for example, set tcf_enabled = true in your fides.toml and run nox -s dev -- shell
PATCH Privacy preferences with just a fides user device id and a tc string should work:

This is a random string generated by IAB TCF Encoder - it doesn't actually overlap any of my datamap, so nothing is actually saved

curl -X 'PATCH' \
  'http://localhost:8080/api/v1/privacy-preferences' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "browser_identity": {
    "fides_user_device_id": "f54b192b-6b97-4312-b5f5-4608eb8f0558"
  },
  "tc_string": "CPzKmsNPzKmsNGJAAAENCZCAACIAABQAAAAAABMABEgAAA.II7Nd_X__bX9n-_7_6ft0eY1f9_r37uQzDhfNs-8F3L_W_LwX32E7NF36tq4KmR4ku1bBIQNtHMnUDUmxaolVrzHsak2cpyNKJ_JkknsZe2dYGF9Pn9lD-YKZ7_5_9_f52T_9_9_-39z3_9f___dv_-__-vjf_599n_v9fV_78_Kf9______-____________8A"
}'

Response - notice how the preferences arrays are empty lists. We still decode the string here -

{
  "preferences": [],
  "purpose_consent_preferences": [],
  "purpose_legitimate_interests_preferences": [],
  "special_purpose_preferences": [],
  "vendor_consent_preferences": [],
  "vendor_legitimate_interests_preferences": [],
  "feature_preferences": [],
  "special_feature_preferences": [],
  "system_consent_preferences": [],
  "system_legitimate_interests_preferences": [],
  "tc_mobile_data": {
    "IABTCF_CmpSdkID": 393,
    "IABTCF_CmpSdkVersion": 0,
    "IABTCF_PolicyVersion": 2,
    "IABTCF_gdprApplies": 1,
    "IABTCF_PublisherCC": "AA",
    "IABTCF_PurposeOneTreatment": 0,
    "IABTCF_UseNonStandardTexts": 0,
    "IABTCF_TCString": "CPzKmsNPzKmsNGJAAAENCZCAACIAABQAAAAAABMABEgAAA.II7Nd_X__bX9n-_7_6ft0eY1f9_r37uQzDhfNs-8F3L_W_LwX32E7NF36tq4KmR4ku1bBIQNtHMnUDUmxaolVrzHsak2cpyNKJ_JkknsZe2dYGF9Pn9lD-YKZ7_5_9_f52T_9_9_-39z3_9f___dv_-__-vjf_599n_v9fV_78_Kf9______-____________8A",
    "IABTCF_VendorConsents": "11",
    "IABTCF_VendorLegitimateInterests": "1001",
    "IABTCF_PurposeConsents": "001000100000000000000000",
    "IABTCF_PurposeLegitimateInterests": "000101000000000000000000",
    "IABTCF_SpecialFeaturesOptIns": "000000000000",
    "IABTCF_PublisherConsent": null,
    "IABTCF_PublisherLegitimateInterests": null,
    "IABTCF_PublisherCustomPurposesConsents": null,
    "IABTCF_PublisherCustomPurposesLegitimateInterests": null
  }
}

Add a system to the datamap whose data use is in the GVL - I added emerse
GET TCF Privacy Experience for France, and get the accept all TC string from the request body

curl -X 'GET' \
  'http://localhost:8080/api/v1/privacy-experience?show_disabled=true&region=fr_idg&systems_applicable=false&include_gvl=false&include_meta=true&page=1&size=50' \
  -H 'accept: application/json'

Send a PATCH with that accept all string.

curl -X 'PATCH' \
  'http://localhost:8080/api/v1/privacy-preferences' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "browser_identity": {
    "fides_user_device_id": "f54b192b-6b97-4312-b5f5-4608eb8f0558"
  },
  "tc_string": "CPzHq4APzHq4AAMABBENAUEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA"
}'

Response:

{
  "preferences": [],
  "purpose_consent_preferences": [
    {
      "purpose_consent": 1,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_c5104a61-72a3-4aa2-89cd-1a6c87758952",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_c4d2f353-4593-4c2a-813a-7eddabf5a3ad"
    },
    {
      "purpose_consent": 3,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_5812ec3f-a9e3-49ae-a9c5-6451d3b2af26",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_74b48cde-013f-469c-aebd-4aa46d65261b"
    },
    {
      "purpose_consent": 4,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_106f2e3e-bdad-4d90-b06e-67005a07ba86",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_b340a017-467c-48e9-97cf-21b74349b123"
    }
  ],
  "purpose_legitimate_interests_preferences": [
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": 2,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_e87076ae-8fde-4eed-931b-ea70640fb100",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_ba121e46-edba-4d62-8caa-c48d55c79aa1"
    },
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": 7,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_e37f9146-5b2b-47b9-8d0d-8a99afc43fbd",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_b7286ff5-5910-4ba0-bf0c-db31c0be1079"
    },
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": 8,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_ddc910f1-ea49-4666-aaf7-ac3291eda427",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_11b92d34-ae20-44ef-970e-328e98d10031"
    },
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": 9,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_df859df7-b667-4947-a6f8-3c2d7869ae74",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_1aa73b1b-5cf9-4328-aff1-dccc2b2d54a8"
    }
  ],
  "special_purpose_preferences": [],
  "vendor_consent_preferences": [
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": "8",
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_06274f9c-79fc-4f19-b9a3-e61cc4d56701",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_a70334fa-f038-477f-aa49-98f3a29f3dbb"
    }
  ],
  "vendor_legitimate_interests_preferences": [
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": null,
      "vendor_legitimate_interests": "8",
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_4fa88140-9609-4728-b7b2-5633ebfa2b90",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_8fc53cd9-e268-4c1c-850c-45274ff94bf5"
    }
  ],
  "feature_preferences": [],
  "special_feature_preferences": [],
  "system_consent_preferences": [],
  "system_legitimate_interests_preferences": [],
  "tc_mobile_data": {
    "IABTCF_CmpSdkID": 12,
    "IABTCF_CmpSdkVersion": 1,
    "IABTCF_PolicyVersion": 4,
    "IABTCF_gdprApplies": 1,
    "IABTCF_PublisherCC": "AA",
    "IABTCF_PurposeOneTreatment": 0,
    "IABTCF_UseNonStandardTexts": 0,
    "IABTCF_TCString": "CPzHq4APzHq4AAMABBENAUEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA",
    "IABTCF_VendorConsents": "00000001",
    "IABTCF_VendorLegitimateInterests": "00000001",
    "IABTCF_PurposeConsents": "101100000000000000000000",
    "IABTCF_PurposeLegitimateInterests": "010000111000000000000000",
    "IABTCF_SpecialFeaturesOptIns": "000000000000",
    "IABTCF_PublisherConsent": null,
    "IABTCF_PublisherLegitimateInterests": null,
    "IABTCF_PublisherCustomPurposesConsents": null,
    "IABTCF_PublisherCustomPurposesLegitimateInterests": null
  }
}

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

…ferences.

pattisdr · 2023-10-03T22:04:25Z

@eastandwestwind @allisonking just tagging you for an early look so we're on the same page about saving privacy preferences to the backend. Want to get your thoughts on the strategy.

So a tricky part, I'm also comparing the experience itself to the TC string to settle this point:

Just because a value is a 0 in the TC string, doesn't mean the user explicitly opted out, it might not have been presented as an option for them to opt in or out of in the first place, because that purpose is not in the data map at all. So I loop through the elements from the datamap first and then see if it's True in the string, otherwise it's saved as False.

cypress · 2023-10-03T22:14:01Z

Passing run #4491 ↗︎

0	4	0	0	0

Details:

Merge `6219d10` into `8933d7d`...
Project: fides	Commit: `b57fd2ce56 ℹ️`
Status: Passed	Duration: 00:57 💡
Started: Oct 6, 2023 2:03 AM	Ended: Oct 6, 2023 2:04 AM

Review all test suite changes for PR #4221 ↗︎

codecov · 2023-10-03T22:46:02Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (354c952) 87.69% compared to head (6219d10) 87.72%.
Report is 1 commits behind head on fidesplus_1128_legal_basis_dimension.

Additional details and impacted files

@@                           Coverage Diff                            @@
##           fidesplus_1128_legal_basis_dimension    #4221      +/-   ##
========================================================================
+ Coverage                                 87.69%   87.72%   +0.02%     
========================================================================
  Files                                       331      331              
  Lines                                     20777    20869      +92     
  Branches                                   2698     2708      +10     
========================================================================
+ Hits                                      18221    18307      +86     
- Misses                                     2091     2095       +4     
- Partials                                    465      467       +2

Files	Coverage Δ
...i/api/v1/endpoints/privacy_preference_endpoints.py	`98.23% <100.00%> (+0.17%)`	⬆️
src/fides/api/common_exceptions.py	`91.48% <100.00%> (+0.09%)`	⬆️
src/fides/api/schemas/privacy_experience.py	`100.00% <100.00%> (ø)`
src/fides/api/schemas/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/schemas/tcf.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/experience_meta.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/tc_string.py	`97.95% <94.73%> (-2.05%)`	⬇️
src/fides/api/util/tcf/tc_mobile_data.py	`91.42% <88.46%> (-8.58%)`	⬇️

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…15_dev_friendly_privacy_preferences # Conflicts: # src/fides/api/schemas/privacy_preference.py # tests/ops/api/v1/endpoints/test_privacy_preference_endpoints.py

…15_dev_friendly_privacy_preferences

src/fides/api/util/tcf/tc_mobile_data.py

src/fides/api/util/tcf/tc_string.py

tests/ops/util/test_tc_string.py

src/fides/api/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/schemas/privacy_experience.py

adamsachs

@pattisdr this looks great! just some very minor comments but this was impressively easy to follow given the complexity. thanks for the helpful self-review comments as always 👍

maybe hold off on merging into the base branch so that change can be fully reviewed independently? if that's a pain, then ignore me.

src/fides/api/util/tcf/tc_string.py

src/fides/api/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/util/tcf/tc_mobile_data.py

src/fides/api/util/tcf/tc_string.py

tests/fixtures/application_fixtures.py

src/fides/api/schemas/privacy_preference.py

First pass - add ability to take in a TC string only and save the pre…

e24f21a

…ferences.

pattisdr requested review from eastandwestwind and allisonking October 3, 2023 22:00

pattisdr added 5 commits October 4, 2023 12:45

Decode the TC string into mobile data alongside the preferences.

16c80bf

Merge branch 'fidesplus_1128_legal_basis_dimension' into fidesplus_11…

8b0dd53

…15_dev_friendly_privacy_preferences # Conflicts: # src/fides/api/schemas/privacy_preference.py # tests/ops/api/v1/endpoints/test_privacy_preference_endpoints.py

Add tests around converting a TC string to mobile.

3674e8e

Catch any remaining decode string errors

cbe4250

Merge branch 'fidesplus_1128_legal_basis_dimension' into fidesplus_11…

2d0dbe6

…15_dev_friendly_privacy_preferences

pattisdr marked this pull request as ready for review October 4, 2023 21:50

pattisdr requested a review from adamsachs October 5, 2023 13:43

pattisdr commented Oct 5, 2023

View reviewed changes

adamsachs approved these changes Oct 5, 2023

View reviewed changes

Add a code comment around experience performance

6219d10

pattisdr merged commit c0120f0 into fidesplus_1128_legal_basis_dimension Oct 6, 2023

pattisdr deleted the fidesplus_1115_dev_friendly_privacy_preferences branch October 6, 2023 16:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Developer Friendly Privacy Preferences #4221

Developer Friendly Privacy Preferences #4221

pattisdr commented Oct 3, 2023 •

edited

Loading

pattisdr commented Oct 3, 2023

cypress bot commented Oct 3, 2023 •

edited

Loading

codecov bot commented Oct 3, 2023 •

edited

Loading

adamsachs left a comment

Developer Friendly Privacy Preferences #4221

Developer Friendly Privacy Preferences #4221

Conversation

pattisdr commented Oct 3, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

pattisdr commented Oct 3, 2023

cypress bot commented Oct 3, 2023 • edited Loading

Passing run #4491 ↗︎

Review all test suite changes for PR #4221 ↗︎

codecov bot commented Oct 3, 2023 • edited Loading

Codecov Report

adamsachs left a comment

Choose a reason for hiding this comment

pattisdr commented Oct 3, 2023 •

edited

Loading

cypress bot commented Oct 3, 2023 •

edited

Loading

codecov bot commented Oct 3, 2023 •

edited

Loading