Build and Accept Backend AC Strings #4295

pattisdr · 2023-10-18T12:31:52Z

Closes https://github.com/ethyca/fidesplus/issues/1162

Description Of Changes

Add proper AC string support from the backend, both in saving privacy preferences from a string, and in returning meta information when building developer-friendly privacy experiences. Behavior nuances documented here.

❗ IMPORTANT: This adds a new AC Enabled setting! This must be set to True for AC functionality to work.

Code Changes

Saving Privacy Preferences:

Allows preferences to be saved from the AC string portion of a Fides string if AC mode enabled
- Allows a Fides string to include both a TC string and an AC string. New format is a TC string and an AC string joined by a comma: <tc_string>,<ac_string>.
- TC string and AC string are pulled from the string. TC string preferences saved per usual. Big change here is that we figure out Vendor consent preferences from the AC string and persist those as well
- Response includes the AC string under the IABTCF_AddtlConsent key in the mobile data

Retrieving Privacy Experiences

Updates to building developer-friendly privacy experiences meta section:
- When building accept all and reject all fides strings, return these in fides_string format, which includes the TC string and AC string in the same representation if applicable and AC mode is enabled
- Accept all and reject all mobile data also has populated IABTCF_AddtlConsent sections if AC mode is enabled

General

When building a TCModel from the TCF Experience, which has been a prerequisite for building a TC string, add a new ac_vendor_consents section to support building an AC string too.
Stricter handling to prevent AC systems defined in an invalid way from showing up in the Experience
New AC Enabled environment variable

Steps to Confirm

Note these instructions do not require you to be running the dictionary

Set both tcf_enabled AND ac_enabled to True in your toml file or use an env var
Run nox -s dev -- shell
Add a GVL system following these instructions
Add an AC system locally following these instructions

GET Experiences with AC String

Fetch an EEA Privacy Experience, with include_meta=true query param

http://localhost:8080/api/v1/privacy-experience?show_disabled=true&region=fr&systems_applicable=false&include_gvl=false&include_meta=true&page=1&size=50

From a previous PR, the AC system shows up under the Vendor Consents section of the experience
Note the meta section of the response. The accept_all_fides_string concatenates the TC string and the AC string. The TC string still goes under the IABTCF_TCString section and the IABTCF_AddtlConsent is now populated with the AC String. Reject all string looks like this: 1~. This is also what the accept-all string looks like if there are no AC systems.

 "meta": {
        "version_hash": "a2e85860c68b",
        "accept_all_fides_string": "CPz_s0APz_s0AGXABBENAWEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA,1~100",
        "accept_all_fides_mobile_data": {
          "IABTCF_CmpSdkID": 407,
          "IABTCF_CmpSdkVersion": 1,
          "IABTCF_PolicyVersion": 4,
          "IABTCF_gdprApplies": 1,
          "IABTCF_PublisherCC": "AA",
          "IABTCF_PurposeOneTreatment": 0,
          "IABTCF_UseNonStandardTexts": 0,
          "IABTCF_TCString": "CPz_s0APz_s0AGXABBENAWEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA",
          "IABTCF_VendorConsents": "00000001",
          "IABTCF_VendorLegitimateInterests": "00000001",
          "IABTCF_PurposeConsents": "101100000000000000000000",
          "IABTCF_PurposeLegitimateInterests": "010000111000000000000000",
          "IABTCF_SpecialFeaturesOptIns": "000000000000",
          "IABTCF_PublisherConsent": null,
          "IABTCF_PublisherLegitimateInterests": null,
          "IABTCF_PublisherCustomPurposesConsents": null,
          "IABTCF_PublisherCustomPurposesLegitimateInterests": null,
          "IABTCF_AddtlConsent": "1~100"
        },
        "reject_all_fides_string": "CPz_s0APz_s0AGXABBENAWEAAAAAAAAAAAAAAAAAAAAA.IAEAEAAA,1~",
        "reject_all_fides_mobile_data": {
          "IABTCF_CmpSdkID": 407,
          "IABTCF_CmpSdkVersion": 1,
          "IABTCF_PolicyVersion": 4,
          "IABTCF_gdprApplies": 1,
          "IABTCF_PublisherCC": "AA",
          "IABTCF_PurposeOneTreatment": 0,
          "IABTCF_UseNonStandardTexts": 0,
          "IABTCF_TCString": "CPz_s0APz_s0AGXABBENAWEAAAAAAAAAAAAAAAAAAAAA.IAEAEAAA",
          "IABTCF_VendorConsents": "",
          "IABTCF_VendorLegitimateInterests": "",
          "IABTCF_PurposeConsents": "000000000000000000000000",
          "IABTCF_PurposeLegitimateInterests": "000000000000000000000000",
          "IABTCF_SpecialFeaturesOptIns": "000000000000",
          "IABTCF_PublisherConsent": null,
          "IABTCF_PublisherLegitimateInterests": null,
          "IABTCF_PublisherCustomPurposesConsents": null,
          "IABTCF_PublisherCustomPurposesLegitimateInterests": null,
          "IABTCF_AddtlConsent": "1~"
        }
      }
    }

PATCH Preferences with AC strings

Let's take the accept-all fides string from the previous request and pass that in to save preferences
PATCH http://localhost:8080/api/v1/privacy-preferences

{
  "browser_identity": {
    "fides_user_device_id": "d81b51c4-10b1-4fc5-b283-1bb5fe3659d5"
  },
  "fides_string": "CPz10AAPz10AAGXABBENAWEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA,1~100"
}

Both the gvl.8 and the gacp.100 vendors show up in the Vendor Consent section:

System gacp.100 was pulled off of the AC string and saved as a Vendor Consent. gvl.8 was saved per previous releases.

"vendor_consent_preferences": [
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": "gvl.8",
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_05f2e200-caa3-4ada-8a51-1c9a8d9bbbf4",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_aa7910f5-75e6-46e4-8906-23785f33f693"
    },
    {
      "purpose_consent": null,
      "purpose_legitimate_interests": null,
      "special_purpose": null,
      "vendor_consent": "gacp.100",
      "vendor_legitimate_interests": null,
      "feature": null,
      "special_feature": null,
      "system_consent": null,
      "system_legitimate_interests": null,
      "id": "cur_1a7ab22b-832a-46bf-831a-4d0000e5b309",
      "preference": "opt_in",
      "privacy_notice_history": null,
      "privacy_preference_history_id": "pri_de1201be-154c-494d-9396-67c695f67dd7"
    }
  ]

The AC string shows up in the fides mobile data in the response under IABTCF_AddtlConsent:

"fides_mobile_data": {
    "IABTCF_CmpSdkID": 407,
    "IABTCF_CmpSdkVersion": 1,
    "IABTCF_PolicyVersion": 4,
    "IABTCF_gdprApplies": 1,
    "IABTCF_PublisherCC": "AA",
    "IABTCF_PurposeOneTreatment": 0,
    "IABTCF_UseNonStandardTexts": 0,
    "IABTCF_TCString": "CPz10AAPz10AAGXABBENAWEAALAAAEOAAAAAAEAEACACAAAA.IAEAEAAA",
    "IABTCF_VendorConsents": "00000001",
    "IABTCF_VendorLegitimateInterests": "00000001",
    "IABTCF_PurposeConsents": "101100000000000000000000",
    "IABTCF_PurposeLegitimateInterests": "010000111000000000000000",
    "IABTCF_SpecialFeaturesOptIns": "000000000000",
    "IABTCF_PublisherConsent": null,
    "IABTCF_PublisherLegitimateInterests": null,
    "IABTCF_PublisherCustomPurposesConsents": null,
    "IABTCF_PublisherCustomPurposesLegitimateInterests": null,
    "IABTCF_AddtlConsent": "1~100"
  }

Passing in reject all string opts out of any AC vendors:

curl -X 'PATCH' \
  'http://localhost:8080/api/v1/privacy-preferences' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "browser_identity": {
    "fides_user_device_id": "d81b51c4-10b1-4fc5-b283-1bb5fe3659d5"
  },
  "fides_string": "CPz_s0APz_s0AGXABBENAWEAAAAAAAAAAAAAAAAAAAAA.IAEAEAAA,1~"

}'

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

…his to IABTCF_AddtlConsent in TC mobile data. Have the accept all and reject all fides strings be concatenations of tc and ac strings where applicable. - Adds ac_string to the TCModel which is an internal construct because it makes sense to build that at the same time we build this model.

- When passing in a fides string, separately split off a TC string and an AC string.

cypress · 2023-10-18T12:45:50Z

Passing run #4779 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `e098697` into `4f75d75`...
Project: fides	Commit: `b07a3539dd ℹ️`
Status: Passed	Duration: 00:53 💡
Started: Oct 24, 2023 3:36 PM	Ended: Oct 24, 2023 3:36 PM

Review all test suite changes for PR #4295 ↗︎

codecov · 2023-10-18T13:23:39Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (4f75d75) 87.76% compared to head (e098697) 87.81%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4295      +/-   ##
==========================================
+ Coverage   87.76%   87.81%   +0.05%     
==========================================
  Files         331      333       +2     
  Lines       20946    21058     +112     
  Branches     2718     2740      +22     
==========================================
+ Hits        18383    18493     +110     
- Misses       2096     2097       +1     
- Partials      467      468       +1

Files	Coverage Δ
...i/api/v1/endpoints/privacy_preference_endpoints.py	`98.29% <100.00%> (+0.06%)`	⬆️
src/fides/api/common_exceptions.py	`91.48% <100.00%> (ø)`
src/fides/api/schemas/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/experience_meta.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/fides_string.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/tc_model.py	`95.03% <100.00%> (+0.07%)`	⬆️
src/fides/api/util/tcf/tcf_experience_contents.py	`94.37% <100.00%> (+0.13%)`	⬆️
src/fides/config/consent_settings.py	`100.00% <100.00%> (ø)`
src/fides/api/util/tcf/tc_mobile_data.py	`92.30% <90.00%> (+0.87%)`	⬆️
src/fides/api/util/tcf/ac_string.py	`96.66% <96.66%> (ø)`
... and 1 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…tring is in an unexpected format, not just the TC string. Update adding an `ac_vendor_consents` section to the TCModel to mirror what is added for the different sections to build the TC string, instead of adding the AC string to that model directly.

…t. Update doc strings for clarity.

src/fides/api/util/tcf/experience_meta.py

src/fides/api/util/tcf/tc_model.py

src/fides/api/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/util/tcf/ac_string.py

pattisdr · 2023-10-18T21:51:40Z

@NevilleS matched this against the original AC, realized I overlooked If Google AC Mode is enabled. Do you want this to be a separate environment variable or can this just be derived from a combination of TCF being enabled + AC systems in the datamap?

EDIT: Yes, we need separate AC mode setting

pattisdr · 2023-10-19T13:14:02Z

Ah! I need to make this change, it's allowed in my implementation:

#4263

We can reject ,AC, that's invalid because the core TC string is always needed for a complete signal

EDIT: Done!

…first core TC string portion is required for a complete signal

- Allow AC strings in the format "1~" - Build reject-all strings in the format "1~" instead of None - Accept all strings where there are no AC systems are in the format "1~"

…ides string supplied.

…ns with other legal bases beyond Consent have been defined. This is technically possible but unlikely, but we don't want these to show up in the Experience. Specifically, Google doesn't allow AC vendors to claim legint as their basis. Update base privacy declarations filter for Experience to look for 3 things: 1) Privacy Declarations with valid TCF Data Use AND Consent legal basis (will pick up AC systems that match this criteria if applicable) 2) Privacy Declarations with valid TCF Data Use AND Legitimate Interests legal basis AND NOT AC Systems. 3) AC Systems without Privacy Declarations This also fixes a potential legitimate bug where AC Systems with a TCF purpose with a Legitimate Interests Legal Basis defined could technically show up under the Purpose Consents section. This also means, AC Systems with Privacy Declarations with any other legal basis don't show up, or Privacy Declarations specifically defined with non TCF purposes only don't show up.

- Throws error if attempting to save preferences against AC string unless AC mode enabled - New: Restrict AC systems from experience altogether unless AC mode enabled - New: Throws error if attempting to save preferences against with a TC string or against TCF preferences directly unless TC mode enabled (previously, ignored, but saved nothing) - Only build contents for IABTCF_AddtlConsent if AC mode enabled - Only include AC string in accept all and reject all strings if AC mode enabled

…ta format we get an error.

.fides/db_dataset.yml

.fides/fides.toml

src/fides/api/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/util/tcf/tcf_experience_contents.py

src/fides/api/util/tcf/tc_mobile_data.py

adamsachs

looks great @pattisdr, great work! really just some nits, but i think the functionality is just about good to go from my POV and it's also an easy-to-follow implementation!

src/fides/api/util/tcf/ac_string.py

src/fides/api/util/tcf/experience_meta.py

src/fides/api/util/tcf/ac_string.py

src/fides/api/util/tcf/tcf_experience_contents.py

src/fides/config/consent_settings.py

tests/ops/util/test_ac_string.py

pattisdr · 2023-10-23T22:38:08Z

🙏 Thanks so much for your review here @adamsachs, it's my bigger one and unblocks several other pieces. I'll get this updated and ready to go!

- Add validation to enforce that TCF mode must be enabled for AC mode to work - Split fides_string pieces into their own separate file - Remove TCF enabled check when building experience meta

pattisdr added 4 commits October 18, 2023 07:11

Allow saving preferences from an AC string.

8454197

- When passing in a fides string, separately split off a TC string and an AC string.

Update test now that fixture has reduced the number of features.

50fd44e

Adjust failing tests and newly named functions.

aa1da1c

pattisdr added 3 commits October 18, 2023 15:14

Update changelog

1452010

Update code comments

a01966d

pattisdr changed the title ~~Build Backend AC Strings~~ Build and Accept Backend AC Strings Oct 18, 2023

Throw additional errors if the fides_string is in an unexpected forma…

d3b7223

…t. Update doc strings for clarity.

pattisdr commented Oct 18, 2023

View reviewed changes

pattisdr requested review from allisonking and adamsachs October 18, 2023 21:44

pattisdr added 7 commits October 19, 2023 10:37

Prevent fides string being passed in in this format: ",1~1.100". The …

bbc556b

…first core TC string portion is required for a complete signal

AC String adjustments for 1~ instead of null.

7dcd851

- Allow AC strings in the format "1~" - Build reject-all strings in the format "1~" instead of None - Accept all strings where there are no AC systems are in the format "1~"

Revert making convert_fides_str_to_mobile_data throw an error if no f…

2392430

…ides string supplied.

Add some further testing around AC string edge cases.

05fec09

Add test that if you pass in just an ac string to decode to mobile da…

90c776c

…ta format we get an error.

pattisdr commented Oct 21, 2023

View reviewed changes

Merge branch 'main' into PROD-1167_backend_ac_string

9ff9d2d

adamsachs approved these changes Oct 23, 2023

View reviewed changes

pattisdr added 3 commits October 24, 2023 09:36

Respond to CR comments

da2f2e7

- Add validation to enforce that TCF mode must be enabled for AC mode to work - Split fides_string pieces into their own separate file - Remove TCF enabled check when building experience meta

Merge main, conflicts CHANGELOG

f313cff

Update error message.

e098697

pattisdr merged commit 0956cfe into main Oct 24, 2023
38 of 40 checks passed

pattisdr deleted the PROD-1167_backend_ac_string branch October 24, 2023 16:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build and Accept Backend AC Strings #4295

Build and Accept Backend AC Strings #4295

pattisdr commented Oct 18, 2023 •

edited

Loading

cypress bot commented Oct 18, 2023 •

edited

Loading

codecov bot commented Oct 18, 2023 •

edited

Loading

pattisdr commented Oct 18, 2023 •

edited

Loading

pattisdr commented Oct 19, 2023 •

edited

Loading

adamsachs left a comment

pattisdr commented Oct 23, 2023

Build and Accept Backend AC Strings #4295

Build and Accept Backend AC Strings #4295

Conversation

pattisdr commented Oct 18, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

GET Experiences with AC String

PATCH Preferences with AC strings

Pre-Merge Checklist

cypress bot commented Oct 18, 2023 • edited Loading

Passing run #4779 ↗︎

Review all test suite changes for PR #4295 ↗︎

codecov bot commented Oct 18, 2023 • edited Loading

Codecov Report

pattisdr commented Oct 18, 2023 • edited Loading

pattisdr commented Oct 19, 2023 • edited Loading

adamsachs left a comment

Choose a reason for hiding this comment

pattisdr commented Oct 23, 2023

pattisdr commented Oct 18, 2023 •

edited

Loading

cypress bot commented Oct 18, 2023 •

edited

Loading

codecov bot commented Oct 18, 2023 •

edited

Loading

pattisdr commented Oct 18, 2023 •

edited

Loading

pattisdr commented Oct 19, 2023 •

edited

Loading