[Backend] Track Privacy Experience that surfaced Notices/ Pull data from request headers #3311

pattisdr · 2023-05-15T23:37:14Z

❗ Contains migration
❗ Dependent on #3292

Closes #3133
Closes #3196

Description Of Changes

Start tracking additional fields when saving consent preferences to demonstrate compliance and move some of the existing fields from the request body to extract from request headers.

Code Changes

Start tracking new fields:

PrivacyPreferenceHistory.anonymized_ip_address which is really a truncated version of the ip address and stored encrypted
PrivacyPreferenceHistory.method - The method of consent preference in a modal (buttons, individual notices)
PrivacyPreferenceHistory.privacy_experience_config_history_id - the particular version of the Config that surfaced the notice (contains the copy)
PrivacyPreferenceHistory.privacy_experience_history_id - the particular Experience - contains the region, type, and delivery mechanism
Starts surfacing these new fields in the historical report

Start pulling some fields off of request headers or experience themselves:

request_origin (privacy center/overlay now comes off of the experience)
user_agent now comes off of request headers
url_recorded is now retrieved from the Referer header
privacy_experience_config_history_id is pulled off of the experience if applicable

Bug fix:

Fixes saving preferences with for users under a fides user device id - bug where uniqueness constraint was being violated

Steps to Confirm

In the postman collection

Save some notices POST {{host}}/privacy-notice
Save a preference for those notices PATCH {{host}}/privacy-preferences
- Verify method, experience_config_history_id and privacy_experience_history can be passed in now
Get historical privacy preferences GET {{host}}/historical-privacy-preferences
- Verify truncated ip address is in the response

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

cypress · 2023-05-15T23:54:04Z

Passing run #2046 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `4cfba74` into `5303975`...
Project: fides	Commit: `401cec22b1 ℹ️`
Status: Passed	Duration: 00:44 💡
Started: May 18, 2023 3:50 AM	Ended: May 18, 2023 3:51 AM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

codecov · 2023-05-16T00:13:54Z

Codecov Report

Patch coverage: 96.61% and project coverage change: +0.02 🎉

Comparison is base (5303975) 87.06% compared to head (4cfba74) 87.09%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3311      +/-   ##
==========================================
+ Coverage   87.06%   87.09%   +0.02%     
==========================================
  Files         309      309              
  Lines       18847    18898      +51     
  Branches     2467     2472       +5     
==========================================
+ Hits        16410    16459      +49     
- Misses       1996     1997       +1     
- Partials      441      442       +1

Impacted Files	Coverage Δ
...i/ops/api/v1/endpoints/privacy_notice_endpoints.py	`100.00% <ø> (ø)`
src/fides/api/ops/models/privacy_experience.py	`100.00% <ø> (ø)`
...s/api/v1/endpoints/privacy_preference_endpoints.py	`98.63% <94.11%> (-1.37%)`	⬇️
...s/api/v1/endpoints/privacy_experience_endpoints.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/models/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/schemas/privacy_preference.py	`100.00% <100.00%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

allisonking

okay I think I'm following, so the flow from the fides-js side would be:

query for /privacy-experience which will give us an obj with the experience config and all notices. it will also contain privacy_experience_history_id and within the experience config, experience_config_history_id
When the user saves, we should send a payload that includes

{
  method: 'button' | 'individual notice' | 'gpc'  // any string, but maybe these cover them for now?
  privacy_experience_history_id: 'string from /privacy-experience'
  privacy_experience_config_history_id: "string from /privacy-experience's nested experience_config obj"
}

and IP address the backend will derive itself?

pattisdr · 2023-05-17T16:35:02Z

yes @allisonking I think I can get the IP from the FastAPI request.client.host. I'm also now getting the user agent and url_recorded from headers. I also don't need the privacy_experience_config_history_id anymore, just the privacy_experience_history_id and I'll get the config from that.

I also am assuming I'm getting the fides user device id under browser_identity as well as a list of preferences with each notice history, does that line up?

{
   "browser_identity": {
       "ga_client_id": "UA-XXXXXXXXX",
       "ljt_readerID": "test_sovrn_id",
       "fides_user_device_id": "{{fides_user_device_id}}"
   },
   "preferences": [{
       "privacy_notice_history_id": "{{privacy_notice_history_id}}",
       "preference": "opt_out"
   }],
   "user_geography": "us_ca",
   "privacy_experience_history_id": "{{privacy_experience_history_id}}",
   "method": "button"
}

allisonking · 2023-05-17T17:08:39Z

got it, thanks for the payload, that's helpful! I think I'm a little behind on this, but just to state my understanding:

Currently, the privacy center sends a PATCH to /consent-request/{id}/preferences. This endpoint is being deprecated
We will switch, with the migration to start using privacy notices, the privacy center to PATCH to /consent-request/{id}/privacy-preferences instead and that endpoint will expect the payload you provided above
And fides-js will do the same thing as step 2 here (since it was never using the deprecated endpoint)

does that sound right?

pattisdr · 2023-05-17T17:29:57Z

@allisonking yes! both payloads for both the privacy center and the components for saving preferences are intended to be identical, although the privacy center additionally may have a code in it, if it's the verified identity workflow

… experience config history and experience history records that displayed the notices to which the user consented. - Move request_origin, url_recorded, and user_agent from being in the request body. Instead pull these from the request headers or the privacy experience where applicable. - Also pull the ip address and mask before saving.

…80 of the 128 bits to 0 for ipv6. Encrypt this field. Also add "method" to privacy preferences"

…tion.

…, as well as the truncated ip address and the method.

…. Just pass in the privacy experience history id, and we can extract the config history from there if applicable.

- Remove unused Request dependency - Remove unnecessary caching of notices when updating experience. - Stringify experience-related FK's on PrivacyPreferenceHistory to avoid circular dependencies now that we have a method in PrivacyExperience that needs to surface CurrentPrivacyPreferences - Add an enum for method - button, individual notices

pattisdr marked this pull request as ready for review May 17, 2023 01:06

pattisdr requested review from eastandwestwind and allisonking May 17, 2023 14:24

allisonking approved these changes May 17, 2023

View reviewed changes

Base automatically changed from fides_3193_syncing_privacy_notices_and_experiences to main May 17, 2023 23:44

pattisdr added 7 commits May 17, 2023 20:53

Fix bug if ids are supplied as empty strings.

c36de1c

Update anonymize_ip_address to set last octet to 0 for ipv4 and last …

b378958

…80 of the 128 bits to 0 for ipv6. Encrypt this field. Also add "method" to privacy preferences"

Add new dataset annotations and remove attributes from postman collec…

6a6b825

…tion.

Expose privacy experience and config details in the historical report…

53a9978

…, as well as the truncated ip address and the method.

Reduce passing in the config history directly when saving preferences…

ba6d804

…. Just pass in the privacy experience history id, and we can extract the config history from there if applicable.

pattisdr force-pushed the fides_3133_user_data_on_privacy_preferences branch from bb66dcd to 291da6d Compare May 18, 2023 03:00

Add upfront validation on consent method.

4cfba74

pattisdr merged commit 1407f13 into main May 18, 2023

pattisdr deleted the fides_3133_user_data_on_privacy_preferences branch May 18, 2023 04:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backend] Track Privacy Experience that surfaced Notices/ Pull data from request headers #3311

[Backend] Track Privacy Experience that surfaced Notices/ Pull data from request headers #3311

pattisdr commented May 15, 2023 •

edited

Loading

cypress bot commented May 15, 2023 •

edited

Loading

codecov bot commented May 16, 2023 •

edited

Loading

allisonking left a comment

pattisdr commented May 17, 2023 •

edited

Loading

allisonking commented May 17, 2023

pattisdr commented May 17, 2023 •

edited

Loading

[Backend] Track Privacy Experience that surfaced Notices/ Pull data from request headers #3311

[Backend] Track Privacy Experience that surfaced Notices/ Pull data from request headers #3311

Conversation

pattisdr commented May 15, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented May 15, 2023 • edited Loading

Passing run #2046 ↗︎

codecov bot commented May 16, 2023 • edited Loading

Codecov Report

allisonking left a comment

Choose a reason for hiding this comment

pattisdr commented May 17, 2023 • edited Loading

allisonking commented May 17, 2023

pattisdr commented May 17, 2023 • edited Loading

pattisdr commented May 15, 2023 •

edited

Loading

cypress bot commented May 15, 2023 •

edited

Loading

codecov bot commented May 16, 2023 •

edited

Loading

pattisdr commented May 17, 2023 •

edited

Loading

pattisdr commented May 17, 2023 •

edited

Loading