Update how we Save and Propagate Privacy Preferences [#3013] #3016

pattisdr · 2023-04-10T01:29:13Z

❗ Contains migration; check downrev before merge.
Closes #3013

Code Changes

Add two new models: CurrentPrivacyPreference (which is similar to the Consent table to which we will soon stop writing) and PrivacyPreferenceHistory.
- Every time a user saves preferences, we will create a PrivacyPreferenceHistory record for each preference which has a FK to a PrivacyNoticeHistory record. We link to the historical notice as notices are subject to change over time.
- We then upsert CurrentPrivacyPreferences for the given user in a separate table for the given PrivacyNotice
- A PrivacyRequest is then created, and each PrivacyPreferenceHistory record that was just created is given a FK to thatPrivacyRequest. Request propagation will retrieve these PrivacyPreferenceHistory records, again linking to the historical record here, as the CurrentPrivacyPreference record is subject to change.
- PrivacyPreferenceHistory is largely useful for propagation as well as for consent reporting. CurrentPrivacyPreference contains the user's latest saved preference for a given notice. `CurrentPrivacyPreference is separated for ease of querying and display of current preferences.
Largely leave the old workflow untouched for now (where we save consent preferences with respect to data uses defined in the privacy center). However, start queueing a privacy request each time preferences are saved even if no preferences are executable.
Add new endpoint consent_request_verify_for_privacy_preferences which allows you to verify your identity to retrieve your current current preferences (returns records from the CurrentPrivacyPreference table). This is similar to the existing endpoint that instead returns records from the Consent table.

POST {{host}}/consent-request/{{consent_request_id}}/verify-for-privacy-preferences

{
 "code": "{{verification_code}}"
}

Add new endpoint save_privacy_preferences which allows you to save privacy preferences with respect to privacy notice history records. This creates PrivacyPreferenceHistory records, upserts CurrentPrivacyPreferences records, creates a PrivacyRequest to propagate, then returns the CurrentPrivacyPreferences that were in the request body.

PATCH {{host}}/consent-request/{{consent_request_id}}/privacy-preferences

{
   "browser_identity": {
       "ga_client_id": "UA-XXXXXXXXX",
       "ljt_readerID": "test_sovrn_id"
   },
   "code": "{{verification_code}}",
   "preferences": [{
       "privacy_notice_history_id": "{{privacy_notice_history_id}}",
       "preference": "opt_out"
   }],
   "request_origin": "privacy_center",
   "url_recorded": "example.com/privacy_center",
   "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2)...",
   "user_geography": "us_ca"
}

Return the privacy notice history id alongside the endpoints that get and create privacy notices so this is available when setting privacy preferences respect to those historical notices.
Update the consent email template to handle both old (consent) and new style (privacy preferences) being fed into it.
Start adding skipped execution logs in both the consent and erasure email connectors when we opt not to send an email (for many reasons) for better record keeping
Add shared logic to both consent email send and saas connector run_consent_request to be more discerning about whether the current preferences are applicable to the given system
Adjust so saas connectors are only firing at most one series of requests: right now, either the entire opt in or the entire opt out series. This was achieved previously by restricting so only one executable preference would be on the connector.

Steps to Confirm

See Testing Notes Here.
Big picture, the old workflow for saving consent should still work largely the same. The new workflow should work too although it is partially API-only driven at the moment.

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

Description Of Changes

Primary changes here:

Start saving privacy preferences with respect to the "privacy notices"
- Start storing a snapshot of the privacy preferences that were changed every single time they are saved for reporting purposes
- Separately, have a new table that stores the most recently saved privacy preference for a given notice for ease of displaying a user's current preferences
Expose endpoints to retrieve and save the new privacy preferences after verifying your identity (similar to the consent counterparts that are soon to be obsolete)
For both the old workflow and the new workflow introduced here, start creating privacy requests every single time preferences are saved.
Start adding restrictions as to whether we fire requests for a particular consent saas connector or send consent emails: in particular, start comparing privacy notice history data uses and system data uses. the privacy notice history must be enforceable system wide.
Start adding skipped execution logs for a connector when we don't fire a request (instead of completed logs) and start adding skipped execution logs instead of no execution logs when we opt to not send an email for a connector for clearer record-keeping
Note that the old workflow where we save consent with respect to data uses defined in the privacy center config.json is still preserved here for the time being.

…ence for storing privacy preferences with respect to the new privacy notices. - Every single time a "privacy preference" is saved, create a PrivacyPreferenceHistory record for consent reporting purposes. Then, upsert the CurrentPrivacyPreference record for a given provided identity and privacy notice. - Create a privacy request each time to propagate preferences where applicable. Link the PrivacyPreferenceHistory records to that PrivacyRequest, not the CurrentPrivacyPreference record, as the latter is subject to change. - Add two new endpoints, to verify a user's identity and then get their CurrentPrivacyPreferences as well as to verify a user's identity and then save privacy preferences. These are similar to the existing Consent endpoints, except they interface with these new tables. - Add some logic for saas connectors to be more discerning as to whether we should propagate a specific preference for a given system. - When running consent requests on a saas connector, create a "skipped" log instead of a "completed" log if we don't fire a request to the given service. - Return a privacy notice history id when getting privacy notices. The privacy preferences should be saved with respect to the historical id.

…ent util should_opt_in_to_service.

…w - saving privacy preferences for privacy notices. - Add some saas fixtures for tests we know will fail before we try to make a connection so we don't need secrets saved. - Test old workflow on whether we will opt into a service.

…acy preferences: - Adjust definition for whether a consent email connector needs to send an email to factor in new workflow (privacyrequest.privacy_preferences), but still support the old workflow (privacyrequest.consent_preferences). Share the same code that we do for the consent email connectors in this calculation. - Adjust batch email send to look for both relevant privacy preferences to the given connector (new workflow) and consent preferences (old workflow) - Adjust the consent email template to include consent preferences (old workflow) and privacy preferences (new workflow) - Add skipped execution logs for consent email connectors if we skip sending. - Add skipped execution logs for erasure email connectors if we skip sending for consistency. - Adjust the email template for consent to be able to work for old consent preferences and new-style privacy preferences.

…urn current privacy preferences and another that saves preferences and returns the current privacy preferences that were changed.

…y preferences - and only allow 50 privacy preferences to be saved at once.

…th respect to data uses) regardless of whether there are executable preferences to propagate.

cypress · 2023-04-10T01:42:29Z

Passing run #1278 ↗︎

0	3	0	0	0

Details:

Merge `b8ff8ab` into `73559d6`...
Project: fides	Commit: `f00e444053 ℹ️`
Status: Passed	Duration: 00:42 💡
Started: Apr 11, 2023 6:29 PM	Ended: Apr 11, 2023 6:29 PM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

…versal analytics configs that don't have secrets.

codecov · 2023-04-10T02:41:38Z

Codecov Report

Patch coverage: 99.69% and project coverage change: +0.40 🎉

Comparison is base (73559d6) 86.92% compared to head (b8ff8ab) 87.32%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3016      +/-   ##
==========================================
+ Coverage   86.92%   87.32%   +0.40%     
==========================================
  Files         303      307       +4     
  Lines       17322    17592     +270     
  Branches     2230     2260      +30     
==========================================
+ Hits        15057    15363     +306     
+ Misses       1849     1815      -34     
+ Partials      416      414       -2

Impacted Files	Coverage Δ
src/fides/api/ctl/sql_models.py	`98.20% <91.66%> (+0.04%)`	⬆️
src/fides/api/ops/api/v1/api.py	`100.00% <100.00%> (ø)`
.../ops/api/v1/endpoints/consent_request_endpoints.py	`87.75% <100.00%> (-0.92%)`	⬇️
...s/api/v1/endpoints/privacy_preference_endpoints.py	`100.00% <100.00%> (ø)`
.../ops/api/v1/endpoints/privacy_request_endpoints.py	`89.88% <100.00%> (+1.03%)`	⬆️
src/fides/api/ops/api/v1/urn_registry.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/common_exceptions.py	`96.77% <100.00%> (+0.10%)`	⬆️
src/fides/api/ops/db/base.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/models/connectionconfig.py	`95.83% <100.00%> (+0.04%)`	⬆️
src/fides/api/ops/models/privacy_notice.py	`100.00% <100.00%> (ø)`
... and 13 more

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

…test verifying that if privacy request creation fails, privacy preferences are still saved.

src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/api/v1/endpoints/privacy_request_endpoints.py

src/fides/api/ops/email_templates/templates/consent_request_email_fulfillment.html

src/fides/api/ops/service/connectors/saas_connector.py

src/fides/api/ops/service/privacy_request/request_runner_service.py

src/fides/api/ops/task/graph_task.py

src/fides/api/ops/util/consent_util.py

pattisdr · 2023-04-10T12:21:35Z

@adamsachs here is the rewrite of #2991 based on discussions Friday. Would you have time to review? This PR looks bigger than it is, but a lot of the changes are new tests.

@seanpreston I'm tagging you here because this new direction we're going with "privacy preference" consent reporting (new tables created/new endpoints) likely won't solve your issue of surfacing the old-style consent history.

@NevilleS tagged you if you were interested in seeing where this evolved since Friday - I did end up adding an additional table for current preferences (similar to Consent) that wasn't specified there.

adamsachs

i'm not through this fully yet, but wanted to post an incremental review before i end my day here, so that - if you want - you can start to address any comments (none of them major so far!) in the meantime. i haven't done any testing yet, or looked over the automated test updates, so that's one big area i need to get to, and some additional code comments may come from that, as i get a bit more hands-on. i've also not dug into the email connector changes - just because that felt like a somewhat independent chunk that i could wall off for the time being - but i will look to review those ASAP tomorrow.

what i have looked over - which i think gets to the core of the changes in the PR - generally looks really good to me! i don't see any major issues so far.

there are two categories of optimizations that i wanted to entertain: one questioning whether we should be indexing some more db columns, and another about whether it'd be possible to DRY up some of the matching we do between privacy notices and systems. the former is obviously something we should look into now (though i very well may be misguided in my points), while the latter doesn't need to be addressed in this PR by any means. there are a few other minor points throughout, but they should be easily resolved or dismissed.

i'll keep on looking at this as fast as i can, but it's looking very promising!

src/fides/api/ctl/sql_models.py

src/fides/api/ops/models/privacy_preference.py

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/service/connectors/saas_connector.py

src/fides/api/ops/util/consent_util.py

- Dry up getting system data uses. - Remove unnecessary nullable=True from privacy preference models. - Add hashed_email and hashed_phone_number which are indexed. - Index CurrentPrivacyPreference.provided_identity_id and have remove the join to ProvidedIdentity in consent_request_verify_for_privacy_preferences - Overhaul the verify_privacy_notice_and_historical_records docstring - Dry up some of the code checking if a privacy notice history record applies to a system - Add method PrivacyNoticeBase.applies_to_system that can be used for privacy notices or history records - Remove unused schema intended for consent reporting - am detailing a different one in another PR.

adamsachs

ok, went through the latest changes (look great generally!) and wanted to publish another round of comments just so you can get that feedback ASAP.

getting to testing now...

src/fides/api/ops/models/privacy_notice.py

src/fides/api/ops/util/consent_util.py

src/fides/api/ops/models/privacy_preference.py

src/fides/api/ops/models/privacy_notice.py

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/models/privacy_preference.py

docs/fides/docs/development/postman/Fides.postman_collection.json

tests/ctl/core/test_system.py

tests/ops/models/test_privacy_preference.py

tests/ops/service/connectors/test_saas_connector.py

tests/ops/util/test_consent_util.py

…quirements. - Add missing index to CurrentPrivacyPreference.provided_identity_id - Add new privacy_notice/privacy_notice_history internal description field. Remove obsolete displayed_in_banner and displayed_in_privacy_modal in favor of displayed_in_overlay and displayed_in_api. - Remove bad trailing comma from postman collection - Add new "not_applicable" to enforcement enum - Adjust Request Origin enum to be privacy_center, overlay, and API to match corresponding changes to Privacy Notice - Fix copy/paste error on test name. - Remove docstring with future fields we need to add

pattisdr · 2023-04-11T18:51:33Z

OK I think I've addressed your comments @adamsachs and have completed another round of manual testing. A few new fields were added or adjusted based on changing requirements. Test failures seem to match what's on main.

adamsachs

@pattisdr i think this is ready to go from my POV!

thanks for the diligence in addressing all the comments. and 👏 for turning around a huge enhancement in such quick time, in a clean and easy-to-follow manner!

pattisdr · 2023-04-11T22:20:41Z

Likewise @adamsachs, appreciate the detailed, prompt review! ⭐

pattisdr added 8 commits April 9, 2023 11:59

Add detailed tests for SaasConnector.run_consent_preferences and cons…

3bae644

…ent util should_opt_in_to_service.

Update postman collection with two endpoints that verify code and ret…

9cdf461

…urn current privacy preferences and another that saves preferences and returns the current privacy preferences that were changed.

Add pagination to the endpoint where we verify identity to get privac…

2ad827f

…y preferences - and only allow 50 privacy preferences to be saved at once.

Rename tests.

e3a864a

Create privacy request for old workflow (consent preferences saved wi…

6f87f98

…th respect to data uses) regardless of whether there are executable preferences to propagate.

pattisdr added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Apr 10, 2023

Add new dataset annotations, don't use secrets fixtures in google/uni…

60d8b30

…versal analytics configs that don't have secrets.

pattisdr added 2 commits April 9, 2023 22:11

Remove unused try/catch because the not found is caught earlier. Add …

4250e5e

…test verifying that if privacy request creation fails, privacy preferences are still saved.

Add test where saas connector run_consent_request mock executes.

083c8fd

pattisdr marked this pull request as ready for review April 10, 2023 04:08

pattisdr changed the title ~~[WIP] Update how we Save and Propagate Privacy Preferences [WIP] [#3013]~~ Update how we Save and Propagate Privacy Preferences [#3013] Apr 10, 2023

pattisdr commented Apr 10, 2023

View reviewed changes

pattisdr requested review from adamsachs and seanpreston April 10, 2023 12:18

pattisdr requested a review from NevilleS April 10, 2023 17:41

Call defined add_skipped_log method.

556c4ed

adamsachs reviewed Apr 10, 2023

View reviewed changes