PROD-2049: Implement sending emails via property-specific messaging templates

[eastandwestwind]

Closes PROD-2049

Description Of Changes

This change adds the core logic to determine both:

1. Whether or not to use property-specific messaging templates
2. Which messaging template to use when sending emails from Fides

We support property-specific messaging for the following message types

• PRIVACY_REQUEST_REVIEW_APPROVE
• PRIVACY_REQUEST_REVIEW_DENY
• PRIVACY_REQUEST_RECEIPT
• PRIVACY_REQUEST_COMPLETE_ACCESS
• PRIVACY_REQUEST_COMPLETE_DELETION
• SUBJECT_IDENTITY_VERIFICATION (both consent and privacy request)

Code Changes

• Adds new env var to track whether or not we enable property-specific messaging
• Adds property_id field to PrivacyRequest model
• Implements retrieving appropriate messaging template by property id if ENV var is enabled

Steps to Confirm

• With FIDES__NOTIFICATIONS__ENABLE_PROPERTY_SPECIFIC_MESSAGING disabled, confirm that no regressions exist with configuring email templates and sending out emails via Fides (consult the above list for email types to test)
• With FIDES__NOTIFICATIONS__ENABLE_PROPERTY_SPECIFIC_MESSAGING enabled, confirm that we can send property-specific emails. To do this, to the following:
• Configure your email messaging config (e.g. Mailgun) as you did before to enable sending emails from Fides
• Enable your property-specific emails. In the postgres container, run update messaging_template set is_enabled=true;

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
  • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated
• If there are any database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!

[codecov]

Codecov Report

Attention: Patch coverage is 68.23529% with 27 lines in your changes missing coverage. Please review.

Project coverage is 86.81%. Comparing base (ccb0b7c) to head (1283f3f).
Report is 1 commits behind head on main.

Files	Patch %	Lines
.../api/service/messaging/message_dispatch_service.py	63.41%	11 Missing and 4 partials ⚠️
.../api/api/v1/endpoints/privacy_request_endpoints.py	38.46%	6 Missing and 2 partials ⚠️
.../api/api/v1/endpoints/consent_request_endpoints.py	55.55%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4950      +/-   ##
==========================================
- Coverage   86.89%   86.81%   -0.09%     
==========================================
  Files         346      346              
  Lines       21408    21481      +73     
  Branches     2826     2843      +17     
==========================================
+ Hits        18603    18649      +46     
- Misses       2308     2329      +21     
- Partials      497      503       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[galvana]

Everything worked with FIDES__NOTIFICATIONS__ENABLE_PROPERTY_SPECIFIC_MESSAGING disabled but I ran into this serialization error when I set it to true and tried to use the default templates.

fides  |   File "/fides/src/fides/api/api/v1/endpoints/privacy_request_endpoints.py", line 1131, in verify_identification_code
fides  |     _send_privacy_request_receipt_message_to_user(
fides  |   File "/fides/src/fides/api/api/v1/endpoints/privacy_request_endpoints.py", line 257, in _send_privacy_request_receipt_message_to_user
fides  |     dispatch_message_task.apply_async(
fides  |   File "/opt/fides/lib/python3.10/site-packages/celery/app/task.py", line 564, in apply_async
fides  |     content_type, content_encoding, data = serialization.dumps(
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/serialization.py", line 219, in dumps
fides  |     with _reraise_errors(EncodeError):
fides  |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
fides  |     self.gen.throw(typ, value, traceback)
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/serialization.py", line 45, in _reraise_errors
fides  |     reraise(wrapper, wrapper(exc), sys.exc_info()[2])
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/exceptions.py", line 34, in reraise
fides  |     raise value.with_traceback(tb)
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/serialization.py", line 41, in _reraise_errors
fides  |     yield
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/serialization.py", line 220, in dumps
fides  |     payload = encoder(data)
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/utils/json.py", line 63, in dumps
fides  |     return _dumps(s, cls=cls, **dict(default_kwargs, **kwargs))
fides  |   File "/usr/local/lib/python3.10/json/__init__.py", line 238, in dumps
fides  |     **kw).encode(obj)
fides  |   File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
fides  |     chunks = self.iterencode(o, _one_shot=True)
fides  |   File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
fides  |     return _iterencode(o, 0)
fides  |   File "/opt/fides/lib/python3.10/site-packages/kombu/utils/json.py", line 47, in default
fides  |     return super().default(o)
fides  |   File "/usr/local/lib/python3.10/json/encoder.py", line 179, in default
fides  |     raise TypeError(f'Object of type {o.__class__.__name__} '
fides  | kombu.exceptions.EncodeError: Object of type MessagingTemplate is not JSON serializable

[galvana]

Overall this looks good! Just a few things.

1. I left a comment on the expected behavior between properties and privacy requests and I'd like to hear your thoughts.
2. I got an error when running with FIDES__NOTIFICATIONS__ENABLE_PROPERTY_SPECIFIC_MESSAGING=true, I left a root-level comment with the stacktrace
3. I had to enable the default templates in order to get messaging to work. Is it expected that they're set to disabled by default? I imagine we would want them enabled for OSS users?

[eastandwestwind]

Thanks so much for the review @galvana !

1. Updated as per your suggestion, including adding validation for valid property_id, if passed in to both consent and DSR requests
2. Fixed the error- turns out I can't pass in DB models to celery workers.
3. Yes, you'll need to enable the default templates first, I've updated the testing instructions to reflect this. We don't support property-specific messaging in OSS at all. This is only a Plus feature.

[galvana]

Thanks for making the changes @eastandwestwind!

Fixed the error- turns out I can't pass in DB models to celery workers.

Thanks for the fix here! I re-ran this scenario and it worked as expected

Yes, you'll need to enable the default templates first, I've updated the testing instructions to reflect this. We don't support property-specific messaging in OSS at all. This is only a Plus feature.

I was thinking of it wrong, this makes sense!

[cypress]

Passing run #8293 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

PROD-2049: Implement sending emails via property-specific messaging templates (#...
Project: fides	Commit: 0f4ea85bf5
Status: Passed	Duration: 00:34 💡
Started: Jun 13, 2024 1:48 PM	Ended: Jun 13, 2024 1:49 PM

Review all test suite changes for PR #4950 ↗︎