PROD-1869 - Support config for pre-response webhooks

[eastandwestwind]

Closes https://ethyca.atlassian.net/browse/PROD-1869

Description Of Changes

Backend infra to support future pre-response hook functionality.

This PR:

• allows configuration of pre-response hooks
• creates the necessary DB tables for storage of pre-response hook configs and replies from pre-response hooks

Code Changes

• Adds 2 new db tables
• Adds CRUD operations for config of pre-response hooks
• Adds unit tests for CRUD ops

Steps to Confirm

• Full unit test coverage has been created for the new pre_approval_webhook_endpoints.py, but if you wish to manually test, confirm all the new endpoints are successful.
• First you'll need to create a valid ConnectionConfig. PATCH /api/v1/connection with:

[
   {
      "name":"my_connection_config",
      "key":"my_connection_config",
      "connection_type":"https",
      "access":"read",
      "secrets":{
         "url":"http://example.com",
         "authorization":"test_authorization"
      }
   }
]

• Now you can create any number of pre-approval webhook that is linked to this connection config. PUT /api/v1/dsr/webhook/pre_approval with:

[
   {
      "connection_config_key":"my_connection_config",
      "name":"My webhook",
      "key":"my_webhook"
   }
]

• GET /api/v1/dsr/webhook/pre_approval should return this item
• GET /api/v1/dsr/webhook/pre_approval/my_webhook should also return this item
• You should be able to edit your pre approval webhook. PATCH /api/v1/dsr/webhook/pre_approval/my_webhook with, for example, {"name": "Some other name"}
• GET /api/v1/dsr/webhook/pre_approval/my_webhook should reflect this change
• You should be able to delete your webhook. DELETE /api/v1/dsr/webhook/pre_approval/my_webhook should be successful, and GET /api/v1/dsr/webhook/pre_approval should not return that webook at all.

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Apr 19, 2024 4:47pm

[cypress]

Passing run #7341 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge 129f6b6 into ab76b3c...
Project: fides	Commit: f38b7ffad6 ℹ️
Status: Passed	Duration: 00:35 💡
Started: Apr 19, 2024 4:58 PM	Ended: Apr 19, 2024 4:59 PM

Review all test suite changes for PR #4795 ↗︎

[pattisdr]

Starting review!

[pattisdr]

@eastandwestwind before I dig into your endpoints, passing it back early to get the relationships set up and the endpoints registered -

[codecov]

Codecov Report

Attention: Patch coverage is 95.93496% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 86.65%. Comparing base (ab76b3c) to head (129f6b6).

Files	Patch %	Lines
...api/v1/endpoints/pre_approval_webhook_endpoints.py	93.58%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4795      +/-   ##
==========================================
+ Coverage   86.27%   86.65%   +0.38%     
==========================================
  Files         339      342       +3     
  Lines       20100    20223     +123     
  Branches     2587     2596       +9     
==========================================
+ Hits        17341    17524     +183     
+ Misses       2293     2225      -68     
- Partials      466      474       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pattisdr]

Starting review!

[pattisdr]

@eastandwestwind endpoints looking good, mostly just some trivial copy/paste issues.

The main thing I see missing are model-level tests (like tests/ops/models/test_connectionconfig.py). They don't need to be large, because you're testing some of this through your endpoints, but I'm particularly looking for tests that create a PreApprovalWebhook and PreApprovalWebhookReply and assert their relationships on both sides. For example, assert PreApprovalWebhook.connection_config and ConnectionConfig.pre_approval_webhooks are the types you expect, same for PreApprovalWebhookReplay and the privacy request relationship.

[pattisdr]

I think we should remove uselist=False because a connection config can have multiple pre approval webhooks right? If uselist=False, this will just return one object.

[pattisdr]

Model-level tests are good for flushing these out -

[eastandwestwind]

Ahhh I see, that makes sense! I'll remove this

[eastandwestwind]

This means I likely should add uselist=False to the privacy_request field on my PreApprovalWebhookReply api model:

privacy_request = relationship(
        "PrivacyRequest", back_populates="pre_approval_webhook_reply", uselist=False  # type: ignore
    )

[pattisdr]

great @eastandwestwind thank you for taking the time to write these nice model tests. My last question is about the naming of the privacy request <> pre approval webhook reply relationship because it maybe should be plural and represented in the test

[pattisdr]

Great work @eastandwestwind

[eastandwestwind]

@pattisdr have you seen these codecov/path CI check failures before? Is this something I'll need to address via more unit tests?

[pattisdr]

Yes, I don't always aim for 100%, depending on the circumstance, you can look at the failures and see if there are any lines you think should be covered - I just try to be reasonable with test coverage -