DynamoDB Connector - Consent Policy Spike

[Kelsey-Ethyca]

Description

Fides needs to be able to integrate with DynamoDB for the purposes of consent management (DSR support will come later).
Datastore consent management most commonly involves storing a list of consent records that the owner of the datastore can then query later on to ascertain what's permissible for user data. This ticket is to suggest a path to the storing of consent preferences dynamically within DynamoDB, and to build tickets + start implementing the feature once design is signed off.

Acceptance Criteria

• Fides can build queries to insert consent records into a DynamoDB collection
• A user can customise which DynamoDB collection the consent preferences are stored in

Questions

• How can we best extend the Fides dataset configurations to facilitate storing of consent records within connected datastores? (e.g. if somebody has connected Postgres, what do they need to add to their Postgres dataset to support consent records)

[pattisdr]

This is an interesting one. Not only do we need a new database connector, but that database connector needs to propagate consent preferences. We don't have this scenario covered at all yet. Saas Connectors have the ability to define consent endpoints but there's no parallel concept for database connectors.

The thing is that saas connectors don't really use a dataset/datasetconfig at all yet: the consent saas connectors do have a dummy dataset created for them, but the email saas connectors don't have a dataset/datasetconfig. Anything related to consent has not yet been needed to be defined on the dataset.

So if we are defining a dataset now, we need a way to specify which fields should be updated for consent, especially is this same dataset is shared w/ DSR's later. It's just a concept we don't have in the dataset yet.

Additionally, will need to do the typical figuring out how to connect to dynamo (ConnectionConfig/connection_secrets) and how to query dynamo (see QueryConfig), just like we did for setting up mongo, for example. And then we'll need to add a run_consent_request method to the new DynamoConnector and probably stub out the access/erasure methods.

[pattisdr]

FYI @SteveDMurphy this was one of the last db connectors I did, Timescale. I think it's a good reference because it shows the pieces that generally tend to be added for a database connector. The thing that was easy with Timescale is that it was basically postgres so we didn't need special querying logic (see queryconfig). Your connector would likely need custom querying logic in the queryconfig. So this database connector just does DSR (access and erasure)

ethyca/fidesops#1327

[pattisdr]

Some more background related to Adrian's proposal of using Policies:

For determining which types of fields we target on database connectors for DSR's, we look at the Policy > Rule > Rule Target.

The "consent" concept currently only has a Policy and a Rule, no Rule Target. I briefly explored it here: #2125 (comment) (in the form of a different table), but it ended up being overkill for the initiation simplifications we were asked to include at the time. We didn't need so much configuration overhead.

[Roger-Ethyca]

moving to done