Unified Fides Resources Feature

[pattisdr]

❗ Prerequisite: We need a fideslang release, so we can link to the proper fideslang version here
❗ Before merge:

• Get up to date with main again and push to this branch to re-trigger CI
• Double check if downrev needs to be bumped: this branch has several migrations. Bump the first in the series.
• See if any new saas connectors have been added - their tests will fail because they're using old DatasetConfig.dataset fields
• Other uses of now-obsolete resources should get flagged via test failures.

❗ If you primarily care about your ctl datasets, migration plan would be to delete your dataset configs prior to upgrading.

Fixes #2262

Code Changes

This feature branch contains the work below from several PR's - they've all been individually reviewed, and I've been keeping this branch up-to-date with main:

• Unified Fides Resources: Remove several fidesops schemas for DSR's in favor of updated Fideslang schemas #2009
• Unified Fides Resources: Add new datasetconfig.ctl_dataset_id field to unify fides dataset resources #2046
• Unified Fides Resources: Remove DatasetConfig.dataset field #2096
• Unified Fides Resources: Update UI dataset config routes to use new unified routes #2113
• Unified Fides Resources: Validate request body on crud endpoints on upsert. Validate dataset data categories before save. #2134
• Unified Fides Resources: Added a dataset dropdown selector when configuring a connector to link an existing dataset to the connector configuration. #2162
• Unified Fides Resources: Update test env setup and quickstart to use new routes instead of routes we plan to deprecate at some point #2225

Steps to Confirm

• Run nox -s test_env. Verify in your db that DatasetConfig.dataset column is gone. Instead there's a DatasetConfig.ctl_dataset_id FK to ctl_dataset

fides=# select * from datasetconfig;
                    id                    |          created_at           |          updated_at           |           connection_config_id           |           fides_key           |              ctl_dataset_id              
------------------------------------------+-------------------------------+-------------------------------+------------------------------------------+-------------------------------+------------------------------------------
 dat_153c7bc6-f486-47f7-9acc-b961481e5b55 | 2023-01-17 16:46:47.683558+00 | 2023-01-17 16:46:47.683558+00 | con_d3d5aff5-9365-4b2d-974c-a8a26d83af5c | postgres_example_test_dataset | ctl_8e9491eb-e8b9-4794-a4a1-9ca667393043
 dat_e8d7fb9c-78f6-49d3-be86-05b5ee2b4830 | 2023-01-17 16:46:47.826331+00 | 2023-01-17 16:46:47.826331+00 | con_f6e8acef-1d94-4df3-a61e-c9cae9ce2a8b | mongo_test                    | ctl_1451aa46-3409-4cff-ba16-726aa65a96c4

• Go the Admin UI > Configure Privacy Requests > Click "Connection Manager" left pane, and then click on Mongo DB Connector three dots > Configure
• Click on Mongo Dataset configuration tab
• Verify yaml looks as expected. Note organization_fides_key has been automatically added, and there are fides_meta instead of fidesops_meta keys
• Go to Privacy Center - submit an access request for [email protected]
• Return to admin-ui request manager and approve that privacy request
• Verify results in fides_uploads
• Experiment with adding and updating more connection configs / dataset configs in the connection manager, including adding saas connectors like mailchimp
  • Note that this operation makes an upsert to ctl Datasets and then a Patch to DatasetConfig to link that Ctl Dataset to the DatasetConfig. Check your Network tab.

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

Description Of Changes

Merges the unified-fides-resources branch. Big picture, we've been storing Datasets in two separate places with our two separate fides and fidesops repos and this branch consolidates to one location.

The CTL side has been storing this in a ctl_datasets table and the OPS side has been storing in the datasetconfig.dataset table. We move to storing in one place: ctl_datasets, and adding a datasetconfig.ctl_dataset_id FK.

[allisonking]

I just went through your steps to confirm (thanks for all the details!) and everything worked as expected! 🚀 can we cut fideslang now and then update this PR, or do we need to wait for something else?

[seanpreston]

I've published the fideslang release here

[pattisdr]

Thanks @seanpreston. Bumped fideslang version but added do not merge label here so we wait for this week's release first.

[pattisdr]

Current tests failing on this branch, need to verify if these are on main:

• saas requests failing with `test_friendbuy_nextgen_erasure_request_task
• tests/ctl/core/test_system.py::test_scan_system_aws_passes

[pattisdr]

Getting this branch up to date now -

[pattisdr]

Confirmed failing tests are also on main (these are both tests that don't get run as frequently - you need the unsafe labels check)

https://github.com/ethyca/fides/actions/runs/3968715043/jobs/6802446026
https://github.com/ethyca/fides/actions/runs/3968715043/jobs/6802445249

[pattisdr]

Getting up-to-date again, conflicts with #2249, which had a migration, downrevs need to be bumped too

[seanpreston]

Hey @pattisdr I gave this some QA this morning, it's looking great!

[pattisdr]

@seanpreston thank you!

[pattisdr]

One last main merge ^