Undeclared data categories #4781

galvana · 2024-04-09T03:06:30Z

Closes PROD-1873

Description Of Changes

Updating datamap report table with two new columns:

System undeclared data categories - A list of data categories aggregated from all of a system's datasets that are not declared in any of the system's data uses
Data use undeclared data categories - A list of data categories from a data use's dataset that are not declared in the current data use or any sibling data uses

Code Changes

Adding new columns to DatamapReportTable.tsx
Misc fixes for front-end cache invalidation
Helper functions on System and PrivacyDeclaration models to aggregate and report on associated data categories

Steps to Confirm

Start fidesplus from the branch in (https://github.com/ethyca/fidesplus/pull/1384)
Start the Admin UI in fides
Navigate to Data Inventory > Manage Datasets
Upload the Postgres sample dataset

System undeclared data categories

Navigate to Data Inventory > Add System, set the System name and set Dataset references
to the Postgres sample dataset, click Save
Navigate to Data Inventory > Reporting, the System undeclared data categories should not be visible
Click on Edit columns and enable the System undeclared data categories column, click Save
The enabled column should be visible and expandable
System undeclared data categories should say "12 system undeclared data categories"
Navigate back to the system and add a new data use with the user data category
Navigate back to the datamap report table. Enable the column again. The System undeclared data categories column should now show a single undeclared data category, System Data: Operations Data

Data use undeclared data categories

Remove the dataset from the system, create a new data use on the system, set the data category to System and add the Postgres dataset
Navigate to Data Inventory > Reporting, the Data use undeclared data categories should not be visible
Click on Edit columns and enable the Data use undeclared data categories column, click Save
The enabled column should be visible and expandable
Data use undeclared data categories should say "11 system undeclared data categories"
Navigate back to the system and change the data category on the data use to user
Navigate back to the datamap report table. Enable the column again. The Data use undeclared data categories column should now show a single undeclared data category, System Data: Operations Data

Pre-Merge Checklist

All CI Pipelines Succeeded
Issue Requirements are Met
Update CHANGELOG.md

vercel · 2024-04-09T03:06:33Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Apr 22, 2024 11:52pm

cypress · 2024-04-09T03:19:06Z

Passing run #7320 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `2ee07d0` into `c7df743`...
Project: fides	Commit: `6d219f6f22 ℹ️`
Status: Passed	Duration: 00:33 💡
Started: Apr 18, 2024 8:09 PM	Ended: Apr 18, 2024 8:10 PM

Review all test suite changes for PR #4781 ↗︎

codecov · 2024-04-09T03:42:32Z

Codecov Report

Attention: Patch coverage is 95.45455% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 86.62%. Comparing base (c7df743) to head (2ee07d0).
Report is 1 commits behind head on main.

Files	Patch %	Lines
src/fides/api/models/sql_models.py	93.75%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4781      +/-   ##
==========================================
+ Coverage   86.61%   86.62%   +0.01%     
==========================================
  Files         339      340       +1     
  Lines       20090    20132      +42     
  Branches     2586     2595       +9     
==========================================
+ Hits        17400    17440      +40     
- Misses       2217     2218       +1     
- Partials      473      474       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…lumns

galvana · 2024-04-15T22:17:01Z

clients/admin-ui/src/features/common/table/v2/column-settings/ColumnSettingsModal.tsx

@@ -47,7 +47,8 @@ export const ColumnSettingsModal = <T,>({
        .map((c) => ({
          id: c.id,
          displayText: c.columnDef?.meta?.displayText || c.id,
-          isVisible: c.getIsVisible(),
+          isVisible:
+            tableInstance.getState().columnVisibility[c.id] ?? c.getIsVisible(),


This is to allow us to set default visibility, see: https://github.com/ethyca/fides/pull/4781/files#diff-cd47c40887e69a55fee3a54b819214c466bf33045c6be67473345c87cef25bbcR983-R986

clients/admin-ui/src/features/datamap/datamap.slice.ts

clients/admin-ui/src/features/datamap/modals/SettingsModal.tsx

clients/admin-ui/src/features/datamap/reporting/DatamapReportTable.tsx

requirements.txt

galvana · 2024-04-15T22:20:40Z

src/fides/api/models/sql_models.py

+    system = relationship(
+        System, back_populates="privacy_declarations", lazy="selectin"
+    )


Need to be careful of this change, it makes the system retrieval on privacy declarations eager

Are you concerned about performance on this PR?

Yes, that's my main concern with that change. I'm planning on running nox -s performance_tests between main and this branch to see if there are any significant timing changes.

Systems already feel very bogged down, at some point we may reach a tipping point where we need to rearchitect some of this - not this specifically I just mean generally System/Privacy Declaration

pattisdr · 2024-04-16T15:12:51Z

Plan to start review Wed 4/17 -

pattisdr

this is looking great, I just have a few questions, also these are the best testing instructions I've ever seen. Really helpful for someone that wasn't involved in the conversations for this feature.

src/fides/api/models/sql_models.py

src/fides/api/util/taxonomy_utils.py

src/fides/api/models/sql_models.py

Undeclared data category helper functions

c81aff6

galvana added 2 commits April 10, 2024 12:00

Updating data category matching

0227a22

Adding undeclared data categories column

47762b4

vercel bot deployed to Preview April 10, 2024 22:53 View deployment

Adding header menu

c2f44f6

vercel bot deployed to Preview April 10, 2024 23:00 View deployment

Merge branch 'main' into PROD-1873-undeclared_data_categories

f3589ed

vercel bot deployed to Preview April 15, 2024 15:38 View deployment

galvana added 2 commits April 15, 2024 09:04

Updating undeclared data category check

9418192

Updating datamap report table to show two undeclared data category co…

76b5b43

…lumns

vercel bot deployed to Preview April 15, 2024 21:21 View deployment

Fixing tests

c982332

galvana requested a review from adamsachs April 15, 2024 22:08

galvana marked this pull request as ready for review April 15, 2024 22:13

galvana changed the title ~~Undeclared data category helper functions~~ Undeclared data categories Apr 15, 2024

galvana commented Apr 15, 2024

View reviewed changes

pattisdr self-requested a review April 16, 2024 15:10

Merge branch 'main' into PROD-1873-undeclared_data_categories

d6223a4

vercel bot deployed to Preview April 17, 2024 16:27 View deployment

pattisdr reviewed Apr 17, 2024

View reviewed changes

src/fides/api/models/sql_models.py Show resolved Hide resolved

src/fides/api/util/taxonomy_utils.py Show resolved Hide resolved

src/fides/api/models/sql_models.py Outdated Show resolved Hide resolved

src/fides/api/models/sql_models.py Show resolved Hide resolved

galvana requested a review from pattisdr April 18, 2024 19:51

Changes based on PR feedback

2ee07d0

pattisdr approved these changes Apr 18, 2024

View reviewed changes

galvana added 2 commits April 22, 2024 16:47

Merge branch 'main' into PROD-1873-undeclared_data_categories

3c1169f

Updating change log

eafdc7f

galvana merged commit 887b31c into main Apr 22, 2024
20 checks passed

galvana deleted the PROD-1873-undeclared_data_categories branch April 22, 2024 23:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Undeclared data categories #4781

Undeclared data categories #4781

galvana commented Apr 9, 2024 •

edited by pattisdr

Loading

vercel bot commented Apr 9, 2024 •

edited

Loading

cypress bot commented Apr 9, 2024 •

edited

Loading

codecov bot commented Apr 9, 2024 •

edited

Loading

galvana Apr 15, 2024

galvana Apr 15, 2024

pattisdr Apr 18, 2024

galvana Apr 18, 2024

pattisdr Apr 18, 2024 •

edited

Loading

pattisdr commented Apr 16, 2024

pattisdr left a comment

Undeclared data categories #4781

Undeclared data categories #4781

Conversation

galvana commented Apr 9, 2024 • edited by pattisdr Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

vercel bot commented Apr 9, 2024 • edited Loading

cypress bot commented Apr 9, 2024 • edited Loading

Passing run #7320 ↗︎

Review all test suite changes for PR #4781 ↗︎

codecov bot commented Apr 9, 2024 • edited Loading

Codecov Report

galvana Apr 15, 2024

Choose a reason for hiding this comment

galvana Apr 15, 2024

Choose a reason for hiding this comment

pattisdr Apr 18, 2024

Choose a reason for hiding this comment

galvana Apr 18, 2024

Choose a reason for hiding this comment

pattisdr Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

pattisdr commented Apr 16, 2024

pattisdr left a comment

Choose a reason for hiding this comment

galvana commented Apr 9, 2024 •

edited by pattisdr

Loading

vercel bot commented Apr 9, 2024 •

edited

Loading

cypress bot commented Apr 9, 2024 •

edited

Loading

codecov bot commented Apr 9, 2024 •

edited

Loading

pattisdr Apr 18, 2024 •

edited

Loading