Integrate vendor legal basis #4216

allisonking · 2023-10-03T17:31:10Z

Closes #4209
Partially addresses #4210

Description Of Changes

Whew ok this is kind of a big one! The backend API had significant changes in order to add legal basis as another dimension. Purposes, Vendors, and Systems all have this new dimension. Whereas before, our experience object looked like

{
  ...
  tcf_purposes: [...],
  tcf_vendors: [...],
  tcf_systems: [...],
}

it now looks like

{
  ...
  tcf_consent_purposes: [...],
  tcf_legitimate_interests_purposes: [...],
  tcf_consent_vendors: [...],
  tcf_legitimate_interests_vendors: [...],
  tcf_consent_systems: [...],
  tcf_legitimate_interests_systems: [...],
  
  tcf_vendor_relationships: [...],
  tcf_system_relationships: [...]
}

In other words, we have a lot more fields to work with! the *relationships fields are to deal with fields that the vendor/system fields have in common, in order not to repeat too much in the payload. More context about that here.

At the same time, the FE had gone ahead and implemented double vendor toggles, saving only to the TC string. Now that the backend can save it, we can get rid of some tech debt code 💥

Code Changes

Update TS types in both admin-ui and fides-js
Add EEA to human readable regions
Delete various files we don't need anymore (CookieTable (never used) and LegalBasisDropdown (no longer needed))
Refactor vendors/systems throughout to handle legal basis coming from the backend
Lots of refactoring to make purposes also handle a legal basis dimension
Update cypress tests (this is a big diff because the payload data changed so much)

Steps to Confirm

Set up your TCF environment
Add various purposes of either purpose or legitimate interest types. Add them to different vendors
Load the privacy center and visit /fides-js-demo.html?geolocation=fi-18. You should see the prefetched experience in the HTML. this should have all the new fields
Make sure everything still works
The one thing that won't work is you won't be able to change Legitimate Interest Purposes individually yet (opting in or out of all will work). We need to add the second toggle to that tab to do so, but I split that out into a separate ticket

Pre-Merge Checklist

All CI Pipelines Succeeded
Issue Requirements are Met
Update CHANGELOG.md

cypress · 2023-10-03T17:48:19Z

Passing run #4509 ↗︎

0	4	0	0	0

Details:

Merge `a8db91c` into `7fece59`...
Project: fides	Commit: `67ca0712f8 ℹ️`
Status: Passed	Duration: 01:12 💡
Started: Oct 6, 2023 8:56 PM	Ended: Oct 6, 2023 8:57 PM

Review all test suite changes for PR #4216 ↗︎

codecov · 2023-10-03T18:23:08Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (4aae6e0) 87.70% compared to head (eb4051c) 87.70%.
Report is 1 commits behind head on fidesplus_1128_legal_basis_dimension.

Additional details and impacted files

@@                          Coverage Diff                          @@
##           fidesplus_1128_legal_basis_dimension    #4216   +/-   ##
=====================================================================
  Coverage                                 87.70%   87.70%           
=====================================================================
  Files                                       333      333           
  Lines                                     20814    20819    +5     
  Branches                                   2697     2698    +1     
=====================================================================
+ Hits                                      18254    18259    +5     
  Misses                                     2095     2095           
  Partials                                    465      465

see 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ntegrate-vendor-legal-basis

allisonking · 2023-10-04T21:53:14Z

clients/fides-js/src/components/tcf/InitialLayer.tsx

+    tcf_special_features: experienceSpecialFeatures = [],
+  } = experience;
+
+  const uniquePurposeIds = useMemo(() => {


we now have two fields: tcf_consent_purposes and tcf_legitimate_interests_purposes which can overlap! this means, if we only want one instance of a purpose, we need to dedupe. Stacks don't care if a purpose is consent or legint, so we combine them for this file.

allisonking · 2023-10-04T21:59:13Z

clients/fides-js/src/components/tcf/TcfOverlay.tsx

    } = experience;

-    const vendorsAndSystems = [...(vendors || []), ...(systems || [])];


this was the bulk of the tech debt I removed 🧹

allisonking · 2023-10-04T22:13:01Z

clients/fides-js/src/components/tcf/TcfPurposes.tsx

-};
+}) => (
+  <div>
+    <PurposeBlock


Right now we are only rendering consent purposes—legint will follow in #4210

allisonking · 2023-10-04T22:13:59Z

clients/fides-js/src/components/tcf/VendorInfoBanner.tsx

-        vendors,
-        LegalBasisForProcessingEnum.LEGITIMATE_INTERESTS
-      ).length;
+    const consent = consentSystems.length + consentVendors.length;


counting also got much easier with the backend update 👍

allisonking · 2023-10-04T22:15:35Z

clients/fides-js/src/lib/tcf/types.ts

@@ -5,6 +5,16 @@ import type {
  UserConsentPreference,
 } from "../consent-types";

+export enum LegalBasisForProcessingEnum {


these are mostly all copied from the updated admin-ui types. I rearranged them a bit in the file to make more intuitive sense (hopefully)

allisonking · 2023-10-04T22:16:39Z

clients/fides-js/src/lib/tcf/vendors.ts

-    return (
-      hasApplicablePurposes?.length || hasApplicableSpecialPurposes?.length
-    );
+const transformVendorDataToVendorRecords = ({


this function is one of the larger changes in this PR. basically the vendor tab wants both vendors and systems together. on top of that, it also wants it combined across legal basis. this function returns a type that makes that easy

eastandwestwind

Thanks for walking through this one with me on screenshare @allisonking !

eastandwestwind · 2023-10-05T15:07:11Z

clients/fides-js/src/components/tcf/TcfVendors.tsx

-            [vendor],
-            LegalBasisForProcessingEnum.LEGITIMATE_INTERESTS
-          ).length === 1;
+          experience.gvl?.dataCategories;


…ntegrate-vendor-legal-basis

allisonking · 2023-10-06T15:33:42Z

Updated a bunch of types after PR review changes to 8933d7d

…ntegrate-vendor-legal-basis

allisonking added 2 commits October 3, 2023 13:27

Update TS types

21cfc58

Add EEA to regions

eb4051c

allisonking changed the base branch from main to fidesplus_1128_legal_basis_dimension October 3, 2023 17:31

Update tcf types

7ae1e60

allisonking added 14 commits October 3, 2023 16:52

Update more types

07e055a

Add transform function to help with working with vendor data

eea9efa

Refactor for updated API

25f7c72

Get everything building again

4d94fec

Delete unused CookiesTable

bacdd33

Delete no longer needed LegalBasisDropdown

3b1ba31

Fix bug in passing gvl to Vendors

99246ef

Handle split out purpose consents and legints

decf663

Update unit tests

577240a

Fix type

9f243af

Do not assume relationships have all entities

26a5471

Fix bug with purpose forbidden legitimate interests

2de65b6

Update cypress tests

3da791e

Merge branch 'fidesplus_1128_legal_basis_dimension' into aking/4209/i…

698778c

…ntegrate-vendor-legal-basis

allisonking commented Oct 4, 2023

View reviewed changes

allisonking added 2 commits October 4, 2023 18:19

Update changelog

3b39644

Update types again

6b0a14c

allisonking marked this pull request as ready for review October 4, 2023 22:27

allisonking requested a review from eastandwestwind October 4, 2023 22:28

allisonking requested a review from galvana October 4, 2023 22:28

eastandwestwind approved these changes Oct 5, 2023

View reviewed changes

allisonking added 4 commits October 6, 2023 10:37

Merge branch 'fidesplus_1128_legal_basis_dimension' into aking/4209/i…

384434d

…ntegrate-vendor-legal-basis

Autogenerate admin-ui types

559dbb7

Update types per new API change

c65f943

Revert cypress config change

d17e9bb

allisonking and others added 3 commits October 6, 2023 15:10

Purpose legal basis (#4234)

e730838

Update privacy center patch call for new API response (#4235)

5f614aa

Merge branch 'fidesplus_1128_legal_basis_dimension' into aking/4209/i…

a8db91c

…ntegrate-vendor-legal-basis

allisonking merged commit e03bca0 into fidesplus_1128_legal_basis_dimension Oct 6, 2023
7 checks passed

allisonking deleted the aking/4209/integrate-vendor-legal-basis branch October 6, 2023 20:59

allisonking mentioned this pull request Oct 9, 2023

Update fides-js to handle saving vendors with a legal basis dimension #4209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate vendor legal basis #4216

Integrate vendor legal basis #4216

allisonking commented Oct 3, 2023 •

edited

Loading

cypress bot commented Oct 3, 2023 •

edited

Loading

codecov bot commented Oct 3, 2023

allisonking Oct 4, 2023 •

edited

Loading

allisonking Oct 4, 2023

allisonking Oct 4, 2023

allisonking Oct 4, 2023

allisonking Oct 4, 2023

allisonking Oct 4, 2023

eastandwestwind left a comment

eastandwestwind Oct 5, 2023

allisonking commented Oct 6, 2023

		} = experience;

		const vendorsAndSystems = [...(vendors \|\| []), ...(systems \|\| [])];

Integrate vendor legal basis #4216

Integrate vendor legal basis #4216

Conversation

allisonking commented Oct 3, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Oct 3, 2023 • edited Loading

Passing run #4509 ↗︎

Review all test suite changes for PR #4216 ↗︎

codecov bot commented Oct 3, 2023

Codecov Report

allisonking Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

allisonking Oct 4, 2023

Choose a reason for hiding this comment

allisonking Oct 4, 2023

Choose a reason for hiding this comment

allisonking Oct 4, 2023

Choose a reason for hiding this comment

allisonking Oct 4, 2023

Choose a reason for hiding this comment

allisonking Oct 4, 2023

Choose a reason for hiding this comment

eastandwestwind left a comment

Choose a reason for hiding this comment

eastandwestwind Oct 5, 2023

Choose a reason for hiding this comment

allisonking commented Oct 6, 2023

allisonking commented Oct 3, 2023 •

edited

Loading

cypress bot commented Oct 3, 2023 •

edited

Loading

allisonking Oct 4, 2023 •

edited

Loading