TCF bundling based on geography #4131

allisonking · 2023-09-21T20:12:18Z

Description Of Changes

We can now allow both a notice experience and a TCF experience!

Code Changes

Sort out some cookie logic which became much more apparently not quite right when switching between notices and TCF
Remove the TCF_ENABLED environment variable from the PC—now we determine if it's TCF purely from the backend 🎉
Cypress tests

Steps to Confirm

Run nox -s "fides_env(slim)" -- prod_prerelease core_version=2.20.2a2
Bring down the privacy center container that the test env creates
Bring up the privacy center on this branch using turbo dev in clients/privacy-center. I used these env variables (note: no more TCF_ENABLED 🎉 )

FIDES_PRIVACY_CENTER__DEBUG=true
FIDES_PRIVACY_CENTER__IS_OVERLAY_ENABLED=true
FIDES_PRIVACY_CENTER__IS_PREFETCH_ENABLED=true
FIDES_PRIVACY_CENTER__IS_GEOLOCATION_ENABLED=true
FIDES_PRIVACY_CENTER__GEOLOCATION_API_URL=https://cdn-api.ethyca.com/location
FIDES_PRIVACY_CENTER__PRIVACY_CENTER_URL=http://localhost:3000

Turn TCF on in the backend via localhost:8080/docs
Now that TCF is enabled, visit localhost:8080 and edit one of the existing systems.
Add a data use such as analytics.reporting.campaign_insights (which is one of the TCF data uses). Give it a legal basis of "Consent" and save.
Visit the consent page and enable all the privacy notices that are Available.
Visit cookie house at localhost:3000. Choose California from the geo picker. You should see "Manage preferences" show up at the bottom. Click it and you should see "Data sales and sharing". The banner won't come up since that's not required for CA
Now choose Frankfurt from the geo picker. The banner should come up, and you can click "manage preferences" to open what should be the TCF modal!
Interact with both modals, switching between geographies, to make sure your choices persist and the cookie looks right

Pre-Merge Checklist

All CI Pipelines Succeeded
Issue Requirements are Met
Update CHANGELOG.md

cypress · 2023-09-21T20:26:37Z

Passing run #4256 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `f2095b3` into `4b452ed`...
Project: fides	Commit: `cb0ac18ee3 ℹ️`
Status: Passed	Duration: 01:15 💡
Started: Sep 21, 2023 8:42 PM	Ended: Sep 21, 2023 8:44 PM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

allisonking · 2023-09-21T21:18:28Z

clients/fides-js/src/components/Overlay.tsx

@@ -38,16 +34,6 @@ interface Props {
  renderModalContent: (props: RenderModalContent) => VNode;
 }

-/**


I realized this wasn't needed anymore once I fixed how the updating cookie logic in place worked

allisonking · 2023-09-21T21:19:11Z

clients/fides-js/src/fides-tcf.ts

@@ -136,7 +141,8 @@ const updateCookie = async (
  };

  const tcString = await generateTcString({ tcStringPreferences, experience });
-  return { ...oldCookie, tc_string: tcString };
+  const tcfConsent = transformTcfPreferencesToCookieKeys(tcStringPreferences);
+  return { ...oldCookie, tc_string: tcString, tcf_consent: tcfConsent };


this probably should have been in #4124 but I didn't realize it was missing until I started switching between TCF and non-TCF

allisonking · 2023-09-21T21:20:54Z

clients/fides-js/src/lib/initialize.ts

@@ -252,18 +262,12 @@ export const initialize = async ({
      isPrivacyExperience(effectiveExperience) &&
      experienceIsValid(effectiveExperience, options)
    ) {
-      // Overwrite cookie consent with experience-based consent values
-      // eslint-disable-next-line no-param-reassign
-      cookie.consent = buildCookieConsentForExperiences(


this was messing up when going from Notices --> TCF because TCF doesn't have any notices, so this was making the consent object {} which would erase if someone had made choices in CA. So this logic got moved out to only notices. TCF was already reassigning the cookie, so this makes notices do the same, by having it also pass in an updateCookie func

This does have an interesting side effect which I'm not sure if we can avoid. Let's say our cookie starts as

{ consent: { data_sales_and_sharing: false } }

Now we switch to TCF. when TCF initializes, there are no notices, so it only has the consent defaults, which it merges with the existing cookie

{ consent: { data_sales: true, tracking: true, analytics: true, data_sales_and_sharing: true } }

I'm not sure if it's ok to have the consent defaults hanging around like that. That's from this function which I think is there so that we continue support for the legacy config. We could potentially say just ignore those defaults if we are in TCF mode?

Hmm, thinking beyond one impl on a customer's site, suppose a web user were to visit 2 different websites, site A uses Fides non-TCF and the site B uses Fides TCF. The user has submitted consent for site A, then navigates to site B. I'd think we'd want to preserve any existing consent the user has given on site A, rather than overwrite with defaults from TCF, right?

site A and site B would use different cookies, right?

I didn't think so. It should just be the same fides_consent cookie

allisonking · 2023-09-21T21:22:37Z

clients/fides-js/src/lib/preferences.ts

-      transformUserPreferenceToBoolean(consentPreference),
-    ])
-  );
-  const consentCookieKey: CookieKeyConsent = Object.fromEntries(noticeMap);


similar to https://github.com/ethyca/fides/pull/4131/files#r1333616066, this moves the cookie assignment logic out of the shared function, and into a notice-specific function. this is because otherwise, when we save TCF (which has no notices), we would overwrite cookie.consent to equal {} which isn't right!

allisonking · 2023-09-21T21:23:15Z

clients/fides-js/src/lib/preferences.ts

-  // Update the cookie object
-  // eslint-disable-next-line no-param-reassign
-  cookie.consent = consentCookieKey;
+  const fidesUserPreferences: Array<ConsentOptionCreate> | undefined =


this is the same, but wrapped in a ternary to be a little more careful about what preferences we pass to the backend. we don't want to accidentally overwrite notice preferences when working with tcf data!

allisonking · 2023-09-21T21:23:31Z

clients/fides-js/src/lib/preferences.ts

  debugLog(debug, "Updating window.Fides");
  window.Fides.consent = cookie.consent;
+  window.Fides.tc_string = cookie.tc_string;
+  window.Fides.tcf_consent = cookie.tcf_consent;


also probably should have been in #4124

allisonking · 2023-09-21T21:24:55Z

clients/fides-js/src/lib/preferences.ts

-  // 2. Update the window.Fides.consent object
+  // 2. Update the cookie object based on new preferences
+  const updatedCookie = await updateCookie(cookie);
+  Object.assign(cookie, updatedCookie);


I learned something weird— assigning to the function param with something like

param = updatedParam;

doesn't work! (param in the parent scope is unchanged). However, doing an Object.assign does work, and so does assigning to individual attributes such as

param.consent = updatedParam.consent

eastandwestwind

2 questions for you!

eastandwestwind · 2023-09-25T20:19:17Z

clients/privacy-center/pages/api/fides-js.ts

+  // We determine server-side whether or not to send the TCF bundle, which is based
+  // on whether or not the experience is marked as TCF. This means for TCF, we *must*
+  // be able to prefetch the experience.


this feels like something we should log. So if the experience is TCF, and prefetch is NOT enabled in the env vars, we should log a warning or something

this is a bit of a chicken and egg problem, where the backend will never return that the experience is TCF if prefetch is not enabled 😕 the flow is

Get the user's location

Prefetch the experience, passing in the user location

Check if it's TCF, which can only happen if both 1 and 2 work

gotcha, I've moved this suggestion here instead- https://ethyca.atlassian.net/browse/PROD-1069?focusedCommentId=36963

eastandwestwind · 2023-09-26T14:05:11Z

clients/fides-js/src/lib/initialize.ts

@@ -252,18 +262,12 @@ export const initialize = async ({
      isPrivacyExperience(effectiveExperience) &&
      experienceIsValid(effectiveExperience, options)
    ) {
-      // Overwrite cookie consent with experience-based consent values
-      // eslint-disable-next-line no-param-reassign
-      cookie.consent = buildCookieConsentForExperiences(


Hmm, thinking beyond one impl on a customer's site, suppose a web user were to visit 2 different websites, site A uses Fides non-TCF and the site B uses Fides TCF. The user has submitted consent for site A, then navigates to site B. I'd think we'd want to preserve any existing consent the user has given on site A, rather than overwrite with defaults from TCF, right?

…geography-bundling

allisonking added 4 commits September 21, 2023 14:20

Determine if TCF is enabled from experience payload

517b6ac

Sort out cookie logic between notices and TCF

4b5ee89

Remove getLatestCookie logic

9f9b88f

Add cypress tests for TCF cookie setting

a686be3

allisonking changed the base branch from main to aking/prod-1063/tcf-prefetch September 21, 2023 20:12

Fix cookie assignment

3ef7b96

allisonking added 2 commits September 21, 2023 16:28

Remove TCF_ENABLED config from PC

491aa18

Update changelog

f2095b3

allisonking commented Sep 21, 2023

View reviewed changes

allisonking marked this pull request as ready for review September 21, 2023 21:35

allisonking requested review from eastandwestwind and NevilleS September 21, 2023 21:35

eastandwestwind reviewed Sep 26, 2023

View reviewed changes

Merge branch 'aking/prod-1063/tcf-prefetch' into aking/prod-1070/tcf-…

f2b5934

…geography-bundling

eastandwestwind approved these changes Sep 27, 2023

View reviewed changes

allisonking added 2 commits September 28, 2023 11:47

Merge branch 'aking/prod-1063/tcf-prefetch' into aking/prod-1070/tcf-…

8e709d4

…geography-bundling

Update changelog

7a1625b

allisonking merged commit 552cf68 into aking/prod-1063/tcf-prefetch Sep 28, 2023

allisonking deleted the aking/prod-1070/tcf-geography-bundling branch September 28, 2023 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TCF bundling based on geography #4131

TCF bundling based on geography #4131

allisonking commented Sep 21, 2023 •

edited

Loading

cypress bot commented Sep 21, 2023 •

edited

Loading

allisonking Sep 21, 2023

allisonking Sep 21, 2023

allisonking Sep 21, 2023 •

edited

Loading

allisonking Sep 21, 2023

eastandwestwind Sep 26, 2023

allisonking Sep 26, 2023

eastandwestwind Sep 27, 2023

allisonking Sep 21, 2023

allisonking Sep 21, 2023

allisonking Sep 21, 2023

allisonking Sep 21, 2023

eastandwestwind left a comment

eastandwestwind Sep 25, 2023

allisonking Sep 26, 2023

eastandwestwind Sep 27, 2023

eastandwestwind Sep 26, 2023

TCF bundling based on geography #4131

TCF bundling based on geography #4131

Conversation

allisonking commented Sep 21, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Sep 21, 2023 • edited Loading

Passing run #4256 ↗︎

Choose a reason for hiding this comment

Choose a reason for hiding this comment

allisonking Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eastandwestwind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

allisonking commented Sep 21, 2023 •

edited

Loading

cypress bot commented Sep 21, 2023 •

edited

Loading

allisonking Sep 21, 2023 •

edited

Loading