Optimize TCF bundle with just-in-time GVL translations #5074

gilluminate · 2024-07-09T22:19:40Z

Description Of Changes

In an effort to reduce initial payload size of TCF version of FidesJS, we now ignoring GVL translations in the experience until just as we're rendering the UI.

This is being tightened up by calling the dedicated endpoint for GVL translations.

Code Changes

A bit of general code maintenance, including removing a circular dependency in the Location switcher component
Remove gvl_translations from the initial fides.js payload.
When needed, fetch the gvl_translations from /api/v1/plus/gvl/translations to use just as the UI is rendering.
Update Cypress and unit tests to reflect the change.

Steps to Confirm

Visit demo page with TCF enabled /fides-js-demo.html?geolocation=fr-id
Inspect Network panel of devtools and note that the payload is ~300k smaller than before
As the page loads, Network panel should show a call to api/v1/privacy-experience
Console debug messages should include Calling Fides GET GVL translations API... and Recieved GVL languages response from Fides API
The Fides has been initialized! message should appear prior to the Recieved GVL languages response from Fides API message in the console. We don't want to delay the overall initialization for this.
Overlay should display with full expected text (no untranslated strings such as "exp.tcf.stacks.1.name")

Pre-Merge Checklist

All CI Pipelines Succeeded
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md

vercel · 2024-07-09T22:19:43Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Jul 17, 2024 6:12pm

cypress · 2024-07-09T22:31:07Z

Passing run #8974 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `384a8ae` into `f390fc9`...
Project: fides	Commit: `668608c1c0 ℹ️`
Status: Passed	Duration: 00:35 💡
Started: Jul 17, 2024 6:22 PM	Ended: Jul 17, 2024 6:23 PM

Review all test suite changes for PR #5074 ↗︎

gilluminate · 2024-07-15T15:13:14Z

clients/fides-js/src/components/tcf/TcfVendors.tsx

+      gvlVendors: activeChunk?.filter((v) => v.isGvl),
+      otherVendors: activeChunk?.filter((v) => !v.isGvl),
    }),
    [activeChunk]
  );

+  if (!activeChunk) {
+    return null;
+  }
+


handling (unlikely) scenario of having 0 vendors configured for a given experience

gilluminate · 2024-07-15T20:06:27Z

clients/fides-js/__tests__/__fixtures__/mock_gvl_translations.json

+    "gvlSpecificationVersion": 3,
    "vendorListVersion": 40,
+    "tcfPolicyVersion": 4,


resolves a Typescript complaint that these were missing

eastandwestwind · 2024-07-16T14:43:12Z

Reviewing now...

eastandwestwind

This is on the right track after testing locally. Let's re-test once BE is in place

eastandwestwind · 2024-07-16T22:50:38Z

Just a note that after https://github.com/ethyca/fidesplus/pull/1510 is merged, lets remove the gvl_translations: Optional[Dict] = {} from src/fides/api/models/privacy_experience.py

gilluminate · 2024-07-15T20:09:48Z

clients/fides-js/__tests__/lib/i18n/i18n-utils.test.ts

@@ -483,49 +463,6 @@ describe("i18n-utils", () => {
        expect(getRecordCounts(loadedMessagesEn)).toMatchObject(expectedCounts);
        expect(getRecordCounts(loadedMessagesEs)).toMatchObject(expectedCounts);
      });
-
-      it("handles a mismatch between the experience_config and gvl_translations APIs by returning only locales available in both", () => {


this is all handled server-side now

gilluminate · 2024-07-15T20:11:44Z

clients/fides-js/src/lib/i18n/i18n-utils.ts

@@ -224,13 +224,7 @@ export function loadMessagesFromFiles(i18n: I18n): Locale[] {

 /**
 * Parse the provided PrivacyExperience object and load all translated strings
- * into the message catalog. Extracts translations from two sources:
- * 1) experience.experience_config
- * 2) experience.gvl_translations


#2 now happens as a separate process during initOverlay

gilluminate · 2024-07-15T20:17:43Z

clients/privacy-center/cypress/e2e/consent-banner-gpp.cy.ts

-            expect(data.signalStatus).to.eql("ready");
-            expect(data.applicableSections).to.eql([-1]);


developer's local Admin UI config could make this behave differently because we're testing from /fides-js-demo.html instead of /fides-js-components-demo.html in this particular test. Removing these 2 extra, flaky checks in favor of the original intent of the test, which is to check that GPP loads and can be pinged. (The other gets tested elsewhere)

gilluminate · 2024-07-15T20:21:29Z

clients/privacy-center/cypress/e2e/consent.cy.ts

-              data_sales: false,
-              tracking: false,
-              analytics: true,


Again, because we're testing with /fides-js-demo.html here instead of /fides-js-components-demo.html, these fields can be different per developer's environment and don't relate to the actual intent of this particular test.

CHANGELOG.md

eastandwestwind

Looking good so far, just some notes to remove the gvl_translations obj later, including the gvl_translations: Optional[Dict] = {} we define on the SQLAlchemy obj

clients/fides-js/src/lib/consent-types.ts

clients/fides-js/src/services/api.ts

mimic future state more closely by extrapolating those available_locales in the FE as well as passing those back to the /api/v1/plus/gvl/translations endpoint even though the filter isn't in place yet

codecov · 2024-07-17T18:38:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.48%. Comparing base (f390fc9) to head (384a8ae).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5074      +/-   ##
==========================================
+ Coverage   86.47%   86.48%   +0.01%     
==========================================
  Files         357      357              
  Lines       22271    22270       -1     
  Branches     2944     2944              
==========================================
+ Hits        19258    19260       +2     
+ Misses       2498     2495       -3     
  Partials      515      515

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cypress · 2024-07-17T18:53:47Z

Passing run #8978 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Optimize TCF bundle with just-in-time GVL translations (#5074)
Project: fides	Commit: `63ced02516`
Status: Passed	Duration: 00:35 💡
Started: Jul 17, 2024 6:53 PM	Ended: Jul 17, 2024 6:53 PM

Review all test suite changes for PR #5074 ↗︎

gilluminate force-pushed the gill/PROD-2281_mediavine-optimize-tcf-bundle branch from f901d27 to d22bfb1 Compare July 9, 2024 23:00

vercel bot deployed to Preview July 12, 2024 22:23 View deployment

gilluminate commented Jul 15, 2024

View reviewed changes

gilluminate force-pushed the gill/PROD-2281_mediavine-optimize-tcf-bundle branch from 469e8c6 to efc0b1b Compare July 15, 2024 20:05

gilluminate commented Jul 15, 2024

View reviewed changes

eastandwestwind self-requested a review July 16, 2024 10:37

eastandwestwind reviewed Jul 16, 2024

View reviewed changes

gilluminate force-pushed the gill/PROD-2281_mediavine-optimize-tcf-bundle branch from 667139b to 8e1ebee Compare July 16, 2024 22:42

gilluminate force-pushed the gill/PROD-2281_mediavine-optimize-tcf-bundle branch from ac24959 to 04673ff Compare July 17, 2024 00:06

gilluminate commented Jul 17, 2024

View reviewed changes

eastandwestwind reviewed Jul 17, 2024

View reviewed changes

clients/fides-js/src/lib/consent-types.ts Outdated Show resolved Hide resolved

clients/fides-js/src/services/api.ts Outdated Show resolved Hide resolved

gilluminate added 15 commits July 17, 2024 12:11

fix extra long strings being rendered in demo pages

28a159d

update logs to be more helpful

e3f2bd8

remove circular dependency

3387920

Side load GVL translations to reduce payload size

8d1132a

Add fetchGvlTranslations to call existing endpoint

f28acc7

set available_locales on the experience

b1f9980

mimic future state more closely by extrapolating those available_locales in the FE as well as passing those back to the /api/v1/plus/gvl/translations endpoint even though the filter isn't in place yet

mimic filtered GVL Language response

f9db42e

update cypress tests

446dfcb

fix more tests

f875db2

offset GVL translation timing. stop awaiting initOverlay

e03da81

reduce flakiness of test. just check success

3791ad6

fix admin-ui language issue: supply locales to preveiw

d54555b

fix unit tests

08b5e4e

Update api.ts

309b1ac

fix flaky test by not looking at data reliant on config

5a871a5

gilluminate added 7 commits July 17, 2024 12:11

error handling for gvl translations

8bdc809

add Cypress test for failing translation API fallback

bbf90ce

update Changelog. fix typo

3a825ac

remove client-side parsing of available_locales

cf7d492

update to use new version of translations API

b0b2456

misc. clean up

ae1c3ca

remove all references to gvl_translations

384a8ae

gilluminate force-pushed the gill/PROD-2281_mediavine-optimize-tcf-bundle branch from f7fa793 to 384a8ae Compare July 17, 2024 18:11

eastandwestwind approved these changes Jul 17, 2024

View reviewed changes

gilluminate merged commit 63ced02 into main Jul 17, 2024
51 checks passed

gilluminate deleted the gill/PROD-2281_mediavine-optimize-tcf-bundle branch July 17, 2024 18:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize TCF bundle with just-in-time GVL translations #5074

Optimize TCF bundle with just-in-time GVL translations #5074

gilluminate commented Jul 9, 2024 •

edited

Loading

vercel bot commented Jul 9, 2024 •

edited

Loading

cypress bot commented Jul 9, 2024 •

edited

Loading

gilluminate Jul 15, 2024

gilluminate Jul 15, 2024

eastandwestwind commented Jul 16, 2024

eastandwestwind left a comment

eastandwestwind commented Jul 16, 2024 •

edited

Loading

gilluminate Jul 15, 2024

gilluminate Jul 15, 2024

gilluminate Jul 15, 2024 •

edited

Loading

gilluminate Jul 15, 2024

eastandwestwind left a comment

codecov bot commented Jul 17, 2024 •

edited

Loading

cypress bot commented Jul 17, 2024

		expect(data.signalStatus).to.eql("ready");
		expect(data.applicableSections).to.eql([-1]);

Optimize TCF bundle with just-in-time GVL translations #5074

Optimize TCF bundle with just-in-time GVL translations #5074

Conversation

gilluminate commented Jul 9, 2024 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

vercel bot commented Jul 9, 2024 • edited Loading

cypress bot commented Jul 9, 2024 • edited Loading

Passing run #8974 ↗︎

Review all test suite changes for PR #5074 ↗︎

gilluminate Jul 15, 2024

Choose a reason for hiding this comment

gilluminate Jul 15, 2024

Choose a reason for hiding this comment

eastandwestwind commented Jul 16, 2024

eastandwestwind left a comment

Choose a reason for hiding this comment

eastandwestwind commented Jul 16, 2024 • edited Loading

gilluminate Jul 15, 2024

Choose a reason for hiding this comment

gilluminate Jul 15, 2024

Choose a reason for hiding this comment

gilluminate Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

gilluminate Jul 15, 2024

Choose a reason for hiding this comment

eastandwestwind left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 17, 2024 • edited Loading

Codecov Report

cypress bot commented Jul 17, 2024

Passing run #8978 ↗︎

Review all test suite changes for PR #5074 ↗︎

gilluminate commented Jul 9, 2024 •

edited

Loading

vercel bot commented Jul 9, 2024 •

edited

Loading

cypress bot commented Jul 9, 2024 •

edited

Loading

eastandwestwind commented Jul 16, 2024 •

edited

Loading

gilluminate Jul 15, 2024 •

edited

Loading

codecov bot commented Jul 17, 2024 •

edited

Loading