Compass schema updates

[allisonking]

Closes https://github.com/ethyca/fidesplus/issues/1154

Description Of Changes

The compass endpoint schemas have changed. This updates them so that things still work, particularly:

• the simplified consent page, when you add a vendor and click the sparkle icon
• adding a system and clicking on the sparkle icon

Code Changes

• Replace manual types with autogenerated dictionary types
• Add a README for how to autogenerate dictionary types
• Code changes for schema refactor
• Update cypress tests and fixtures

Steps to Confirm

• Make sure the simplified consent onboarding page with dictionary suggestions still works
• Make sure adding a system with dictionary suggestions still works

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Issue Requirements are Met
• Update CHANGELOG.md

[cypress]

Passing run #4601 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge 3c9e100 into 4aec372...
Project: fides	Commit: 51c97ffdd9 ℹ️
Status: Passed	Duration: 01:10 💡
Started: Oct 12, 2023 5:32 PM	Ended: Oct 12, 2023 5:33 PM

Review all test suite changes for PR #4254 ↗︎

[adamsachs]

Tested this branch out with new compass, and generally it is looking good! a few notes on things we can improve (i don't think any of which you could have been aware of!), in order of descending priority from my POV:

• we're setting a size=1000 param on the query to compass for Systems. we now have over 1k records so we should bump that, or else we'll be missing vendors. it actually needs to be updated on the compass side though, since 1k is the max currently! i just made that change and pushed to my branch (https://github.com/ethyca/fides-services/commit/929a9c04c91a9e5d1d1dd2a20372bbf1af96e6cc)
  • this is obviously not a sustainable solution - the compass API supports paging and we probably should make use of it here with the client, but we can continue to kick that can, so long as the ~1.6k records we have that will be sent back by compass won't choke things for now
• as we discussed in slack a bit, for the fides System.name we should use the name from compass when specified, and fallback to using the legal_name if the name is not specified on the compass record. around 300 of the 1.6k records have a name specified
• if we can, we should pass thru the new TCF related fields that have been added to the System and PrivacyDeclaration models in fides (and are also now being served out by Compass) --

on the system model/records:

"cookie_max_age_seconds": 31536000, # optional
"uses_cookies": true,
"cookie_refresh": true,
"uses_non_cookie_access": true,
"legitimate_interest_disclosure_url": "https://www.rtbhouse.com/legitimate_interest_assessments_tcf", # optional

on the data use/privacy declaration records (this field actually hadn't been populated in airtable, but i just reran generation scripts to re-populate it!):

"flexible_legal_basis_for_profiling": true

---

an example vendor record from compass that can be used for testing is gvl.10, Index Exchange - its name field is populated, as well as many of those new TCF fields:

{
  "fides_key": null,
  "organization_fides_key": "default_organization",
  "tags": [],
  "name": "Index Exchange",
  "description": "Index Exchange, Inc. is a global advertising marketplace that connects publishers and advertisers, enabling them to buy and sell ad inventory programmatically.",
  "registry_id": null,
  "meta": null,
  "fidesctl_meta": null,
  "system_type": null,
  "data_responsibility_title": null,
  "egress": null,
  "ingress": null,
  "privacy_declarations": [],
  "joint_controller": null,
  "third_country_transfers": null,
  "administrating_department": "Not defined",
  "data_protection_impact_assessment": null,
  "vendor_id": "gvl.10",
  "dataset_references": [],
  "processes_personal_data": true,
  "exempt_from_privacy_regulations": false,
  "reason_for_exemption": null,
  "uses_profiling": false,
  "legal_basis_for_profiling": [],
  "does_international_transfers": true,
  "legal_basis_for_transfers": [
    "Adequacy Decision",
    "SCCs"
  ],
  "requires_data_protection_assessments": false,
  "dpa_location": null,
  "dpa_progress": null,
  "privacy_policy": "https://www.indexexchange.com/privacy",
  "legal_name": "Index Exchange Inc. ",
  "legal_address": "Index Exchange Inc.;74 Wingold Ave;Toronto;M6B 1P5;Canada",
  "responsibility": [],
  "dpo": "[email protected]",
  "joint_controller_info": null,
  "data_security_practices": null,
  "cookie_max_age_seconds": 34128000,
  "uses_cookies": true,
  "cookie_refresh": true,
  "uses_non_cookie_access": true,
  "legitimate_interest_disclosure_url": "https://www.indexexchange.com/privacy/exchange-platform-privacy-policy/#section-3",
  "purposes": [
    1,
    2,
    7
  ],
  "special_purposes": [
    1,
    2
  ],
  "leg_int_purposes": [],
  "flexible_purposes": [
    2,
    7
  ],
  "features": [
    1,
    3
  ],
  "special_features": [
    1
  ],
  "data_declaration": [
    1,
    2,
    3,
    8,
    9,
    10,
    11
  ],
  "device_storage_disclosure_url": "https://cdn.indexexchange.com/device_storage_disclosure.json",
  "overflow": null,
  "territorial_scope": [
    "AT",
    "BE",
    "BG",
    "HR",
    "CY",
    "CZ",
    "DK",
    "EE",
    "FI",
    "FR",
    "DE",
    "GR",
    "HU",
    "IS",
    "IE",
    "IT",
    "LV",
    "LI",
    "LT",
    "LU",
    "MT",
    "NL",
    "NO",
    "PL",
    "PT",
    "RO",
    "SK",
    "SI",
    "ES",
    "SE",
    "CH",
    "GB"
  ],
  "environments": [
    "Web",
    "Native App (Mobile)",
    "Native App (CTV)",
    "Other"
  ],
  "service_types": [
    "SSP",
    "Header Bidding Service"
  ],
  "domains": null,
  "logo": null,
  "gvl_version": "3",
  "cookies": []
}

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (4aec372) 87.75% compared to head (3c9e100) 87.75%.

Additional details and impacted files

@@                    Coverage Diff                     @@
##           integration/compass-1.0.0    #4254   +/-   ##
==========================================================
  Coverage                      87.75%   87.75%           
==========================================================
  Files                            331      331           
  Lines                          20906    20906           
  Branches                        2711     2711           
==========================================================
  Hits                           18346    18346           
  Misses                          2093     2093           
  Partials                         467      467           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheAndrewJackson]

thanks for adding this!!

[TheAndrewJackson]

🚀

The dictField getter is a great idea. Thanks for that refactor!