- Privacy Notices are not needed for the TCF Experience.  Instead, the content is generated at runtime by looking at what TCF data uses are on current systems.
- Add a new experience type: the TCF overlay.
- Expands the abilities to retrieve and save consent preferences with respect to "vendors", "data_uses" and "features", not just "privacy notices".
- Relaxes constraints on PrivacyPreferenceHistory, ServedNoticeHistory, CurrentPrivacyPreference, and LastServedNotice to use these tables to store when preferences were saved and served against not just notices, but vendors, data uses, and features.

…s/vendors.

Improve logic around loading tcf data uses.

… need to track consent mechanism, enforcement level, or has_gpc_flag.

- Start drying up code related to saving privacy preferences and unit testing w.r.t data use, vendor, or feature.

…notices were served for various identities. Refactor methods now that we're sharing a lot of code here.

…s on getting settings.

- Split TCF schemas into their own file
- Add test verifying that we can't create PrivacyPreferenceHistory records against multiple preference types.
- Rename schemas for saving/serializing that notices were served to be more generic.
- Write unit test for saving that tcf items were served.

…with many data uses, rather than a "data use" directly, as a purpose can be associated with many data uses.

- Temporarily pin to a fideslang commit on a branch and pull in the tcf purpose mapping from there.

…thod.

- Fix bug with saving current privacy preferences with respect to a special purpose and special feature.

- Fix duplicate region in EEA_COUNTRIES
- Add test for ConsentSetting model
- Add test for PrivacyExperience.get_should_show_banner in the case where it is a TCF overlay - I return True here
- Add test for PrivacyExperience.get_related_privacy_notices for TCF overlay - exits early with an empty list - not relevant here!
- Add method tests for cache_saved_and_served_on_consent_record for special purpose, notice, and vendor
- Add docstrings to TCF schemas and move TCFExperienceContents out of the file for schemas that are returned via the API
- Add test for PrivacyPrefenceHistory.preference_type property
- Add some checks on PrivacyPreferenceHistory and CurrentPrivacyPreference records on create
- Rename PreferenceType enum and preference_type property to be more generic since they're also shared with served notices
- Docstrings for schemas
- Updated to TCF_COMPONENT_MAPPING and TCF_PREFERENCES_FIELD_MAPPING
- Add test for privacy_notice_id - that it works w/ TCF too

- Added more fields to the vendor record
- "has_notices" flag now also takes into account if TCF content exists too.
- Tried to take care of areas accessing info off of the privacy notice history
- "relevant_systems" definition has changed, and also now factors in TCF components.
- Fix bug where wrong field is returned in historical report
- Add validation on purposes and special purposes and duplicates for both saving preferences and serving TCF components
- Surface tcf version in consent report

- Fix bug related to Consent Settings scope name, copy/paste error
- Fix test that relaxes returning systems as related even if they do not have system wide enforcement

…y and regular overlay, when searching for Privacy Experiences. Region filters will further refine to return the most appropriate overlay altogether.

- Bump downrev of fides migration

# Conflicts:
#	requirements.txt
#	src/fides/api/util/connection_util.py

…so we have the system and privacy declaration changes) but also has gvl changes.

…eld.

- Update ARRAY type used on System.privacy_declarations to be of the postgresql dialect.  (specifically this allows us to use "overlap")

…features.

…and out, that is separate from "vendor". Saving preferences against a vendor apply to multiple systems, while saving against a fides key only applies to the particular system.

- Have TCFFeature record inherit from fideslang Features.
- Only embed TCF components beneath other components if they don't already exist.
- Add the ability to save that consent for specific systems was served, and to also save preferences against a specific system.

…contents

# Conflicts:
#	tests/ops/util/test_tcf_util.py

…, add docstring to privacy experience endpoint, rename confusing query param, define has_tcf_contents_variable close to where we build the base contents.

…key to avoid potential overlaps with custom vendor_ids from the dictionary.

…res, or special features were served.

…idated.

- Sort legal_bases by name

…sing for the same use.

…s appear. In short, a system shows up if it has a gvl-relevant data use, and a legal basis of consent or legitimate interests. If it just has a gvl feature, that is not enough for the system to appear. Further, if a feature is going to show up, it has to be on the same privacy declaration as a gvl data use.

…related purposes.

…s. This method runs when a privacy preference is saved, and it takes a snapshot of the systems that are relevant for that preference at that point in time.

For TCF systems, a starting point for the system to be relevant is that it has to have a GVL-related data use and a legal basis of processing of consent or legitimate interests.  From there, we further filter those systems that match a specific use, feature, or system/vendor id.

- Test cache_saved_and_served_on_consent_record against features
- Update _validate_before_saving_consent_history test to include systems

…ing method not completely handling the tcf overlay type.

Split out served notice history endpoints from related privacy preference endpoints for pylint - privacy preference endpoints file way too large.

…ng) work with TCF without error.

- Don't return TCF data in the privacy center unless TCF is turned on. (hopefully the FE is just looking at privacy preferences saved against notices).

# Conflicts:
#	requirements.txt

Pin fideslang to TCF work branched off of 1.4.6.  Bump downrev of tcf columns.

- Removing auto-generated messaging template commits from my already-large migration.
- Revert consent settings endpoint to being public again
- Add an alias to for has_notices query param.
- Try to better clarify when saving TCF preferences that nothing is saved if preferences are an empty list.
- Fix mypy region issue
- Add code comments to automatically generated migration that adds a lot of new columns
- Get rid of _clone_top_level_record_then_add_legal_bases function
- Add assert_length_of_tcf_sections helper, and add an auto use fixture for clearing the cache
- "..ad_serving" is not a fideslang data use

# Conflicts:
#	requirements.txt
#	tests/fixtures/application_fixtures.py

- Update features to v3
- Get rid of lru_cache.
- Rearrange where we're extending legal bases

…ven (#3870)

Co-authored-by: eastandwestwind <[email protected]>
Co-authored-by: Adam Sachs <[email protected]>

Conflicts: Changelog, migrations.