Enable stateful ingestion in create_cadet_databases_source #383
Labels
DataHub
Issues relating to DataHub - https://datahubproject.io/
metadata quality
python
Pull requests that update Python code
Enabling this is required for any ingestion to remove stale metadata, i.e. metadata that was there yesterday but is gone today. As is stale metadata persist in datahub and hence in find-moj-data
This was tested in the justice-data source and can be replicated for other sources.
see PR: #381
and,
datahub guidance: https://datahubproject.io/docs/metadata-ingestion/docs/dev_guides/add_stateful_ingestion_to_source/
This is the create_cadet_databases ingestion source https://github.com/ministryofjustice/data-catalogue/tree/main/ingestion/create_cadet_databases_source
Data need to be hard deleted from datahub and then reingested with stateful ingestion enabled for it to properly take effect
The text was updated successfully, but these errors were encountered: