Stateful ingestion

[LavMatt]

This PR enables stateful ingestion in the custom ingestion source we've written for justice-data.

This means it will soft-delete entities that are not in the data for ingestion in any latest run when compared to the previous run.

We currently delete all justice-data before each ingestion but this is a better approach.

datahub guidance: https://datahubproject.io/docs/metadata-ingestion/docs/dev_guides/add_stateful_ingestion_to_source/

I tested this by not excluding any ids and then reingesting and then adding the exclusions back in for another ingestion, this is logged by the source report (and the entities are removed from datahub)

[2024-11-27 15:09:24,213] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-prison-population-age)
[2024-11-27 15:09:24,214] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-legal-aid-total-expenditure)
[2024-11-27 15:09:24,214] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi015b)
[2024-11-27 15:09:24,214] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:corpGroup:crosscuttingperformanceenquiries
[2024-11-27 15:09:24,215] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,court-number-of-receipts-rape)
[2024-11-27 15:09:24,215] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-prison-ownership)
[2024-11-27 15:09:24,215] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi014)
[2024-11-27 15:09:24,215] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-prison-population-offence)
[2024-11-27 15:09:24,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi015a)
[2024-11-27 15:09:24,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi016)
[2024-11-27 15:09:24,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,entry-to-completion-crown-mean-all)
[2024-11-27 15:09:24,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-prison-type)
[2024-11-27 15:09:24,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi017a)
[2024-11-27 15:09:24,217] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,mi017b)
[2024-11-27 15:09:24,217] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:216} - Soft-deleting stale entity - urn:li:chart:(justice-data,jin-prison-population-sex)

We should look to implement the same across our other custom ingestion sources.

[MatMoore]

Nice 👍🏻