Toggle to Prevent Deletion of Files Loaded into the Data Warehouse

[neuromantik33]

Feature description

We request the addition of a feature toggle to the DLT data load tool library that allows users to prevent the deletion of files once they are loaded into the final data warehouse. This feature will help create a log of all loaded files and facilitate testing processes.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

Our primary design goal for most of our data ingestion requirements includes the following stages:

1. Extract (dlt): Extract data from third-party sources and store it in blob storage in various unstructured formats (JSON, Parquet, CSV).
2. Load (dlt): Load the extracted data from blob storage into the input schema of our data warehouse. Move unstructured data to an archive path to prevent vendor lock-in and support full loads.
3. Transform (dbt): Apply tests and business transformations to the tables created by dlt using dbt.

Currently, the staging storage used by dlt to facilitate the loading process is not a true log/archive and is deleted after the load operation. I would like to propose a feature toggle that allows users to deactivate the deletion of any files loaded into the final data warehouse.

Proposed solution

Any classes implementing

class SupportsStagingDestination:
    """Adds capability to support a staging destination for the load"""

    def should_load_data_to_staging_dataset_on_staging_destination(
        self, table: TTableSchema
    ) -> bool:
        return False

    def should_truncate_table_before_load_on_staging_destination(self, table: TTableSchema) -> bool:
        # the default is to truncate the tables on the staging destination...
        return True

should have an option to override the default behavior of should_truncate_table_before_load_on_staging_destination (key is yet to be determined). With of course the default being False in order not to break any existing clients.

Related issues

No response