SDP Generic Pipeline

[gwionap]

Area to discuss how the CoE could potentially support on the generic SDP pipeline development.

[Thomas-Hirsch]

One thought is around iceberg and implementing where useful. I'm also interested in it from the user's perspective regarding time travel, how to implement and explain it

[gwionap]

Testing when iceberg is involved

Projects looking at this

• SDP Pipeline: Want to replace s3_data_packer functionality with iceberg and optimise. Integration tests are failing due to needing to mock the iceberg + Athena functionality
• Transform Table: Tech Review group suggested looking into localstack Athena rather than mocking dbt model queries with duckdb.

Suggestion for CoE

Look into how to use Localstack to mock tests for iceberg tables and for the backend of dbt tests.

[gwionap]

This is probably now defunct following conversations with Soumaya.