Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepares for nested references #1774

Merged
merged 26 commits into from
Sep 9, 2024
Merged

Conversation

rudolfix
Copy link
Collaborator

@rudolfix rudolfix commented Aug 31, 2024

Description

towards #1713

  • verifies merge strategies using selector (adapter) to allow delta/iceberg table exceptions
  • adds PreparedTableSchema which is required by all Job Clients to make sure prepared schema are used (typechecker now ensures that). Also slight changes to interfaces in reference so it is clear which type of schema is used
  • verify_schema must be called explicitly
  • prepare_load_table does not need as_staging flag. it was anyway not used consistently. and was needed only to generate SQL on Athena
  • (breaking/obscure) if primary key was used in nested table, linking was not created in relational.py. now linking is skipped when nested row is fitted into table that is not nested (does not have a parent). a rare case of someone that does not want dlt linking
  • is_nested_table is used to tell if table is nested. nested table has a parent hint. which is used only for this
  • allows to update table schemas from tables and table diffs
  • deprecates is_force_iceberg
  • shortens temp table names in sql jobs
  • moves type mappers to capabilities
  • unifies column props and hints, adds dict of known props

@rudolfix rudolfix added ci full run the full load tests on pr breaking This issue introduces breaking change labels Aug 31, 2024
@rudolfix rudolfix self-assigned this Aug 31, 2024
Copy link

netlify bot commented Aug 31, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit f3654b9
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66ded412f8def3000822b85b

@rudolfix rudolfix force-pushed the feat/prepares-for-nested-references branch from 9111fa2 to 07b7ad9 Compare September 1, 2024 16:13
@rudolfix rudolfix added the sprint Marks group of tasks with core team focus at this moment label Sep 2, 2024
@rudolfix rudolfix marked this pull request as ready for review September 3, 2024 15:45
@rudolfix rudolfix force-pushed the feat/prepares-for-nested-references branch from 1a57a19 to 0508b11 Compare September 9, 2024 09:00
@rudolfix rudolfix merged commit dad2a08 into devel Sep 9, 2024
54 of 55 checks passed
@rudolfix rudolfix deleted the feat/prepares-for-nested-references branch September 9, 2024 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking This issue introduces breaking change ci full run the full load tests on pr sprint Marks group of tasks with core team focus at this moment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant