master merge for 0.5.4 release #1756
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add storage_options to delta table create statement
Co-authored-by: Anton Burnashev <[email protected]>
* add delta table partitioning support * document delta table partitioning support * Update docs/website/docs/dlt-ecosystem/destinations/filesystem.md --------- Co-authored-by: Anton Burnashev <[email protected]>
… None and [] as "no data"
* displays log counter when new counter is created * initializes load job counters only when package starts
…tom-auth rest_api: documents pluggable custom auth
…-empty-page RESTClient: stops pagination after empty page (Feat/1637)
* black format * remove code remnant * add scd2 custom boundary timestamp feature * add invalid scd2 active record timestamp test * document scd2 boundary timestamp argument
* collects basic load job metrics in LoadJob * adds remote uri to filesystem copy jobs metrics * adds job id to load package info * adds table name to job metrics * skips run step when serializing trace * adds trace shape test with trace schema * tests job file name too long * docs running pipelines with the same name for different envs * extracts step metrics in common, renames followup jobs * fixes tests * fixes tests * tests delta filesystem for remote_uri * adds exec_info to trace contract test * tests remote_uri for filesystem copy * fixes platform test
* uses normalized column names when linking tables in relational * destination cap if create table if not exits supported * generates IF NOT EXISTS for dlt tables * adds logging for terminal and retry exception in run_managed of load job * passes schema update to be collected in trace in filesystem * fixes job log exception message
…ble format (#1742) * black format * increase minimum deltalake version dependency * enable schema evolution for delta table merge * extract delta table merge logic into separate function * remove big decimal exclusion due to upstream bugfix * evolve delta table schema in empty source case * refactor DeltaLoadFilesystemJob * uses right table path format in delta lake load job * allows to pass schema name when getting delta tables and computing table counts * cleansup usage of remote paths and uris in filesystem load jobs * removes tempfile from file_storage --------- Co-authored-by: Marcin Rudolf <[email protected]>
… coerced to data in incremental (#1748)
Update Snowflake docs
* Expose staging tables truncation to config * Fix comments, add tests * Fix tests * Move implementation from mixing, add tests * Fix docs grammar
* allows to configure external location and named credential for databricks * fixes #1703 * normalizes 'value' when wrapping simple objects in relational, fixes #1754 * simplifies fsspec globbing and allows various url formats that are preserved when reconstituting full url, allows abfss databricks format * adds info on partially loaded packages to docs * renames remote_uri to remote_url in traces * fixes delta for abfss * adds nested tables dlt columns collision test
✅ Deploy Preview for dlt-hub-docs canceled.
|
* always truncates staging tables on athena + replace without iceberg * adds athena staging configs to all staging configs * updates athena tests for staging destination
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
master merge for 0.5.4 release